Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
4269: Remove dependency that requires libstdc++ r=dureuill a=dureuill Removes the dependency that caused the additional runtime dependency on libstdc++ by disabling the default features of the hf tokenizer. ## Discussion - This removes a feature that is using a C++ dependency and is supposed to accelerate the tokenizer. As the tokenizer is likely to be a significant bottleneck for embedding texts using a HF model, this is an issue. - We should at least rerun the movies vector indexing and check that it still works correctly and that it has a runtime in the ballpark of what it used to be. Co-authored-by: Louis Dureuil <louis.dureuil@xinra.net>
- Loading branch information