Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Comment on Analyzer.analyze default is out of date #82

Closed
choznerol opened this issue Apr 5, 2022 · 1 comment · Fixed by #77
Closed

Comment on Analyzer.analyze default is out of date #82

choznerol opened this issue Apr 5, 2022 · 1 comment · Fixed by #77

Comments

@choznerol
Copy link
Contributor

https://github.com/meilisearch/tokenizer/blob/1dfc8ad9f5b338c39c3bc5fd5b2d0c1328314ddc/src/analyzer.rs#L300-L301

This default behavior mentioned in the comment seems to have changed in #27 :

             normalizer: Box::new(IdentityNormalizer),
-            tokenizer: Box::new(UnicodeSegmenter),
+            tokenizer: Box::new(LegacyMeilisearch),
-            normalizer: Box::new(IdentityNormalizer),
+            normalizer: Box::new(normalizer),
@ManyTheFish
Copy link
Member

Hey @choznerol!
You're right, the documentation is wrong.
I'm currently working on a big refactor (#77) which will completely remove this part of the code,
At the same time, I write a better documentation that will remove this kind of outdated comment.

Thanks for your issue!

@bors bors bot closed this as completed in 5418399 May 3, 2022
@bors bors bot closed this as completed in #77 May 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants