You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey @choznerol!
You're right, the documentation is wrong.
I'm currently working on a big refactor (#77) which will completely remove this part of the code,
At the same time, I write a better documentation that will remove this kind of outdated comment.
https://github.com/meilisearch/tokenizer/blob/1dfc8ad9f5b338c39c3bc5fd5b2d0c1328314ddc/src/analyzer.rs#L300-L301
This default behavior mentioned in the comment seems to have changed in #27 :
The text was updated successfully, but these errors were encountered: