-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update ngramizer.jl #148
Update ngramizer.jl #148
Conversation
for index in 1:(n_words - n + 1) | ||
token = join(words[index:(index + n - 1)], " ") | ||
tokens[token] = get(tokens, token, 0) + 1 | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, imo the Travis CI is reporting an error due to change in implementation. Have a look at the tests corresponding to this change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done Sir 💯
Thank you so much for this. May you please, update the documents and add the tests as well? |
Reflects changes made to ngrams() for n>1.
Reflects changes made to ngrams() for n greater than 1
Updated Tests and Documentation to reflect the changes. |
That could have been better implemented as an option IMO. Or by having a variant of |
We could open an issue for the same should be good first PR. |
The previous function returned all the ngrams from n to 1.
![image](https://user-images.githubusercontent.com/17949650/56772594-46cdcc80-67d8-11e9-9fdb-11e705ad6a32.png)
This error is also recorded in the documentation.