You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The ngram vocab currently brackets the <\s> marker and extracts ngrams from "<<\s>>". Those subwords aren't trained because the indices are never added to the subwordsVec in the vocab via:
if word.word() == util::EOS{
subword_indices.push(Vec::new());continue;}
It's fairly unlikely to encounter those ngrams anywhere but we should fix this once we figured out the bug.
The text was updated successfully, but these errors were encountered:
The ngram vocab currently brackets the
<\s>
marker and extracts ngrams from"<<\s>>"
. Those subwords aren't trained because the indices are never added to thesubwords
Vec
in the vocab via:It's fairly unlikely to encounter those ngrams anywhere but we should fix this once we figured out the bug.
The text was updated successfully, but these errors were encountered: