You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
python -m learn_joint_bpe_and_vocab --input corpus.en corpus.ch -s 30000 -o bpe.codes --write-vocabulary bpe.vocab.en bpe.vocab.ch
After running the above command, the following hints appear:no pair has frequency >= 2. Stopping。
I don't understand what this message means. I hope to give it a answer, thank you.
The text was updated successfully, but these errors were encountered:
you have reached the defined number of merge operations.
there is no pair of subword units that occurs more than once.
if you try to learn BPE operations on a small corpus, you might encounter the second condition. Depending on your use case, this might be fine, but I would usually recommend to use a large corpus to learn BPE operations.
python -m learn_joint_bpe_and_vocab --input corpus.en corpus.ch -s 30000 -o bpe.codes --write-vocabulary bpe.vocab.en bpe.vocab.ch
After running the above command, the following hints appear:no pair has frequency >= 2. Stopping。
I don't understand what this message means. I hope to give it a answer, thank you.
The text was updated successfully, but these errors were encountered: