Error: invalid line 2 in BPE codes file when running apply_bpe.py

[jo0704]

Hi,

When running apply_bpe.py to segment given texts with the generated vocabulary I get the following error:

Error: invalid line 2 in BPE codes file: bpeout/vocab
The line should exist of exactly two subword units, separated by whitespace

The exact command lines I used:

echo "#version: 0.2" > bpeout/vocab.seg # add version info
echo bpeout/vocab >> bpeout/vocab.seg
python3 subword-nmt/subword_nmt/apply_bpe.py -c bpeout/vocab.seg <X-EN/de_en/train.de >bpeout/train_out.de

I added the vocab and train file I'm trying to segment:
bpe_vocab.zip

A similar issue was reported here #46 , but it doesn't seem to solve the error in my case.

[rsennrich]

Hi,

your vocab seems to have been produced by a different script, and is invalid. Specifically, there are lines containing just one symbol (__en__), and it's also not distinguishing between word-internal and word-final merge operations.