Vocab length != word vector count

[tocab]

Hey, there is an inconsistency between the count of words in the vocabulary file of the English word vectors for 25000 BPE merges and the count of word vectors with a dimensionality of 200. In the vocab file, there is a count of 25000 words, but the .bin file contains 25777 word vectors after loading it into gensim.

Additionally, the order of the word vectors differs from the vocab file. For example, the word "▁explanation" is on position 19387 in the vocab file, but on position 9138 in the .bin file.

It would be very helpful if the index of the vocabulary in all files would be in the same order and have the same count.

[tocab]

As a workaround, I iterated the words over the vocab file and looked up the words by the words. During this process, the following words were not found in the word vectors:

<s>, </s>, ▁distric, ptember, bruary, ▁performan, orporated, ▁headqu, ▁attem, ▁mathem, ▁passeng, uguese, ▁azerbai, ▁compris, urday, ▁emplo, ▁portra, ▁thous, ▁lithu, ▁leban, ▁councill, ▁specim, ▁molec, ▁entrepren, ▁predecess, ▁glouc, ▁earthqu, ▁istan, imination, ▁infloresc, ▁ingred, chiidae, ▁sofl, ürttemberg, ▁practition, echua, eteries, bridgeshire, ▁nudi, rzys, tokrzys, uchestan, ▁taekw, kopol, giluyeh, ▁fute, ivisie, marthen, ▁gillesp, aziland, scray, alandhar, azulu, alisco

[solomatov]

I have absolutely the same problem. Is there any chance that it will be fixed?

[bheinzerling]

I've almost finished the next version of BPEmb, in which this issue will be fixed. I'm currently figuring out where to host it, since it is a bit larger and doesn't fit on the current web space.

[solomatov]

Thank you for your work! You save everybody a lot of time and resources by providing us with a pretrained models.

[bheinzerling]

This is (finally) fixed in the latest version.