[MRG] Fix for #814

[jayantj]

The current code attempts to decode all characters for vocab words individually as ascii.
Instead, the code is changed to do the following - characters are read as raw bytes first, then decoded as utf-8 once the entire word has been read.

Also includes a test fasttext model with non-ascii vocab words, and checks vector lookup for the same.

Haven't been able to test the newly released FastText vectors for wiki in various languages since the models are large-ish (~10 GB).

Related issue reported here

[anmolgulati]

@jayantj This looks good, but should we wait for to test this on the new pre-trained models first and then ask Lev to merge? Though I feel, it shouldn't create any difference, as you have already added a test for non-ascii words.

[tmylk]

Agree with @anmol01gulati. Does it work on https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md, @jayantj ?

[piskvorky]

The previous behaviour was not good, how did that pass the code review and unit tests :(

Does fastText always output utf8? Is the encoding really fixed?

[piskvorky]

@jayantj @tmylk the fastText file header is incorrect (states I wrote the code in 2013). Please fix.

[piskvorky]

@tmylk use better issue/PR titles, so search engines can index them and users find them, when they hit the same error.

[piskvorky]

ping @jayantj status on the questions above?

[jayantj]

The fastText readme says that the training file should contain utf-8 encoded text. That was the rationale behind choosing utf8 as the encoding while deserializing the model.

Looking at the fastText code and experimenting with different encodings though, fastText doesn't really decode the bytes in the training file and simply treats the input file as a stream of bytes.

The only place it does make use of the values inside the bytes is while tokenizing the text, where it checks each byte against whitespace characters (' ', '\n', '\r', '\t', '\v', '\f')

So, encodings which have the same values for these characters as ascii would produce sane output. (utf16 and utf32 don't, lots of others do). Original encoding information isn't stored anywhere though.

I can think of two ways to handle this -

1. Assume words in input file are utf8 encoded (since FastText readme mentions it, and it might be enforced in the future), and raise a more friendly error when the input isn't valid utf8
2. Provide an parameter encoding allowing the user to specify the input file encoding (defaulting to utf8, of course)

Any thoughts?

[jayantj]

Re: file header, I'll create a single PR with the updated header and the fix for the above issue, whichever one we go with.

[tmylk]

Option 2 sounds as the most flexible