Invalid pointer when binarize wiki-news-300d-1M.vec

[h2soong]

Hi,
When i binarize wiki-news-300d-1M.vec file, program output
*** Error in `./binarize': free(): invalid pointer: 0x00007f64c490e010 ***
From the backtrace, i know there is something wrong free memory, but i don't find the exact position.
I am surprised at program works normally when i compress GloVe.6B.300d.txt file.

[tca19]

The problem comes from the file wiki-news-300d-1M.vec. After reading the last line, the EOF flag was not set automatically so my program was still reading some float values, and writing them outside the array containing all the values, overwriting what was in memory with the value 0. So when I called free() to free the memory of what was placed after the array of values, it was trying to dereference a NULL pointer, causing the program to crash.

I have fixed the problem. You can now train on wiki-news-300d-1M.vec. Here are the results I get.

create_vocab(): 0.013944s
load_vectors(): 0.523795s
Filename     | Spearman | OOV
==============================
MEN.txt      |    0.614 |   0%
WS353.txt    |    0.583 |   0%
SimLex.txt   |    0.310 |   0%
RW.txt       |    0.426 |   3%
SimVerb.txt  |    0.262 |   0%
evaluate(): 0.005732s

These results are obtained with the default hyperparameters, but you can probably get better results by tuning the hyperparameters.