Improve pre-trained embeddings loading #24
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, pre-trained embeddings were loaded by simply opening and looping over the file with
python
. This was a problem for a few reasons, mainly because it required the conversion of embeddings in C binary format (.bin
) to C plain text format (.txt
), which were much much larger.TODO
gensim
'sKeyedVectors.load_word2vec_format
to load pre-trained embeddings into memory from a filedebug
functionality, whereby passing the argument--debug
would only load some N number of embeddings. Gensim has a mechanism for only loading this top N number of embeddings, use this!Changed
gensim
'sKeyedVectors.load_word2vec_format
method, no need to convert from.bin
to.txt
. Saber can load any word embeddings in the word2vec format..bin
file is (at least in our case) is ~65% smaller than the corresponding.txt
file.bin
to.txt
format.Resources
Closes #15.