Added save/load functionality to AnnoyIndexer

[fortiema]

Having Annoy integrated directly into gensim is really great, but one feature that I was personally missing is the ability to save/load indexes. I am working with indexes in the 10s of GB and having to recreate them every time I run my code is a waste of time.

So I added a simple save/load interface that is similar to Annoy.

For example this code:

fname = 'index'
if os.path.exists(fname):
    self.index_en = AnnoyIndexer()
    self.index_en.load(fname)
    self.index_en.model = model
else:
    self.index_en = AnnoyIndexer(model, 1)
    self.index_en.save(fname)

Will create 2 files, index and index.d. Both files must be present when using the load function, otherwise nothing happens.

For this to work, I also added a if case in the constructor to allow for object creation without passing model and num_trees.

try/except on import and using pickle protocol v2 to stay 2-3 compatible.

All comments and suggestions are welcome.

[fortiema]

Added basic test cases to test_similarities for both Word2Vec and Doc2Vec

[tmylk]

Thanks for the PR!

Could you please add a line to CHANGELOG and update the annoy notebook tutorial with the new functionality?

[fortiema]

Sure! Will push when done.

[fortiema]

Any other suggestions to make this interface more robust?

[tmylk]

Please use smart_open as in https://github.com/RaRe-Technologies/gensim/blob/6a289fefd72f038c8cc14826f63624950f5de1f8/gensim/utils.py#L896

[tmylk]

Please use smart_open as in https://github.com/RaRe-Technologies/gensim/blob/6a289fefd72f038c8cc14826f63624950f5de1f8/gensim/utils.py#L907

[tmylk]

It has to raise IOError

[tmylk]

Thanks for the quick fix. Once 2.6 tests runs we could merge.

Also it would be interesting to see a test where 2 parallel processes load the same model from disk and mmap the same index file?

[fortiema]

Great suggestion, let me add this as well!