Static compile of word2vec

[piskvorky]

Avoid weird problems with dynamic pyximport compilation.

Instead, bundle the cython-generated C file and try to compile that, at gensim install time.

If installation fails, show a warning but install gensim anyway (=the compilation is optional).

[piskvorky]

@ianozsvald @kmike merged into develop. This will be part of the next gensim release (~some time next week).

But I still need some feedback on whether it works or not :) A straightforward python setup.py install test install from tarball on your machine welcome.

[ianozsvald]

With Python 3.4 (Anaconda), I've pulled & installed the develop branch using python setup.py install, it installed fine (I saw the cython log go past). Inside IPython it seems to work, no warnings:

In [2]: from gensim.models import word2vec
In [3]: model = word2vec.Word2Vec(['a', 'b']*10)
In [4]: model.vocab 
Out[4]: 
{'a': <gensim.models.word2vec.Vocab at 0x7f109fa08898>,
 'b': <gensim.models.word2vec.Vocab at 0x7f109fa08908>}

In [3]: word2vec.FAST_VERSION 
Out[3]: 1

Is there anything else I can check for?

[piskvorky]

Cheers Ian. Nothing else for now, I appreciate it. If you get your hands on a Windows box in the future, checking there too would be great :)

[kmike]

It works fine for me too (Python 2.7 & Python 3.4, homebrew, OS X), thanks!.

There were some gotchas in Python 2.7. I cloned the repo, executed setup.py install, started ipython and executed code @ianozsvald provided. In Python 2.7 it fails because the extension is bot built inplace, while "import gensim" imports gensim from the checkout, not the installed gensim. I exited the console, changed dir to its parent, started console again and run the code again - it also fails because there is __init.py__ file in gensim top-level folder, so 'import gensim' tried to do import from that (what is this file for?). After changing the dir to another place it works fine.

The warning it was failing with in a first case is

gensim/models/word2vec.py:395: UserWarning: Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`
  warnings.warn("Cython compilation failed, training will be slow. Do you have Cython installed? `pip install cython`")

There are other places where Cython is mentioned; it seems it should be cleaned up.

[boyangli]

Could anyone tell me what's the status of this issue? I am still getting that "Cython compilation failed, training will be slow." I tried all kinds of python configurations. Can't get any to work.

[piskvorky]

@boyangli: this issue was fixed about a year ago. Your troubles must be unrelated.

To know more, try posting your gensim installation log (to its own github issue, or to the gensim mailing list, not here). From the log you should be able to see what the problem was = why the compilation failed.

[boyangli]

Thank you, @piskvorky . I switched to scipy 0.15.1, and the problem seems resolved. It must be a compatibility issue with 0.16. I will post a most detailed description as a separate issue.

[gojomo]

@boyangil – sounds like known issue #382, patched in #415.

[boyangli]

@gojomo If it's been fixed, I'm still seeing it, even after downloading the source code from github directly.

[boyangli]

A new issue has been filed #429