Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FAST_VERSION is -1 when using latest gensim (3.8.1) #2794

Closed
cfd-01 opened this issue Apr 14, 2020 · 15 comments
Closed

FAST_VERSION is -1 when using latest gensim (3.8.1) #2794

cfd-01 opened this issue Apr 14, 2020 · 15 comments
Labels
bug Issue described a bug impact HIGH Show-stopper for affected users performance Issue related to performance (in HW meaning) reach HIGH Affects most or all Gensim users
Milestone

Comments

@cfd-01
Copy link

cfd-01 commented Apr 14, 2020

Problem description

assert gensim.models.doc2vec.FAST_VERSION > -1 fails with gensim 3.8.1, even though it works with 3.7.0.

Steps/code/corpus to reproduce

When I install the latest version of gensim using pip install --upgrade gensim, I observe the following:

>>> import gensim.models.doc2vec
>>> assert gensim.models.doc2vec.FAST_VERSION > -1, "This will be painfully slow otherwise"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: This will be painfully slow otherwise

When I roll back to version 3.7, the assertion passes:

>>> import multiprocessing
>>> import gensim.models.doc2vec
>>> assert gensim.models.doc2vec.FAST_VERSION > -1, "This will be painfully slow otherwise"
>>> 

Versions

>>> import platform; print(platform.platform())
Darwin-18.7.0-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
Python 3.7.6 (default, Dec 30 2019, 19:38:28) 
[Clang 11.0.0 (clang-1100.0.33.16)]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.18.1
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.4.1
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.8.1
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION -1
@piskvorky
Copy link
Owner

piskvorky commented Apr 14, 2020

What did the 3.8.1 installation look like? Can you post the full installation output from pip?

That should contain the concrete issue.

@piskvorky piskvorky added the need info Not enough information for reproduce an issue, need more info from author label Apr 14, 2020
@cfd-01
Copy link
Author

cfd-01 commented Apr 14, 2020

Collecting gensim
  Using cached gensim-3.8.2-cp37-cp37m-macosx_10_9_x86_64.whl (23.7 MB)
Requirement already satisfied, skipping upgrade: six>=1.5.0 in ./venv/lib/python3.7/site-packages (from gensim) (1.12.0)
Requirement already satisfied, skipping upgrade: scipy>=1.0.0 in ./venv/lib/python3.7/site-packages (from gensim) (1.4.1)
Requirement already satisfied, skipping upgrade: smart-open>=1.8.1 in ./venv/lib/python3.7/site-packages (from gensim) (1.10.0)
Requirement already satisfied, skipping upgrade: numpy>=1.11.3 in ./venv/lib/python3.7/site-packages (from gensim) (1.18.1)
Requirement already satisfied, skipping upgrade: requests in ./venv/lib/python3.7/site-packages (from smart-open>=1.8.1->gensim) (2.23.0)
Requirement already satisfied, skipping upgrade: boto3 in ./venv/lib/python3.7/site-packages (from smart-open>=1.8.1->gensim) (1.12.15)
Requirement already satisfied, skipping upgrade: google-cloud-storage in ./venv/lib/python3.7/site-packages (from smart-open>=1.8.1->gensim) (1.27.0)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in ./venv/lib/python3.7/site-packages (from requests->smart-open>=1.8.1->gensim) (2.9)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in ./venv/lib/python3.7/site-packages (from requests->smart-open>=1.8.1->gensim) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./venv/lib/python3.7/site-packages (from requests->smart-open>=1.8.1->gensim) (1.25.8)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ./venv/lib/python3.7/site-packages (from requests->smart-open>=1.8.1->gensim) (2019.11.28)
Requirement already satisfied, skipping upgrade: botocore<1.16.0,>=1.15.15 in ./venv/lib/python3.7/site-packages (from boto3->smart-open>=1.8.1->gensim) (1.15.15)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in ./venv/lib/python3.7/site-packages (from boto3->smart-open>=1.8.1->gensim) (0.9.5)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in ./venv/lib/python3.7/site-packages (from boto3->smart-open>=1.8.1->gensim) (0.3.3)
Requirement already satisfied, skipping upgrade: google-cloud-core<2.0dev,>=1.2.0 in ./venv/lib/python3.7/site-packages (from google-cloud-storage->smart-open>=1.8.1->gensim) (1.3.0)
Requirement already satisfied, skipping upgrade: google-auth<2.0dev,>=1.11.0 in ./venv/lib/python3.7/site-packages (from google-cloud-storage->smart-open>=1.8.1->gensim) (1.13.1)
Requirement already satisfied, skipping upgrade: google-resumable-media<0.6dev,>=0.5.0 in ./venv/lib/python3.7/site-packages (from google-cloud-storage->smart-open>=1.8.1->gensim) (0.5.0)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1 in ./venv/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.15->boto3->smart-open>=1.8.1->gensim) (2.8.1)
Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in ./venv/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.15->boto3->smart-open>=1.8.1->gensim) (0.15.2)
Requirement already satisfied, skipping upgrade: google-api-core<2.0.0dev,>=1.16.0 in ./venv/lib/python3.7/site-packages (from google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (1.16.0)
Requirement already satisfied, skipping upgrade: cachetools<5.0,>=2.0.0 in ./venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (4.1.0)
Requirement already satisfied, skipping upgrade: setuptools>=40.3.0 in ./venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (41.2.0)
Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in ./venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (0.2.8)
Requirement already satisfied, skipping upgrade: rsa<4.1,>=3.1.4 in ./venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (4.0)
Requirement already satisfied, skipping upgrade: googleapis-common-protos<2.0dev,>=1.6.0 in ./venv/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.16.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (1.51.0)
Requirement already satisfied, skipping upgrade: pytz in ./venv/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.16.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (2019.3)
Requirement already satisfied, skipping upgrade: protobuf>=3.4.0 in ./venv/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.16.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (3.11.3)
Requirement already satisfied, skipping upgrade: pyasn1<0.5.0,>=0.4.6 in ./venv/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (0.4.8)
Installing collected packages: gensim
  Attempting uninstall: gensim
    Found existing installation: gensim 3.7.0
    Uninstalling gensim-3.7.0:
      Successfully uninstalled gensim-3.7.0
Successfully installed gensim-3.8.2

@piskvorky
Copy link
Owner

piskvorky commented Apr 14, 2020

Thanks. I installed using the same wheel gensim-3.8.2-cp37-cp37m-macosx_10_9_x86_64.whl in Python3.7 and I can replicate this:

Screen Shot 2020-04-14 at 19 14 53

@menshikh-iv @mpenkov were there any changes between 3.7 and 3.8 when it comes to packaging binary wheels?

@piskvorky piskvorky removed the need info Not enough information for reproduce an issue, need more info from author label Apr 14, 2020
@piskvorky
Copy link
Owner

Btw that unable to import 'smart_open.gcs seems out of place. Is it necessary to pollute the gensim import with such scary looking (and irrelevant) printouts?

@menshikh-iv
Copy link
Contributor

menshikh-iv commented Apr 15, 2020

were there any changes between 3.7 and 3.8 when it comes to packaging binary wheels?

no (and I guess that this is a problem), at least I have several ideas about "what can be wrong"

@piskvorky can you please check that behavior on gensim==3.7.3, gensim==3.8.0, gensim==3.8.1, gensim==3.8.2 with py37? I need this information for investigation.

@piskvorky
Copy link
Owner

piskvorky commented Apr 15, 2020

gensim==3.7.3, gensim==3.8.0, gensim==3.8.1:

from gensim.models import word2vec; print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 0

gensim==3.8.2:

from gensim.models import word2vec; print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION -1

The same machine, OS, Python3.7 & virtual env, just changing the gensim version with pip install gensim==x.y.z.

Probably the most critical issue we have right now: Gensim's most used models are unusable.

@piskvorky piskvorky added bug Issue described a bug performance Issue related to performance (in HW meaning) reach HIGH Affects most or all Gensim users impact HIGH Show-stopper for affected users labels Apr 15, 2020
@menshikh-iv
Copy link
Contributor

Thanks for info, I’ll look into this week @piskvorky

@menshikh-iv
Copy link
Contributor

@piskvorky one more thing, can you do exactly same testing on py35 and py36? I have a feeling that other versions can be affected too?

@piskvorky
Copy link
Owner

Sure. Python3.6:

3.8.1: FAST_VERSION 0
3.8.2: FAST_VERSION -1

@mpenkov mpenkov added this to the 3.8.3 milestone Apr 16, 2020
@menshikh-iv
Copy link
Contributor

menshikh-iv commented Apr 16, 2020

@menshikh-iv
Copy link
Contributor

Related PR MacPython/gensim-wheels#21

@piskvorky
Copy link
Owner

piskvorky commented Apr 16, 2020

I don't have access to 3.5 but I confirm the above wheels for 3.7 and 3.6 work!
FAST_VERSION 0 🚀

@gojomo
Copy link
Collaborator

gojomo commented Apr 16, 2020

What was the problem? Should that file in MacPython/gensim-wheels really name a specific commit, rather than a version? (Or is that just temporary?)

@menshikh-iv
Copy link
Contributor

@gojomo this was temporary, I just need "concrete fresh commit" for my investigations. You can read more about it MacPython/gensim-wheels#21 (comment)

@menshikh-iv
Copy link
Contributor

Fixed by MacPython/gensim-wheels#21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue described a bug impact HIGH Show-stopper for affected users performance Issue related to performance (in HW meaning) reach HIGH Affects most or all Gensim users
Projects
None yet
Development

No branches or pull requests

5 participants