Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cannot import name 'open' from 'smart_open' #489

Closed
littleyee opened this issue Apr 26, 2020 · 16 comments
Closed

cannot import name 'open' from 'smart_open' #489

littleyee opened this issue Apr 26, 2020 · 16 comments
Labels
need-info

Comments

@littleyee
Copy link

I am receiving the error
File "C:\ProgramData\Anaconda2\lib\site-packages\gensim\utils.py", line 45, in
from smart_open import open

ImportError: cannot import name open
I am using python 2.7.16, the gensim is in 3.8.2 and smart-open is 1.10.1. Any ideas of what is going on?
@mpenkov
Copy link
Collaborator

mpenkov commented Apr 27, 2020

What is the output of:

import smart_open
dir(smart_open)

Here is the output I see on my system (different to yours) as an example:

$ python
Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import smart_open
>>> dir(smart_open)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'bytebuffer', 'compression', 'concurrency', 'constants', 'doctools', 'gcs', 'hdfs', 'http', 'local_file', 'logger', 'logging', 'open', 'parse_uri', 'register_compressor', 's3', 's3_iter_bucket', 'smart_open', 'smart_open_lib', 'ssh', 'transport', 'utils', 'version', 'webhdfs']
>>>

@littleyee
Copy link
Author

littleyee commented Apr 28, 2020 •
edited by mpenkov

The output is

Python 2.7.16 (v2.7.16:413a49145e, Mar  4 2019, 01:37:19) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import smart_open
>>> dir(smart_open)
['__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', 'bytebuffer', 'doctools', 'hdfs', 'http', 'logger', 'logging', 'open', 'register_compressor', 's3', 's3_iter_bucket', 'smart_open', 'smart_open_lib', 'ssh', 'version', 'webhdfs']
>>>

@mpenkov
Copy link
Collaborator

mpenkov commented Apr 28, 2020

Interesting. The open function is in the list of functions offered by the smart_open package. I'm not sure why it's not importing.

I originally suspected this is some kind of Python 2 issue, but it works in a Py2 environment here without any problems:

$ python
Python 2.7.17 (default, Apr 15 2020, 17:20:14) 
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from smart_open import open
>>> import smart_open
>>> smart_open.__version__
'1.10.1'
>>>

@Abhishek-Prajapat
Copy link

Abhishek-Prajapat commented Jun 3, 2020 •
edited

I am not getting the open function in the smart_open directory.
`import smart_open

dir(smart_open)

['BZ2File',
'BytesIO',
'DEFAULT_ERRORS',
'IS_PY2',
'P',
'PATHLIB_SUPPORT',
'SSLError',
'SYSTEM_ENCODING',
'Uri',
'builtins',
'cached',
'doc',
'file',
'loader',
'name',
'package',
'path',
'spec',
'boto',
'codecs',
'collections',
'gzip',
'hdfs',
'http',
'importlib',
'io',
'logger',
'logging',
'os',
'pathlib',
'pathlib_module',
'requests',
's3',
's3_iter_bucket',
'six',
'smart_open',
'smart_open_hdfs',
'smart_open_http',
'smart_open_lib',
'smart_open_s3',
'smart_open_webhdfs',
'sys',
'urlparse',
'urlsplit',
'warnings',
'webhdfs']`

@mpenkov
Copy link
Collaborator

mpenkov commented Jun 4, 2020

@Abhishek-Prajapat What smart_open version are you using?

@anantguptadbl
Copy link

anantguptadbl commented Jun 15, 2020 •
edited

@Abhishek-Prajapat , I am also not able to see any open in the smart_open. I am, using the following versions
smart_open-1.8.0
gensim-3.8.3

The issue is that < 1.8.2 , the def open was actually def smart_open
You can install the latest version of smart_open and that will solve the problem. If you have a constraint on downloading the latest and want to solve this specific problem quickly, you can replicate the functions on your side

I solved it by writing my own version of glove2word2vec which is almost copy-paste

glove_input_file = 'glove.6B.50d.txt'
word2vec_output_file = 'glove.6B.50d.txt.word2vec'

def gloveToword2vec(glove_input_file,word2vec_output_file):
    with open(glove_input_file, 'rb') as f:
        num_lines = sum(1 for _ in f)
    with open(glove_input_file, 'rb') as f:
        num_dims = len(f.readline().split()) - 1
    with open(word2vec_output_file, 'wb') as fout:
        fout.write("{0} {1}\n".format(num_lines, num_dims).encode('utf-8'))
        with open(glove_input_file, 'rb') as fin:
            for line in fin:
                fout.write(line)
    print("Completed")

gloveToword2vec(glove_input_file, word2vec_output_file)

@Auburngrads
Copy link

This issue doesn't appear to have been fixed in the latest versions smart_open.

Both 2.0.0 and 2.1.0 state that the smart_open.smart_open() is deprecated in favor of smart_open.open() in smart_open/smart_open_lib.py although this doesn't seem to be what is occurring.

Also, the list of exported functions I see from smart_open does not include open

dir(smart_open)

['BZ2File', 'BytesIO', 'DEFAULT_ERRORS', 'IS_PY2', 'P', 'PATHLIB_SUPPORT', 'SSLError', 'SYSTEM_ENCODING', 'Uri', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'boto', 'codecs', 'collections', 'gzip', 'hdfs', 'http', 'importlib', 'io', 'logger', 'logging', 'os', 'pathlib', 'pathlib_module', 'requests', 's3', 's3_iter_bucket', 'six', 'smart_open', 'smart_open_hdfs', 'smart_open_http', 'smart_open_lib', 'smart_open_s3', 'smart_open_webhdfs', 'sys', 'urlparse', 'urlsplit', 'warnings', 'webhdfs']

My immediate issues with gensim appear to have been fixed by updating gensim/utils.py to read

from smart_open import smart_open

@piskvorky
Copy link
Owner

piskvorky commented Jul 29, 2020 •
edited

@Auburngrads are you sure you're using the latest smart_open?

My guess would be you have two (or more) instances of smart_open installed, perhaps in different virtual environments. And one of them is old (smart_open.smart_open) which is what the gensim environment is picking up.

Otherwise it's really weird. How did you install smart_open? Can you please uninstall it, re-install the latest version again and post the full installation log here (your installation command + all output until the installation completes)?

@Auburngrads
Copy link

@piskvorky possible that I would have multiple versions installed, but seems unlikely.

Not using any virtual environments that I'm aware of and only have smart_open installed to my only custom environment (not found in base). Installed using pip in anaconda prompt.

Removed and reinstalled pip install smart_open, verified version 2.1.0 is installed and have the same issue

Installation log file is attached.
so_log.txt

@piskvorky
Copy link
Owner

piskvorky commented Jul 29, 2020 •
edited

Thanks. When you open the file c:\users\aubur\appdata\local\r-mini~1\envs\r-reticulate\lib\site-packages\smart_open\smart_open_lib.py, do you see the line def open( there, on line 109?

My MD5 hash of that file in release 2.1.0 is MD5 (smart_open_lib.py) = 5fcfed617811f6b63cd8ff44df118e83.

Alternatively, to check whether the same installed file is being imported, what does import smart_open; print(smart_open.__file__) say?

@Auburngrads
Copy link

Auburngrads commented Jul 29, 2020 via email

@piskvorky
Copy link
Owner

What does import smart_open; print(smart_open.__file__) say?

@Auburngrads
Copy link

Auburngrads commented Jul 29, 2020 via email

@piskvorky
Copy link
Owner

piskvorky commented Jul 29, 2020 •
edited

What's up with the r-mini~1 vs r-miniconda? Do both paths exist, or why are the log paths truncated like that?

@Auburngrads
Copy link

Auburngrads commented Jul 29, 2020 via email •
edited by piskvorky

@piskvorky
Copy link
Owner

piskvorky commented Jul 29, 2020 •
edited

Great, thanks for following up. I'm closing this ticket – the issue is with conflicting Python versions as expected, not smart_open as such.

For others stumbling on this ticket in the future: please check your environment Python paths as per above. Make sure the problem is not in your local setup.

Repository owner locked as resolved and limited conversation to collaborators Aug 10, 2020
Repository owner deleted a comment from mohammadjavadnouri Aug 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
need-info
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@piskvorky @mpenkov @Auburngrads @anantguptadbl @littleyee @Abhishek-Prajapat