Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smart_open cannot be imported from within a zip file #347

Closed
eric-gravyty opened this issue Sep 10, 2019 · 1 comment · Fixed by #349
Closed

smart_open cannot be imported from within a zip file #347

eric-gravyty opened this issue Sep 10, 2019 · 1 comment · Fixed by #349

Comments

@eric-gravyty
Copy link

eric-gravyty commented Sep 10, 2019

I'm trying to use smart_open from within an Apache Spark application by sending it to Spark wtihin a zip containing my application's third party dependencies. It works for all the other modules I'm importing, but the _get_version function in __init__.py tries and fails to open the VERSION file.

Traceback (most recent call last):
  File "user_code.py", line 5, in <module>
    import smart_open
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/mnt/yarn/usercache/hadoop/appcache/application_1567622467027_0006/container_1567622467027_0006_01_000001/3rdparty.zip/smart_open/__init__.py", line 39, in <module>
  File "/mnt/yarn/usercache/hadoop/appcache/application_1567622467027_0006/container_1567622467027_0006_01_000001/3rdparty.zip/smart_open/__init__.py", line 35, in _get_version
  File "/mnt/yarn/usercache/hadoop/appcache/application_1567622467027_0006/container_1567622467027_0006_01_000001/3rdparty.zip/smart_open/smart_open_lib.py", line 307, in open
  File "/mnt/yarn/usercache/hadoop/appcache/application_1567622467027_0006/container_1567622467027_0006_01_000001/3rdparty.zip/smart_open/smart_open_lib.py", line 512, in _shortcut_open
NotADirectoryError: [Errno 20] Not a directory: '/mnt/yarn/usercache/hadoop/appcache/application_1567622467027_0006/container_1567622467027_0006_01_000001/3rdparty.zip/smart_open/VERSION'

storing the version in a .py file and importing it should solve this issue.

@piskvorky
Copy link
Owner

Solved by #344, I think. @mpenkov can we finish and merge that PR? It seems to bite quite a few people.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants