Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploying/Packaging doesn't work from inside a virtualenv #747

Closed
tcdude opened this issue Sep 29, 2019 · 9 comments
Closed

Deploying/Packaging doesn't work from inside a virtualenv #747

tcdude opened this issue Sep 29, 2019 · 9 comments
Assignees
Milestone

Comments

@tcdude
Copy link
Contributor

@tcdude tcdude commented Sep 29, 2019

When trying to use what was formerly known as deployng to package my game for win_amd64/win32, I was unable to get the program to work when running inside a virtualenv.

The build would complete with the usual warnings but my game would crash when launched on windows, with this traceback:

Traceback (most recent call last):
  File run_game.py, line 68, in <module>
  File /home/tc/VSCodeProjects/pyweek28/game/__init__.py, line 32, in main
  File /home/tc/VSCodeProjects/pyweek28/game/app.py, line 74, in __init__
  File /home/tc/VSCodeProjects/pyweek28/game/world.py, line 60, in __init__
  File /home/tc/VSCodeProjects/pyweek28/game/world.py, line 257, in setup_terrain
  File /home/tc/VSCodeProjects/pyweek28/build/__whl_cache__/Pillow-6.1.0-cp37-cp37m-win_amd64.whl/PIL/Image.py, line 2060, in save
  File /home/tc/VSCodeProjects/pyweek28/build/__whl_cache__/Pillow-6.1.0-cp37-cp37m-win_amd64.whl/PIL/Image.py, line 393, in preinit
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 1035, in _handle_fromlist
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 219, in _call_with_frames_removed
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 983, in _find_and_load
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 967, in _find_and_load_unlocked
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 677, in _load_unlocked
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 819, in exec_module
  File /home/tc/VSCodeProjects/pyweek28/build/__whl_cache__/Pillow-6.1.0-cp37-cp37m-win_amd64.whl/PIL/JpegImagePlugin.py, line 41, in <module>
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 1035, in _handle_fromlist
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 219, in _call_with_frames_removed
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 983, in _find_and_load
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 967, in _find_and_load_unlocked
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 677, in _load_unlocked
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 819, in exec_module
  File /home/tc/VSCodeProjects/pyweek28/build/__whl_cache__/Pillow-6.1.0-cp37-cp37m-win_amd64.whl/PIL/TiffImagePlugin.py, line 57, in <module>
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 983, in _find_and_load
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 953, in _find_and_load_unlocked
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 219, in _call_with_frames_removed
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 983, in _find_and_load
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 967, in _find_and_load_unlocked
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 677, in _load_unlocked
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap.py, line 819, in exec_module
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/distutils/__init__.py, line 35, in <module>
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap_external.py, line 724, in exec_module
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap_external.py, line 859, in get_code
  File /home/tc/VSCodeProjects/pyweek28/venv/lib/python3.7/importlib/_bootstrap_external.py, line 916, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\tc\Desktop\Devils Tower\distutils\__init__.py'

At first I suspected Pillow to be the culprit, but after repeating the steps of deploying/packaging on a global interpreter level instead of a virtualenv, it just worked.

I'll try to create a minimal example to replicate this behavior to narrow down the problem.

@Moguri

This comment has been minimized.

Copy link
Collaborator

@Moguri Moguri commented Oct 7, 2019

I always build apps from within a virtualenv, so that in itself is not broken. However, it's been a while since I have built anything much more complicated than the asteroids sample, so there may be an issue with pulling in certain dependencies. Hopefully your minimal example can help shed some light on what's going on.

@Moguri

This comment has been minimized.

Copy link
Collaborator

@Moguri Moguri commented Nov 12, 2019

@tcdude Did you get anywhere with a minimal example?

@tcdude

This comment was marked as off-topic.

Copy link
Contributor Author

@tcdude tcdude commented Nov 13, 2019

@Moguri thanks for reminding me, I haven't gotten around to it yet, but will do so shortly. I had unexpected health issues and was regaining my strength over the last couple of weeks, so my focus was elsewhere.

@tcdude

This comment has been minimized.

Copy link
Contributor Author

@tcdude tcdude commented Nov 13, 2019

I managed to reproduce the problem with a minimal example:
p3dtest.zip
UPDATED EXAMPLE: p3dtest.zip

it simply starts ShowBase and accepts the a key, which leads to it crashing when built inside a virtualenv

Deployed with: Ubuntu 18.04 / Python 3.6.8 / virtualenv from Ubuntu Package Manager
Tested on: Windows 10 64bit

The problem can be narrowed down to saving images with Pillow. When using a virtualenv, the app crashes upon pressing the a-key with pretty much the same error message as evident in my first post, whilst when building on a global interpreter level, the error doesn't occur and everything behaves correctly.

Let me know if you need more information regarding this issue.

Edit: Updated the example to reflect the problem with Pillow/PIL, removing other dependencies

@Moguri Moguri self-assigned this Nov 21, 2019
@Moguri

This comment has been minimized.

Copy link
Collaborator

@Moguri Moguri commented Nov 24, 2019

So, I think this is still related to PIL/Pillow. In other words, I think Pillow isn't working when package from a virtualenv. I did test the supplied sample file, and I can confirm that the package application is not running correctly.

@Moguri Moguri removed their assignment Nov 24, 2019
@tcdude

This comment has been minimized.

Copy link
Contributor Author

@tcdude tcdude commented Nov 28, 2019

Quick update, since on Discord it was mentioned that it possibly could be related to how Ubuntu has some custom patches for pip: I just tried the same using a fresh Manjaro 18.1.3 install with the same result.
I guess it's safe to say that PIL/Pillow is the culprit here.

@rdb rdb added the priority label Dec 1, 2019
@rdb rdb self-assigned this Dec 1, 2019
@tcdude

This comment has been minimized.

Copy link
Contributor Author

@tcdude tcdude commented Dec 1, 2019

I also tested this now on Windows (Windows 10 64-bit / 1809), with pretty much the same results, though strangely enough, @rdb was successful in building from inside a virtualenv with Python 3.7 and the problem didn't appear, whereas I encountered the same problem with both Python 3.6 and 3.7 on Windows.

To reiterate: The problem only results when building/deploying from inside a virtualenv and the Image.save() method from Pillow is being called (in the above example program, when the a-key is being pressed!!)

Update:
I get the following traceback by doing the above:

Known pipe types:
  wglGraphicsPipe
(all display modules loaded.)
OpenGL Warning: wglGetPixelFormatAttribivARB: bad attrib=0x20a9
Traceback (most recent call last):
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\EventManager.py", line 49, in eventLoopTask
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\EventManager.py", line 43, in doEvents
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\EventManager.py", line 99, in processEvent
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\Messenger.py", line 333, in send
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\Messenger.py", line 418, in __dispatch
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\game\main.py", line 11, in dosomething
    img.save('test.png')
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\Pillow-6.2.1-cp37-cp37m-win_amd64.whl\PIL\Image.py", line 2056, in save
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\Pillow-6.2.1-cp37-cp37m-win_amd64.whl\PIL\Image.py", line 388, in preinit
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 1035, in _handle_fromlist
    _call_with_frames_removed(import_, from_name)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 219, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 983, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 967, in _find_and_load_unlocked
    module = _load_unlocked(spec)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 677, in _load_unlocked
    spec.loader.exec_module(module)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 819, in exec_module
    exec(code, module.__dict__)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\Pillow-6.2.1-cp37-cp37m-win_amd64.whl\PIL\JpegImagePlugin.py", line 42, in <module>
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 1035, in _handle_fromlist
    _call_with_frames_removed(import_, from_name)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 219, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 983, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 967, in _find_and_load_unlocked
    module = _load_unlocked(spec)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 677, in _load_unlocked
    spec.loader.exec_module(module)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 819, in exec_module
    exec(code, module.__dict__)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\Pillow-6.2.1-cp37-cp37m-win_amd64.whl\PIL\TiffImagePlugin.py", line 44, in <module>
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 983, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 953, in _find_and_load_unlocked
    _call_with_frames_removed(import_, parent)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 219, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 983, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 967, in _find_and_load_unlocked
    module = _load_unlocked(spec)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 677, in _load_unlocked
    spec.loader.exec_module(module)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 819, in exec_module
    exec(code, module.__dict__)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\distutils\__init__.py", line 35, in <module>
    loader.exec_module(real_distutils)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap_external.py", line 724, in exec_module
    code = self.get_code(module.__name__)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap_external.py", line 859, in get_code
    source_bytes = self.get_data(source_path)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap_external.py", line 916, in get_data
    with _io.FileIO(path, 'r') as file:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\tc\\Downloads\\p3dtest\\p3dtest\\dist\\POC\\distutils\\__init__.py'
:task(error): Exception occurred in PythonTask eventManager
Traceback (most recent call last):
  File "run_game.py", line 4, in <module>
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\ShowBase.py", line 3124, in run
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\task\Task.py", line 531, in run
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\task\Task.py", line 485, in step
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\EventManager.py", line 49, in eventLoopTask
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\EventManager.py", line 43, in doEvents
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\EventManager.py", line 99, in processEvent
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\Messenger.py", line 333, in send
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\panda3d-1.10.4.1+opt-cp37-cp37m-win_amd64.whl\direct\showbase\Messenger.py", line 418, in __dispatch
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\game\main.py", line 11, in dosomething
    img.save('test.png')
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\Pillow-6.2.1-cp37-cp37m-win_amd64.whl\PIL\Image.py", line 2056, in save
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\Pillow-6.2.1-cp37-cp37m-win_amd64.whl\PIL\Image.py", line 388, in preinit
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 1035, in _handle_fromlist
    _call_with_frames_removed(import_, from_name)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 219, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 983, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 967, in _find_and_load_unlocked
    module = _load_unlocked(spec)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 677, in _load_unlocked
    spec.loader.exec_module(module)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 819, in exec_module
    exec(code, module.__dict__)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\Pillow-6.2.1-cp37-cp37m-win_amd64.whl\PIL\JpegImagePlugin.py", line 42, in <module>
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 1035, in _handle_fromlist
    _call_with_frames_removed(import_, from_name)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 219, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 983, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 967, in _find_and_load_unlocked
    module = _load_unlocked(spec)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 677, in _load_unlocked
    spec.loader.exec_module(module)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 819, in exec_module
    exec(code, module.__dict__)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\build\__whl_cache__\Pillow-6.2.1-cp37-cp37m-win_amd64.whl\PIL\TiffImagePlugin.py", line 44, in <module>
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 983, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 953, in _find_and_load_unlocked
    _call_with_frames_removed(import_, parent)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 219, in _call_with_frames_removed
    return f(*args, **kwds)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 983, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 967, in _find_and_load_unlocked
    module = _load_unlocked(spec)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 677, in _load_unlocked
    spec.loader.exec_module(module)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap.py", line 819, in exec_module
    exec(code, module.__dict__)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\distutils\__init__.py", line 35, in <module>
    loader.exec_module(real_distutils)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap_external.py", line 724, in exec_module
    code = self.get_code(module.__name__)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap_external.py", line 859, in get_code
    source_bytes = self.get_data(source_path)
  File "C:\Users\tc\Downloads\p3dtest\p3dtest\venv\lib\importlib\_bootstrap_external.py", line 916, in get_data
    with _io.FileIO(path, 'r') as file:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\tc\\Downloads\\p3dtest\\p3dtest\\dist\\POC\\distutils\\__init__.py'
@rdb rdb added the windows label Dec 8, 2019
@rdb

This comment has been minimized.

Copy link
Member

@rdb rdb commented Dec 8, 2019

I finally managed to reproduce the error after realising that I made the false assumption that the built-in venv was used, instead of the virtualenv package which apparently exists on PyPI and is still available for Python 3.

Apparently, this virtualenv package inserts a custom version of distutils/__init__.py that does some really bizarre monkey patching before trying to find the original distutils (which no longer exists in our deployed build). Worse, it expects the original distutils to exist in source form.

Based on this I strongly recommend using the built-in venv instead of the virtualenv on PyPI. But, perhaps we can detect that virtualenv is being used in build_apps, and then apply a hack like this.

@rdb rdb added this to the 1.10.5 milestone Dec 8, 2019
@tcdude

This comment has been minimized.

Copy link
Contributor Author

@tcdude tcdude commented Dec 8, 2019

This is good enough for me. Knowing that it behaves correctly when using the built-in venv is acceptable, since it implies only a slight change in my workflow. I think it would be good to add a note/warning in the documentation, that virtualenvcan lead to problems, such as this.

Edit:
Maybe a better addition to the documentation could be to advise people, that deployment is tested and known to work with either the included venv module in Python or on a global interpreter level and other implementations of virtual environments could lead to unforeseen problems?

Thank you for your time and energy.

@rdb rdb closed this in b43b0c9 Dec 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.