Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microsoft Store installed python (3.7 - Windows 10) based virtualenvs cannot access pyd DLLs #1362

Open
JoDoX opened this issue Jun 2, 2019 · 23 comments
Labels
bug

Comments

@JoDoX
Copy link

@JoDoX JoDoX commented Jun 2, 2019

  1. Install Python 3.7 from Microsoft Store
  2. get error on trying to create new virtualenv
Microsoft Windows [Version 10.0.18908.1000]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\jodox>mkdir \temp\example

C:\Users\jodox>cd \temp\example

C:\temp\example>virtualenv .virtualenv
Using base prefix 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0'
New python executable in C:\temp\example\.virtualenv\Scripts\python.exe
Command C:\temp\example\.vir...v\Scripts\python.exe -m pip config list had error code 1
Installing setuptools, pip, wheel...

  Complete output from command C:\temp\example\.vir...v\Scripts\python.exe - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_internal\__init__.py", line 19, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\__init__.py", line 8, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\connectionpool.py", line 7, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\socket.py", line 49, in <module>
    import _socket
ImportError: DLL load failed: Access is denied.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 13, in <module>
ImportError: cannot import name 'main' from 'pip' (C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\__init__.py)
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\Scripts\virtualenv.exe\__main__.py", line 9, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 831, in main
    symlink=options.symlink,
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 1123, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 973, in install_wheel
    _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 1060, in _install_wheel_with_search_dir
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=script)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 924, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command C:\temp\example\.vir...v\Scripts\python.exe - setuptools pip wheel failed with error code 1

C:\temp\example>pip list
Package          Version
---------------- ----------
certifi          2019.3.9
pipenv           2018.11.26
virtualenv       16.6.0
virtualenv-clone 0.5.3

C:\temp\example>
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 2, 2019

@zooba this feels like bad packaging on Windows store side 🤔?

@JoDoX

This comment has been minimized.

Copy link
Author

@JoDoX JoDoX commented Jun 2, 2019

@gaborbernat I think bug in virtualenv, because venv module work fine

Microsoft Windows [Version 10.0.18908.1000]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\jodox>cd \temp\example

(.venv) C:\temp\example>python3 -m venv .venv

C:\temp\example>.\.venv\Scripts\activate.bat

(.venv) C:\temp\example>python -V
Python 3.7.3

(.venv) C:\temp\example>python -c "import sys;print(sys.executable)"
C:\temp\example\.venv\Scripts\python.exe

(.venv) C:\temp\example>python -c "import socket"

(.venv) C:\temp\example>
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 2, 2019

Not really man. venv provisions the created venv with whatever you have bundled. Here importing the network stack fails (which virtualenv does to install latest pip/setuptools). Pass in a --no-download and virtualenv should also work.

@JoDoX

This comment has been minimized.

Copy link
Author

@JoDoX JoDoX commented Jun 2, 2019

@gaborbernat I don't want virtualenv, but pipenv use it and no have option to pass args

C:\temp\example>pipenv --help
Usage: pipenv [OPTIONS] COMMAND [ARGS]...

Options:
  --where             Output project home information.
  --venv              Output virtualenv information.
  --py                Output Python interpreter information.
  --envs              Output Environment Variable options.
  --rm                Remove the virtualenv.
  --bare              Minimal output.
  --completion        Output completion (to be eval'd).
  --man               Display manpage.
  --support           Output diagnostic information for use in GitHub issues.
  --site-packages     Enable site-packages for the virtualenv.  [env var:
                      PIPENV_SITE_PACKAGES]
  --python TEXT       Specify which version of Python virtualenv should use.
  --three / --two     Use Python 3/2 when creating virtualenv.
  --clear             Clears caches (pipenv, pip, and pip-tools).  [env var:
                      PIPENV_CLEAR]
  -v, --verbose       Verbose mode.
  --pypi-mirror TEXT  Specify a PyPI mirror.
  --version           Show the version and exit.
  -h, --help          Show this message and exit.


Usage Examples:
   Create a new project using Python 3.7, specifically:
   $ pipenv --python 3.7

   Remove project virtualenv (inferred from current directory):
   $ pipenv --rm

   Install all dependencies for a project (including dev):
   $ pipenv install --dev

   Create a lockfile containing pre-releases:
   $ pipenv lock --pre

   Show a graph of your installed dependencies:
   $ pipenv graph

   Check your installed dependencies for security vulnerabilities:
   $ pipenv check

   Install a local setup.py into your virtual environment/Pipfile:
   $ pipenv install -e .

   Use a lower-level pip command:
   $ pipenv run pip freeze

Commands:
  check      Checks for security vulnerabilities and against PEP 508 markers
             provided in Pipfile.
  clean      Uninstalls all packages not specified in Pipfile.lock.
  graph      Displays currently-installed dependency graph information.
  install    Installs provided packages and adds them to Pipfile, or (if no
             packages are given), installs all packages from Pipfile.
  lock       Generates Pipfile.lock.
  open       View a given module in your editor.
  run        Spawns a command installed into the virtualenv.
  shell      Spawns a shell within the virtualenv.
  sync       Installs all packages specified in Pipfile.lock.
  uninstall  Un-installs a provided package and removes it from Pipfile.
  update     Runs lock, then sync.


C:\temp\example>pipenv install
Creating a virtualenv for this project…
Pipfile: C:\temp\example\Pipfile
Using C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\python.exe (3.7.3) to create virtualenv…
[==  ] Creating virtual environment...Already using interpreter C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\python.exe
Using base prefix 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0'
New python executable in C:\Users\jodox\.virtualenvs\example-Q6QloNNM\Scripts\python.exe
Command C:\Users\jodox\.virt...M\Scripts\python.exe -m pip config list had error code 1
Installing setuptools, pip, wheel...

  Complete output from command C:\Users\jodox\.virt...M\Scripts\python.exe - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_internal\__init__.py", line 19, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\__init__.py", line 8, in <module>
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\connectionpool.py", line 7, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\socket.py", line 49, in <module>
    import _socket
ImportError: DLL load failed: Access is denied.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 13, in <module>
ImportError: cannot import name 'main' from 'pip' (C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\__init__.py)
----------------------------------------
...Installing setuptools, pip, wheel...done.

Failed creating virtual environment
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\cli\command.py", line 254, in install
[pipenv.exceptions.VirtualenvCreationException]:       editable_packages=state.installstate.editables,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\core.py", line 1741, in do_install
[pipenv.exceptions.VirtualenvCreationException]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\core.py", line 574, in ensure_project
[pipenv.exceptions.VirtualenvCreationException]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\core.py", line 506, in ensure_virtualenv
[pipenv.exceptions.VirtualenvCreationException]:       python=python, site_packages=site_packages, pypi_mirror=pypi_mirror
[pipenv.exceptions.VirtualenvCreationException]:   File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\pipenv\core.py", line 935, in do_create_virtualenv
[pipenv.exceptions.VirtualenvCreationException]:       extra=[crayons.blue("{0}".format(c.err)),]
[pipenv.exceptions.VirtualenvCreationException]: Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 2580, in <module>
    main()
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 831, in main
    symlink=options.symlink,
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 1123, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 973, in install_wheel
    _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 1060, in _install_wheel_with_search_dir
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=script)
  File "C:\Users\jodox\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv.py", line 924, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command C:\Users\jodox\.virt...M\Scripts\python.exe - setuptools pip wheel failed with error code 1

Failed to create virtual environment.

C:\temp\example>
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 2, 2019

It still boils down to a packaging issue of core, as _socket is not importable 🤔 Will need to validate this though.

@gaborbernat gaborbernat added the bug label Jun 6, 2019
@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 6, 2019

@pfmoore this seems pip related 🤔 but strangely manifests when called from virtualenv by putting the wheel on the python path 👍

Command C:\Users\traveler\gi...v\Scripts\python.exe -m pip config list had error code 1
Installing setuptools, pip, wheel...

  Complete output from command C:\Users\traveler\gi...v\Scripts\python.exe - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_internal\__init__.py", line 19, in <module>
  File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\__init__.py", line 8, in <module>
  File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\connectionpool.py", line 7, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\socket.py", line 49, in <module>
    import _socket
ImportError: DLL load failed: Access is denied.

Edit: NM, seems our new virtualenv doesn't have access to _socket, my bad 🤦‍♂

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 6, 2019

So trying to import _socket on a freshly created virtualenv fails with:

# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.cp37-win_amd64.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.py
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.pyw
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\_socket.pyc
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.cp37-win_amd64.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.pyd
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: Access is denied.

While doing the same from the installed python

�# trying C:\Users\traveler\git\bre\env\Scripts\_socket.cp37-win_amd64.pyd
# trying C:\Users\traveler\git\bre\env\Scripts\_socket.pyd
# trying C:\Users\traveler\git\bre\env\Scripts\_socket.py
# trying C:\Users\traveler\git\bre\env\Scripts\_socket.pyw
# trying C:\Users\traveler\git\bre\env\Scripts\_socket.pyc
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.cp37-win_amd64.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.pyd
# extension module '_socket' loaded from 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\\DLLs\\_socket.pyd'
# extension module '_socket' executed from 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\\DLLs\\_socket.pyd'
import '_socket' # <_frozen_importlib_external.ExtensionFileLoader object at 0x0000022BDBDF2390>

Now inspecting the access data for this file:

PS C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs> Get-Acl _socket.pyd | fl


Path   : Microsoft.PowerShell.Core\FileSystem::C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\_socket.pyd
Owner  : NT AUTHORITY\SYSTEM
Group  : NT AUTHORITY\SYSTEM
Access : BUILTIN\Users Allow  ReadAndExecute, Synchronize
         S-1-15-3-1715986413-2752779232-2586321702-1327295686-2285127808-2907678250-160211910 Allow  ReadAndExecute, Synchronize
         BUILTIN\Users Allow  Read, Synchronize
         NT SERVICE\TrustedInstaller Allow  FullControl
         S-1-15-3-1024-3635283841-2530182609-996808640-1887759898-3848208603-3313616867-983405619-2501854204 Allow  ReadAndExecute, Synchronize
         NT AUTHORITY\SYSTEM Allow  FullControl
         NT AUTHORITY\LOCAL SERVICE Allow  ReadAndExecute, Synchronize
         NT AUTHORITY\NETWORK SERVICE Allow  ReadAndExecute, Synchronize
         NT AUTHORITY\RESTRICTED Allow  ReadAndExecute, Synchronize
Audit  :
Sddl   : O:SYG:SYD:AI(XA;ID;0x1200a9;;;BU;(WIN://SYSAPPID Contains "PYTHONSOFTWAREFOUNDATION.PYTHON.3.7_QBZ5N2KFRA8P0"))(A;ID;0x1200a9;;;S-1-15-3-1715986413-2752779232-2586321702-1327295686-2285127808-2907678250-160211910)(A;ID;FR;;;BU)(A;ID;FA;;;S-1-5-80-956008885-3418522649-1831038044-1853292631-2271478464)
         (A;ID;0x1200a9;;;S-1-15-3-1024-3635283841-2530182609-996808640-1887759898-3848208603-3313616867-983405619-2501854204)(A;ID;FA;;;SY)(A;ID;0x1200a9;;;LS)(A;ID;0x1200a9;;;NS)(A;ID;0x1200a9;;;RC)

Looking at this I'm not sure what's this missing 🤔

@gaborbernat gaborbernat changed the title Not work on Windows 10 with Python 3.7 from Microsoft Store Microsoft Store installed python (3.7 - Windows 10) based virtualenvs cannot access pyd DLLs Jun 6, 2019
@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Jun 6, 2019

I've never used the Store distribution of Python. I'd suggest asking @zooba for help here, as he's the expert on that distribution.

@JoDoX Can you do a simple test? Start up Python, and run

import _socket
import socket

That should work with no errors. If you get an error, then the issue is with something in Python, rather than with virtualenv (or pip).

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 6, 2019

@pfmoore see my post above. Those commands work if starting it from the installed python. However, they fail from within the virtual environment python. Seems to be related somehow with the fact that the DLLs can be only accessed if run from the installed Python. Once you copy the executable it seems to lose the ability to access the upstream DLLs. Not sure given this the issue is virtualenv or Windows store packaging related.

Edit:

Creating a venv and looking at the startup log shows somehow venvs actually try loading the pyd from the original location. It actually loads everything from the original location. Both Include and Lib are empty. In virtualenv we copy these over for now.... Maybe there's a better way to create venvs than what we have now 🤔 Will need to investigate.

# C:\Users\traveler\git\venv\lib\encodings\__pycache__\__init__.cpython-37.pyc matches C:\Users\traveler\git\venv\lib\encodings\__init__.py
# code object from 'C:\\Users\\traveler\\git\\venv\\lib\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# trying C:\Users\traveler\git\venv\lib\codecs.cp37-win_amd64.pyd
# trying C:\Users\traveler\git\venv\lib\codecs.pyd
# trying C:\Users\traveler\git\venv\lib\codecs.py
# C:\Users\traveler\git\venv\lib\__pycache__\codecs.cpython-37.pyc matches C:\Users\traveler\git\venv\lib\codecs.py
# code object from 'C:\\Users\\traveler\\git\\venv\\lib\\__pycache__\\codecs.cpython-37.pyc'

vs

# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.cp37-win_amd64.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.pyd
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.py
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.pyw
# trying C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\DLLs\encodings.pyc
# C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\lib\encodings\__pycache__\__init__.
@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Jun 6, 2019

@gaborbernat Ah, sorry - I skimmed your post and misunderstood what you were saying. My bad.

I certainly wouldn't rule out that copying the Python executable might cause issues. We really need @zooba here, as I'm not aware of any developer-level docs on how the Store build works, and I'm not sufficiently familiar with the code to go diving into it.

@zooba

This comment has been minimized.

Copy link
Contributor

@zooba zooba commented Jun 6, 2019

Yeah, you can't copy the executable. Once it's outside the app package it won't have its context properly activated by the operating system.

You can either symlink it or copy the python.exe that's in venv/scripts/nt. It's a special copy of the py.exe launcher that will look for pyvenv.cfg to find the actual one to launch. (Or if you don't want to use that file, you can build another version of it that looks for your preferred config settings.)

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Jun 6, 2019

Once it's outside the app package it won't have its context properly activated by the operating system.

Somewhat offtopic, but can you point me at the code that makes this be the case (and/or the relevant MS docs on this)? I'd like to understand the store app better than I currently do (I basically understand nothing about MS Store apps at the moment), so I don't have to keep pinging you on this stuff ;-)

@zooba

This comment has been minimized.

Copy link
Contributor

@zooba zooba commented Jun 6, 2019

I'm not sure that this part is even documented, and if it is, it'll probably be deep inside the MSIX docs or possibly in some AppV specifications if you can find that (which is the older version of the isolation technology used).

The packages are meant to be used as apps, primarily, which means a single entry point (and prevent other apps from using our DLLs). So we can't LoadLibrary files in the package until we've verified that we belong to the package, and in the general case the only way to do that is to launch the original python.exe.

Oh, and to be clear, these are all deliberate, by-design OS restrictions. Python did not choose to make any of this happen, so there's no code in CPython for it. Though the parts of launcher.c hidden behind the VENV_REDIRECT define may be interesting (and unsupported, internal details...)

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Jun 6, 2019

Thanks. I did some brief digging, and https://docs.microsoft.com/en-us/windows/msix/desktop/desktop-to-uwp-behind-the-scenes looks like it might be relevant (although I haven't read it yet, I'm mostly posting this so I can find the link again later ;-))

@gaborbernat I suspect the "right" way for virtualenv to support store Python will be to copy the redirector fromvenv/scripts/nt, as @zooba suggested. But that's extremely close to the idea of using venv to do the environment creation, that I suggested a while back, and you weren't keen on because of the compatibility implications. I suspect that it would be even worse if we did it for the Store application, but not the "standard" Python with the same version. (Plus, I'm not sure that mixing the redirector with things like virtualenv's custom site.py would even work).

I'm happy to help out with coding, but I'll need your guidance on the approach you want to take.

(Oh, and @zooba, while I have your attention, how well does the store Python co-exist with a "normal" Python install? If I want to test this stuff, will I be better with a clean VM, to avoid interfering with my main Python install? Or can I install and uninstall the store Python with no side-effects?)

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 10, 2019

So I gave this a good thought. In the end, I think it makes sense to switch the core logic to:

  • if possible create the venv with the stdlib provided venv, patch stdlib created venv to offer the same guarantees our own,
  • otherwise, use our own logic.

A few places where stdlib provided might venv might not be present:

  • python 2,
  • distributions where the venv is packaged separately and not installed at the moment (such as ubuntu python3.7-venv). @pfmoore taught? We'll need to come up with a list of guarantees we offer at the moment to implement this.
@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Jun 10, 2019

I'd be a strong +1 on this. With Python 2 reaching EOL very soon, we're going to be in a situation where basically all supported versions of Python have venv (and those that don't, still have the mechanisms - pyvenv.cfg). So if we can avoid the hacks we currently use as much as possible, that would be a huge win (and it would also make compatibility between virtualenv and venv a non-issue :-))

The hard part is deciding what constitutes the list of guarantees we offer. I'm not actually sure that there are many - the only difficult one is the sys.real_prefix vs sys.base_prefix issue, and honestly I believe we should move to follow the stdlib on this in any case (with a suitable transition period).

@pradyunsg

This comment has been minimized.

Copy link
Member

@pradyunsg pradyunsg commented Jun 10, 2019

I concur with @pfmoore on this.

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 10, 2019

Just to be clear, Python 2 support is not planned to be dropped at least for the next 3 years. The new solution will just prefer to use stdlib venv where possible and fallback to builtin logic otherwise.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Jun 10, 2019

Understood. The existing process needs to remain so that Python 2 is supported, but if we can avoid it where core venv is available that's great.

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 10, 2019

Started a separate discussion in #1366 👍

@zooba

This comment has been minimized.

Copy link
Contributor

@zooba zooba commented Jun 17, 2019

(Oh, and @zooba, while I have your attention, how well does the store Python co-exist with a "normal" Python install? If I want to test this stuff, will I be better with a clean VM, to avoid interfering with my main Python install? Or can I install and uninstall the store Python with no side-effects?)

Sorry I missed this, @pfmoore (Github notifications don't always get my attention...).

You can install/uninstall the store Python with no side-effects - this is actually one of it's biggest features. (Though a downside of this is that it can't update the real user registry, and so PEP 514 isn't quite sufficient, since overwriting values would break the clean install/uninstall cycle.)

@gaborbernat

This comment has been minimized.

Copy link
Contributor

@gaborbernat gaborbernat commented Jun 18, 2019

I think we talked over Twitter that it would not be an issue of overwrite if per pep 514 we use a different company (e.g. Microsoft Store) instead of PythonCore. And then we just need to update py.exe to accept that too.

@neoOpus

This comment has been minimized.

Copy link

@neoOpus neoOpus commented Aug 15, 2019

using venv instead of virtualenv solves the issue for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.