venv doesn't work on Windows when no venvlauncher executable present

[mingwandroid]

BPO	35644
Nosy	@pfmoore, @tjguk, @zware, @zooba, @mingwandroid

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-03-21.16:34:17.007>
created_at = <Date 2019-01-02.20:30:12.685>
labels = ['invalid', 'type-bug', '3.7', 'OS-windows']
title = "venv doesn't work on Windows when no venvlauncher executable present"
updated_at = <Date 2019-03-21.16:34:17.006>
user = 'https://github.com/mingwandroid'

bugs.python.org fields:

activity = <Date 2019-03-21.16:34:17.006>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2019-03-21.16:34:17.007>
closer = 'steve.dower'
components = ['Windows']
creation = <Date 2019-01-02.20:30:12.685>
creator = 'Ray Donnelly'
dependencies = []
files = []
hgrepos = []
issue_num = 35644
keywords = []
message_count = 9.0
messages = ['332892', '332894', '332895', '332897', '332913', '332914', '332943', '332952', '338557']
nosy_count = 5.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'Ray Donnelly']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35644'
versions = ['Python 3.7']

[mingwandroid]

Happy New Year!

I'm not sure if this is a misunderstanding on my part, a docs bug or a code bug.

At https://docs.python.org/3/library/venv.html we see:

"The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages"

and

"This will create the tutorial-env directory if it doesn’t exist, and also create directories inside it containing a copy of the Python interpreter, the standard library, and various supporting files."

However, when testing with https://www.python.org/ftp/python/3.7.2/python-3.7.2-amd64.exe I see no Python interpreter (nor DLL) in my venv directory:

python.exe -m venv %TEMP%\venv
%TEMP%\venv\Scripts\activate.bat
dir %TEMP%\venv

gives:

 Directory of C:\Users\RDONNE~1\AppData\Local\Temp\venv

02/01/2019  19:38    <DIR>          .
02/01/2019  19:38    <DIR>          ..
02/01/2019  19:38    <DIR>          Include
02/01/2019  19:38    <DIR>          Lib
02/01/2019  19:38               121 pyvenv.cfg
02/01/2019  19:38    <DIR>          Scripts
               1 File(s)            121 bytes
               5 Dir(s)  912,281,780,224 bytes free

pyvenv.cfg contains:

home = C:\Users\rdonnelly\AppData\Local\Programs\Python\Python37
include-system-site-packages = false
version = 3.7.2

Further to this, after activating, I do not see the venv directory in sys.path:

python -c "import sys; print(sys.path)"
['',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37\\python37.zip',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37\\DLLs',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37\\lib',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37',
 'C:\\Users\\rdonnelly\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages']

From past experience, the old virtualenv project would copy the interpreter and DLL across.

Any help here would be appreciated!

[mingwandroid]

I found the executable is in the Scripts directory, closing. The real issue I'm facing is on Anaconda Distribution's build of Python 3 which I'm updating to 3.7.2.

Closing,

Cheers!

[mingwandroid]

Bit of an update to this, I'm re-opening it as there appears to be a regression from Python 3.7.1 to 3.7.2 for the case when there is no venvlauncher.exe present (i.e. when there are no python{w,}.exes in Lib\venv\scripts\nt). The old code of copying sys.executable is no longer run (on Windows only).

Currently Anaconda Distribution's venv is done this way. Should we change it to use the same method as official CPython Windows releases?

Here is a diff I think we need to apply for now, if you feel it is reasonable to make a PR then I'm happy to do so.

$ diff -urN Lib/venv/__init__.py.orig Lib/venv/__init__.py
--- Lib/venv/__init__.py.orig   2019-01-02 20:56:45.015131800 +0000
+++ Lib/venv/__init__.py        2019-01-02 20:56:55.330130800 +0000
@@ -188,9 +188,9 @@
         binpath = context.bin_path
         path = context.env_exe
         copier = self.symlink_or_copy
+        copier(context.executable, path)
         dirname = context.python_dir
         if os.name != 'nt':
-            copier(context.executable, path)
             if not os.path.islink(path):
                 os.chmod(path, 0o755)
             for suffix in ('python', 'python3'):

[mingwandroid]

The commit that my patch modifies is: 1c3de54

Cheers.

[zooba]

That patch will break the case where the launcher *is* present, which is why it was changed. If you're sure the launcher won't be present under Scripts/nt, then that patch is okay. You'll probably need to add code to also copy python37.dll and vcruntime140.dll, since those are needed by the regular python.exe but not the launcher.

However, why not use the launcher? All it does is find pyvenv.cfg, read its home variable and launches the python.exe it finds there. If you copy your build of venvlauncher over (or even our build) it will do the same thing.

That said, I'm a little concerned about the sys.path in your first post. I assumed nobody had hit problems since there haven't been any complaints (I've been using the Store version exclusively since 3.7.2 was released), but there seems to be something wrong there. Perhaps it was due to your build?

[zooba]

The installed venv seems to be okay, so I'm going to assume it's something about your own build.

Now I realise I haven't ever tried virtualenv against 3.6 or later... no idea what state that's in.

[mingwandroid]

Thanks Steve, the sys.path value from the first comment can be discarded, it was running the wrong Python!

The 'old' mechanism (which my patch reverts to) does copy all the necessary DLLs already. I released builds with this patch now and venv works fine (tested with pyperformance which uses venv).

However, we are more than happy to switch to the venvlauncher method as not deviating from upstream unnecessarily is always a good thing!

Do you have any pointers about how to build venvlauncher? I'll try to schedule some time for that.

[zooba]

It should just build directly from venv[w]launcher.vcxproj, though you'll need to rename venv[w]launcher.exe to python[w].exe (otherwise they'd conflict in the build directory).

[zooba]

Doesn't seem to be anything to fix upstream here.