python -m venv does not create python3.X symlink (but using python3.6 does)

[tmoschou]

Creating a CPython 3.5.3 virtual environment via pyenv-virtualenv does not create a python3.5 symlink. However this is not a problem if I create the virtual env manually using python3.5 -m venv.

Steps to reproduce

/tmp/test $ pyenv install 3.5.3
/tmp/test $ pyenv virtualenv 3.5.3 my-pyenv-venv
/tmp/test $ pyenv local my-pyenv-venv
(my-pyenv-venv) /tmp/test $ pyenv which python3.5
pyenv: python3.5: command not found

The `python3.5' command exists in these Python versions:
  3.5.3

Where as

/tmp/test $ pyenv install 3.5.3
/tmp/test $ pyenv local 3.5.3
/tmp/test $ python3.5 -m venv my-venv
/tmp/test $ source my-venv/bin/activate
(my-venv) /tmp/test $ which python3.5
/tmp/test/my-venv/bin/python3.5

[blueyed]

That seems to be caused by python -m venv venv not creating the symlink(s), whereas python3.5 -m venv venv does?! - (i.e. it depends on the shim's name being used)

In your second example, using python instead of python3.5 should use the same Python, but will not create the symlink.

[blueyed]

It's related to Python itself apparently: /usr/bin/python3.6 -m venv venv36 creates venv36/bin/python3.6, while using /usr/bin/python3 or /usr/bin/python does not do so.
(using Python 3.6.0 on Arch Linux, installed as /usr/bin/python3.6, and python/python3 being symlinks to it.

[xsc27]

It does not make the symlink for 3.6 for me. I am trying to use 2.7, 3.4, 3.5, and 3.6. It only creates the symlink for 2.7.

Not sure if this is useful, but it does make the symlinks for the different versions of pip, i.e. pip3.4, pip3.5, etc.

[alisianoi]

Wow, I was bitten very hard by this! This python3.6 symlink is actually very important if you are using pyenv together with tox and are following the documentation for tox that tells you to set envlist=py27,py36.

Also, if you do pyenv install 3.6.1 followed by pyenv global 3.6.1 followed by pyenv which python3.6, you will get the correct path of pyenv/versions/3.6.1/bin/python3.6. The same is true for pyenv install 2.7.13 -> pyenv global 2.7.12 -> pyenv which python2.7.

However, once you start using the virtualenv plugin with pyenv virtualenv 3.6.1 venv-3 and pyenv virtualenv 2.7.13 venv-2, then the python2.7 link exists in the venv-2 while at the same time python3.6 does not exist in venv-3

Anyway, thank you for your work on python-virtualenv, I hope you spend some time to fix this.

My os is Archlinux, can test/provide more details if required.

[aldanor]

Still a problem for 3.6.2, wonder if anyone found any workarounds other than creating missing symlink manually?

[gangefors]

@aldanor Does this work for you? It's still manual in the way that you have to supply an option, but it's better than doing ln -s ....

[seliger]

This is apparently a behavior issue in the way venv works. Not only does this impact 3.6.x, I also confirmed that the same issue occurs with venv in general using the stock 3.5 Python that comes with Ubuntu 16.04.

The trouble is here. Starting at line 111:

        if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env:
            executable = os.environ['__PYVENV_LAUNCHER__']
        else:
            executable = sys.executable
        dirname, exename = os.path.split(os.path.abspath(executable))
        context.executable = executable

This carries down to the setup_python method starting at line 187. It takes whatever executable that is called as the source, and then creates the others based on a hard coded list (e.g. python and python3).

    def setup_python(self, context):
        """
        Set up a Python executable in the environment.
        :param context: The information for the environment creation request
                        being processed.
        """
        binpath = context.bin_path
        path = context.env_exe
        copier = self.symlink_or_copy
        copier(context.executable, path)
        dirname = context.python_dir
        if os.name != 'nt':
            if not os.path.islink(path):
                os.chmod(path, 0o755)
            for suffix in ('python', 'python3'):
                path = os.path.join(binpath, suffix)
                if not os.path.exists(path):
                    # Issue 18807: make copies if
                    # symlinks are not wanted
                    copier(context.env_exe, path, relative_symlinks_ok=True)
                    if not os.path.islink(path):
                        os.chmod(path, 0o755)
        else:
            subdir = 'DLLs'
            include = self.include_binary
            files = [f for f in os.listdir(dirname) if include(f)]
            for f in files:
                src = os.path.join(dirname, f)
                dst = os.path.join(binpath, f)
                if dst != context.env_exe:  # already done, above
                    copier(src, dst)
            dirname = os.path.join(dirname, subdir)
            if os.path.isdir(dirname):
                files = [f for f in os.listdir(dirname) if include(f)]
                for f in files:
                    src = os.path.join(dirname, f)
                    dst = os.path.join(binpath, f)
                    copier(src, dst)
            # copy init.tcl over
            for root, dirs, files in os.walk(context.python_dir):
                if 'init.tcl' in files:
                    tcldir = os.path.basename(root)
                    tcldir = os.path.join(context.env_dir, 'Lib', tcldir)
                    if not os.path.exists(tcldir):
                        os.makedirs(tcldir)
                    src = os.path.join(root, 'init.tcl')
                    dst = os.path.join(tcldir, 'init.tcl')
                    shutil.copyfile(src, dst)
                    break

The best assumption I can make is that they assume you'd call with with the fully qualified executable name (e.g. python3.6).

I searched https://bugs.python.org for anything related, but as of yet, I've not come up with anything conclusive.

The obvious way to work around the issue would be to force pyenv-virtualenv to use the fully qualified binary name every time. I am submitting a PR as a workaround. The PR makes the assumption that ${PYENV_VERSION} is available for pyenv_virtualenv to use (and testing thus far seems to be positive).

I am also considering filing a bug with the Python folks because the behavior for venv is incongruous.

[seliger]

FWIW, I have also created this bug report upstream:

https://bugs.python.org/issue32444

[seliger]

Upstream confirmed that the way venv works right now is the intended behavior. With that said, I now propose that the workaround I suggested be accepted into mainline, if that is the intended behavior.

The question on the table now becomes: How should pyenv handle this use case? Should it lock you into a version that will not upgrade as upstream suggests below, or does further logic need to be included to specify how python is invoked with -m venv?

Another way to consider this: Which tool should pyenv use to generate the venv? Right now it appears to support conda, virtualenv, and venv. It looks like it wants to do the right thing and use the "most correct" choice, given the environment and command line parameters.

As stated in #202, it is possible to work around this issue by putting -p python3.6 on the command line, which forces pyenv to use virtualenv (at least it does for me because I have it installed at the system level). Is that "good enough" or is the intent here to allow the "most correct" choice (-m venv) to be used? If -m venv is the intended best path forward, it probably should be reworked so that the user's intent as to which binaries are laid down can be specified.

---

R. David Murray rdmurray@bitdance.com added the comment:

This is a feature that actually supports your use case, as well as the use cases of those who don't want to strap the python version: you get what you ask for. If you call venv with 'python3', you get a venv that will use your new python if you upgrade your python, but if you call it with python3.x, you will get a venv that will not upgrade, which is exactly what you want for your use case.

nosy: +r.david.murray
resolution: -> not a bug
stage: -> resolved
status: open -> closed
versions: +Python 3.7 -Python 3.5