New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`python -m venv` does not create python3.X symlink (but using python3.6 does) #206

Open
tmoschou opened this Issue Apr 12, 2017 · 9 comments

Comments

Projects
None yet
7 participants
@tmoschou

tmoschou commented Apr 12, 2017

Creating a CPython 3.5.3 virtual environment via pyenv-virtualenv does not create a python3.5 symlink. However this is not a problem if I create the virtual env manually using python3.5 -m venv.

Steps to reproduce

/tmp/test $ pyenv install 3.5.3
/tmp/test $ pyenv virtualenv 3.5.3 my-pyenv-venv
/tmp/test $ pyenv local my-pyenv-venv
(my-pyenv-venv) /tmp/test $ pyenv which python3.5
pyenv: python3.5: command not found

The `python3.5' command exists in these Python versions:
  3.5.3

Where as

/tmp/test $ pyenv install 3.5.3
/tmp/test $ pyenv local 3.5.3
/tmp/test $ python3.5 -m venv my-venv
/tmp/test $ source my-venv/bin/activate
(my-venv) /tmp/test $ which python3.5
/tmp/test/my-venv/bin/python3.5

@blueyed blueyed changed the title from Creating a python 3.5.3 virtual env causes 'python3.5: command not found' to python3.5 symlink not created for virtualenv Apr 19, 2017

@blueyed

This comment has been minimized.

Show comment
Hide comment
@blueyed

blueyed Apr 19, 2017

Collaborator

That seems to be caused by python -m venv venv not creating the symlink(s), whereas python3.5 -m venv venv does?! - (i.e. it depends on the shim's name being used)

In your second example, using python instead of python3.5 should use the same Python, but will not create the symlink.

Collaborator

blueyed commented Apr 19, 2017

That seems to be caused by python -m venv venv not creating the symlink(s), whereas python3.5 -m venv venv does?! - (i.e. it depends on the shim's name being used)

In your second example, using python instead of python3.5 should use the same Python, but will not create the symlink.

@blueyed blueyed added the bug label Apr 19, 2017

@blueyed

This comment has been minimized.

Show comment
Hide comment
@blueyed

blueyed Apr 19, 2017

Collaborator

It's related to Python itself apparently: /usr/bin/python3.6 -m venv venv36 creates venv36/bin/python3.6, while using /usr/bin/python3 or /usr/bin/python does not do so.
(using Python 3.6.0 on Arch Linux, installed as /usr/bin/python3.6, and python/python3 being symlinks to it.

Collaborator

blueyed commented Apr 19, 2017

It's related to Python itself apparently: /usr/bin/python3.6 -m venv venv36 creates venv36/bin/python3.6, while using /usr/bin/python3 or /usr/bin/python does not do so.
(using Python 3.6.0 on Arch Linux, installed as /usr/bin/python3.6, and python/python3 being symlinks to it.

@blueyed blueyed changed the title from python3.5 symlink not created for virtualenv to `python -m venv` does not create python3.X symlink (but using python3.6 does) Apr 19, 2017

@CMeza99

This comment has been minimized.

Show comment
Hide comment
@CMeza99

CMeza99 Apr 24, 2017

It does not make the symlink for 3.6 for me. I am trying to use 2.7, 3.4, 3.5, and 3.6. It only creates the symlink for 2.7.

Not sure if this is useful, but it does make the symlinks for the different versions of pip, i.e. pip3.4, pip3.5, etc.

CMeza99 commented Apr 24, 2017

It does not make the symlink for 3.6 for me. I am trying to use 2.7, 3.4, 3.5, and 3.6. It only creates the symlink for 2.7.

Not sure if this is useful, but it does make the symlinks for the different versions of pip, i.e. pip3.4, pip3.5, etc.

@alisianoi

This comment has been minimized.

Show comment
Hide comment
@alisianoi

alisianoi Jun 6, 2017

Wow, I was bitten very hard by this! This python3.6 symlink is actually very important if you are using pyenv together with tox and are following the documentation for tox that tells you to set envlist=py27,py36.

Also, if you do pyenv install 3.6.1 followed by pyenv global 3.6.1 followed by pyenv which python3.6, you will get the correct path of pyenv/versions/3.6.1/bin/python3.6. The same is true for pyenv install 2.7.13 -> pyenv global 2.7.12 -> pyenv which python2.7.

However, once you start using the virtualenv plugin with pyenv virtualenv 3.6.1 venv-3 and pyenv virtualenv 2.7.13 venv-2, then the python2.7 link exists in the venv-2 while at the same time python3.6 does not exist in venv-3

Anyway, thank you for your work on python-virtualenv, I hope you spend some time to fix this.

My os is Archlinux, can test/provide more details if required.

alisianoi commented Jun 6, 2017

Wow, I was bitten very hard by this! This python3.6 symlink is actually very important if you are using pyenv together with tox and are following the documentation for tox that tells you to set envlist=py27,py36.

Also, if you do pyenv install 3.6.1 followed by pyenv global 3.6.1 followed by pyenv which python3.6, you will get the correct path of pyenv/versions/3.6.1/bin/python3.6. The same is true for pyenv install 2.7.13 -> pyenv global 2.7.12 -> pyenv which python2.7.

However, once you start using the virtualenv plugin with pyenv virtualenv 3.6.1 venv-3 and pyenv virtualenv 2.7.13 venv-2, then the python2.7 link exists in the venv-2 while at the same time python3.6 does not exist in venv-3

Anyway, thank you for your work on python-virtualenv, I hope you spend some time to fix this.

My os is Archlinux, can test/provide more details if required.

@aldanor

This comment has been minimized.

Show comment
Hide comment
@aldanor

aldanor Oct 15, 2017

Still a problem for 3.6.2, wonder if anyone found any workarounds other than creating missing symlink manually?

aldanor commented Oct 15, 2017

Still a problem for 3.6.2, wonder if anyone found any workarounds other than creating missing symlink manually?

@gangefors

This comment has been minimized.

Show comment
Hide comment
@gangefors

gangefors Oct 30, 2017

@aldanor Does this work for you? It's still manual in the way that you have to supply an option, but it's better than doing ln -s ....

gangefors commented Oct 30, 2017

@aldanor Does this work for you? It's still manual in the way that you have to supply an option, but it's better than doing ln -s ....

@seliger

This comment has been minimized.

Show comment
Hide comment
@seliger

seliger Dec 28, 2017

This is apparently a behavior issue in the way venv works. Not only does this impact 3.6.x, I also confirmed that the same issue occurs with venv in general using the stock 3.5 Python that comes with Ubuntu 16.04.

The trouble is here. Starting at line 111:

        if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env:
            executable = os.environ['__PYVENV_LAUNCHER__']
        else:
            executable = sys.executable
        dirname, exename = os.path.split(os.path.abspath(executable))
        context.executable = executable

This carries down to the setup_python method starting at line 187. It takes whatever executable that is called as the source, and then creates the others based on a hard coded list (e.g. python and python3).

    def setup_python(self, context):
        """
        Set up a Python executable in the environment.
        :param context: The information for the environment creation request
                        being processed.
        """
        binpath = context.bin_path
        path = context.env_exe
        copier = self.symlink_or_copy
        copier(context.executable, path)
        dirname = context.python_dir
        if os.name != 'nt':
            if not os.path.islink(path):
                os.chmod(path, 0o755)
            for suffix in ('python', 'python3'):
                path = os.path.join(binpath, suffix)
                if not os.path.exists(path):
                    # Issue 18807: make copies if
                    # symlinks are not wanted
                    copier(context.env_exe, path, relative_symlinks_ok=True)
                    if not os.path.islink(path):
                        os.chmod(path, 0o755)
        else:
            subdir = 'DLLs'
            include = self.include_binary
            files = [f for f in os.listdir(dirname) if include(f)]
            for f in files:
                src = os.path.join(dirname, f)
                dst = os.path.join(binpath, f)
                if dst != context.env_exe:  # already done, above
                    copier(src, dst)
            dirname = os.path.join(dirname, subdir)
            if os.path.isdir(dirname):
                files = [f for f in os.listdir(dirname) if include(f)]
                for f in files:
                    src = os.path.join(dirname, f)
                    dst = os.path.join(binpath, f)
                    copier(src, dst)
            # copy init.tcl over
            for root, dirs, files in os.walk(context.python_dir):
                if 'init.tcl' in files:
                    tcldir = os.path.basename(root)
                    tcldir = os.path.join(context.env_dir, 'Lib', tcldir)
                    if not os.path.exists(tcldir):
                        os.makedirs(tcldir)
                    src = os.path.join(root, 'init.tcl')
                    dst = os.path.join(tcldir, 'init.tcl')
                    shutil.copyfile(src, dst)
                    break

The best assumption I can make is that they assume you'd call with with the fully qualified executable name (e.g. python3.6).

I searched https://bugs.python.org for anything related, but as of yet, I've not come up with anything conclusive.

The obvious way to work around the issue would be to force pyenv-virtualenv to use the fully qualified binary name every time. I am submitting a PR as a workaround. The PR makes the assumption that ${PYENV_VERSION} is available for pyenv_virtualenv to use (and testing thus far seems to be positive).

I am also considering filing a bug with the Python folks because the behavior for venv is incongruous.

seliger commented Dec 28, 2017

This is apparently a behavior issue in the way venv works. Not only does this impact 3.6.x, I also confirmed that the same issue occurs with venv in general using the stock 3.5 Python that comes with Ubuntu 16.04.

The trouble is here. Starting at line 111:

        if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env:
            executable = os.environ['__PYVENV_LAUNCHER__']
        else:
            executable = sys.executable
        dirname, exename = os.path.split(os.path.abspath(executable))
        context.executable = executable

This carries down to the setup_python method starting at line 187. It takes whatever executable that is called as the source, and then creates the others based on a hard coded list (e.g. python and python3).

    def setup_python(self, context):
        """
        Set up a Python executable in the environment.
        :param context: The information for the environment creation request
                        being processed.
        """
        binpath = context.bin_path
        path = context.env_exe
        copier = self.symlink_or_copy
        copier(context.executable, path)
        dirname = context.python_dir
        if os.name != 'nt':
            if not os.path.islink(path):
                os.chmod(path, 0o755)
            for suffix in ('python', 'python3'):
                path = os.path.join(binpath, suffix)
                if not os.path.exists(path):
                    # Issue 18807: make copies if
                    # symlinks are not wanted
                    copier(context.env_exe, path, relative_symlinks_ok=True)
                    if not os.path.islink(path):
                        os.chmod(path, 0o755)
        else:
            subdir = 'DLLs'
            include = self.include_binary
            files = [f for f in os.listdir(dirname) if include(f)]
            for f in files:
                src = os.path.join(dirname, f)
                dst = os.path.join(binpath, f)
                if dst != context.env_exe:  # already done, above
                    copier(src, dst)
            dirname = os.path.join(dirname, subdir)
            if os.path.isdir(dirname):
                files = [f for f in os.listdir(dirname) if include(f)]
                for f in files:
                    src = os.path.join(dirname, f)
                    dst = os.path.join(binpath, f)
                    copier(src, dst)
            # copy init.tcl over
            for root, dirs, files in os.walk(context.python_dir):
                if 'init.tcl' in files:
                    tcldir = os.path.basename(root)
                    tcldir = os.path.join(context.env_dir, 'Lib', tcldir)
                    if not os.path.exists(tcldir):
                        os.makedirs(tcldir)
                    src = os.path.join(root, 'init.tcl')
                    dst = os.path.join(tcldir, 'init.tcl')
                    shutil.copyfile(src, dst)
                    break

The best assumption I can make is that they assume you'd call with with the fully qualified executable name (e.g. python3.6).

I searched https://bugs.python.org for anything related, but as of yet, I've not come up with anything conclusive.

The obvious way to work around the issue would be to force pyenv-virtualenv to use the fully qualified binary name every time. I am submitting a PR as a workaround. The PR makes the assumption that ${PYENV_VERSION} is available for pyenv_virtualenv to use (and testing thus far seems to be positive).

I am also considering filing a bug with the Python folks because the behavior for venv is incongruous.

@seliger

This comment has been minimized.

Show comment
Hide comment
@seliger

seliger Dec 29, 2017

FWIW, I have also created this bug report upstream:

https://bugs.python.org/issue32444

seliger commented Dec 29, 2017

FWIW, I have also created this bug report upstream:

https://bugs.python.org/issue32444

@seliger

This comment has been minimized.

Show comment
Hide comment
@seliger

seliger Dec 29, 2017

Upstream confirmed that the way venv works right now is the intended behavior. With that said, I now propose that the workaround I suggested be accepted into mainline, if that is the intended behavior.

The question on the table now becomes: How should pyenv handle this use case? Should it lock you into a version that will not upgrade as upstream suggests below, or does further logic need to be included to specify how python is invoked with -m venv?

Another way to consider this: Which tool should pyenv use to generate the venv? Right now it appears to support conda, virtualenv, and venv. It looks like it wants to do the right thing and use the "most correct" choice, given the environment and command line parameters.

As stated in #202, it is possible to work around this issue by putting -p python3.6 on the command line, which forces pyenv to use virtualenv (at least it does for me because I have it installed at the system level). Is that "good enough" or is the intent here to allow the "most correct" choice (-m venv) to be used? If -m venv is the intended best path forward, it probably should be reworked so that the user's intent as to which binaries are laid down can be specified.


R. David Murray rdmurray@bitdance.com added the comment:

This is a feature that actually supports your use case, as well as the use cases of those who don't want to strap the python version: you get what you ask for. If you call venv with 'python3', you get a venv that will use your new python if you upgrade your python, but if you call it with python3.x, you will get a venv that will not upgrade, which is exactly what you want for your use case.

nosy: +r.david.murray
resolution: -> not a bug
stage: -> resolved
status: open -> closed
versions: +Python 3.7 -Python 3.5

seliger commented Dec 29, 2017

Upstream confirmed that the way venv works right now is the intended behavior. With that said, I now propose that the workaround I suggested be accepted into mainline, if that is the intended behavior.

The question on the table now becomes: How should pyenv handle this use case? Should it lock you into a version that will not upgrade as upstream suggests below, or does further logic need to be included to specify how python is invoked with -m venv?

Another way to consider this: Which tool should pyenv use to generate the venv? Right now it appears to support conda, virtualenv, and venv. It looks like it wants to do the right thing and use the "most correct" choice, given the environment and command line parameters.

As stated in #202, it is possible to work around this issue by putting -p python3.6 on the command line, which forces pyenv to use virtualenv (at least it does for me because I have it installed at the system level). Is that "good enough" or is the intent here to allow the "most correct" choice (-m venv) to be used? If -m venv is the intended best path forward, it probably should be reworked so that the user's intent as to which binaries are laid down can be specified.


R. David Murray rdmurray@bitdance.com added the comment:

This is a feature that actually supports your use case, as well as the use cases of those who don't want to strap the python version: you get what you ask for. If you call venv with 'python3', you get a venv that will use your new python if you upgrade your python, but if you call it with python3.x, you will get a venv that will not upgrade, which is exactly what you want for your use case.

nosy: +r.david.murray
resolution: -> not a bug
stage: -> resolved
status: open -> closed
versions: +Python 3.7 -Python 3.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment