Failing to create venv when passing --python version on linux

[Gitznik]

Describe the bug

It looks like since #1168 pipx is no longer able to create a virtual environment when passing --python (at least on linux and with pyenv managed versions). It still works fine if a venv already exists.

From my tests I assume it's caused by how os.path.realpath creates the file path to the python executable provided. I've only tested this with python versions being managed by pyenv.

which python3.9
-> /home/robert/.pyenv/shims/python3.9 # This is fine

python3.9 -c "import sys; print(sys.executable)"
-> /home/robert/.pyenv/versions/3.9.18/bin/python3.9 # This is fine

python3.9 -c "import os;print(os.path.realpath('python3.9'));"
-> /home/robert/private/code/pipx/python3.9 # This is wrong!

How to reproduce
pipx run --python=python3.11 pycowsay moo

(develop-3-12) ➜  pipx git:(main) ✗ pipx run --python=python3.9 --verbose pycowsay moo
pipx >(setup:853): pipx version is 0.1.dev661+g7d82745.d20240106
pipx >(setup:854): Default python interpreter is '/home/robert/private/code/pipx/.nox/develop-3-12/bin/python'
pipx >(run_package:156): venv location is /home/robert/.local/pipx/.cache/7c9b64e6b834c48
pipx >(package_name_from_spec:370): Determined package name: pycowsay
pipx >(package_name_from_spec:371): Package name determined in 0.0s
creating virtual environment...
pipx >(run_subprocess:168): running python3.9 -m venv --without-pip /home/robert/.local/pipx/.cache/7c9b64e6b834c48
Traceback (most recent call last):
  File "/home/robert/private/code/pipx/.nox/develop-3-12/bin/pipx", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/robert/private/code/pipx/src/pipx/main.py", line 914, in cli
    return run_pipx_command(parsed_pipx_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robert/private/code/pipx/src/pipx/main.py", line 197, in run_pipx_command
    commands.run(
  File "/home/robert/private/code/pipx/src/pipx/commands/run.py", line 200, in run
    run_package(
  File "/home/robert/private/code/pipx/src/pipx/commands/run.py", line 157, in run_package
    _download_and_run(
  File "/home/robert/private/code/pipx/src/pipx/commands/run.py", line 234, in _download_and_run
    venv.create_venv(venv_args, pip_args, override_shared)
  File "/home/robert/private/code/pipx/src/pipx/venv.py", line 160, in create_venv
    venv_process = run_subprocess(cmd + venv_args + [str(self.root)])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robert/private/code/pipx/src/pipx/util.py", line 179, in run_subprocess
    completed_process = subprocess.run(
                        ^^^^^^^^^^^^^^^
  File "/home/robert/.pyenv/versions/3.12.1/lib/python3.12/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robert/.pyenv/versions/3.12.1/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/robert/.pyenv/versions/3.12.1/lib/python3.12/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/robert/private/code/pipx/python3.9'

Expected behavior

Creating a venv and then printing pycowsay.

[robinbowes]

This is also failing in the same way on MacOS.

[Gitznik]

Out of interest @robinbowes and @danyeaw - is this happening for you on pyenv managed interpreters as well or on standalone interpreters? Trying to narrow the bug down so I can write a test for my fix.

[robinbowes]

Yea, I pretty much only use pyenv-managed interpreters.

[danyeaw]

@Gitznik It is happening on both. Here is a traceback from GitHub Actions with pipx by default installed on the runner with no pyenv:

https://github.com/gaphor/gaphor/actions/runs/7474993163/job/20342622526

[robinbowes]

Ah yes, I'm seeing this on GitHub actions too.

[Gitznik]

Awesome, thank you both for clarifying!

[ewjoachim]

Bitten by this too in GHA (but I reproduce locally via brew-installed pipx), and if I'm not mistaken, GHA will likely not release a version with an upgraded pipx until probably a few weeks.

I guess we need a workaround.
The shortest failing command is pipx install {actually anything} --python=python3.x.
As far as I can tell, the easiest workaround is pipx install {actually anything} --python=$(which python3.x).

Do you have anything better to suggest?

[ewjoachim]

I'm trying to see if I can come up with a fixing PR. It seems the culprit is 81c2fa5

When we pass --python=python3.x, this ends up being the 0th param of the command python3.x -m venv. The 2 new lines do os.path.realpath(cmd_str_list[0]) which turns it into {pwd}/python3.x which isn't found

[Gitznik]

I'm trying to see if I can come up with a fixing PR. It seems the culprit is 81c2fa5

When we pass --python=python3.x, this ends up being the 0th param of the command python3.x -m venv. The 2 new lines do os.path.realpath(cmd_str_list[0]) which turns it into {pwd}/python3.x which isn't found

Hi @ewjoachim , that is indeed the reason, and I implemented a fix that is currently awaiting review, see the linked PR.

[ewjoachim]

Oh, I missed that PR ! Sorry / congratulations for the PR :)

[robinbowes]

As far as I can tell, the easiest workaround is pipx install {actually anything} --python=$(which python3.x).

Do you have anything better to suggest?

$(command -v python3.x) is preferable.

[Gitznik]

Oh, I missed that PR ! Sorry / congratulations for the PR :)

Thanks and no worries, just wanted to spare you sinking more time into this 🙂