Python version changes depending on command line arguments

[more-ishy]

Too many issues will kill our team's development velocity, drastically.
Make sure you have checked all steps below.

Prerequisite

• Make sure your problem is not listed in the common build problems.
• Make sure no duplicated issue has already been reported in the pyenv issues. You should look for closed issues, too.
• Make sure you are not asking us to help solving your specific issue.
  • GitHub issues is opened mainly for development purposes. If you want to ask someone to help solving your problem, go to some community site like Gitter, StackOverflow, etc.
• Make sure your problem is not derived from packaging (e.g. Homebrew).
  • Please refer to the package documentation for the installation issues, etc.
• Make sure your problem is not derived from plugins.
  • This repository is maintaining pyenv and the default python-build plugin only. Please refrain from reporting issues of other plugins here.

Description

• Platform information (e.g. Ubuntu Linux 16.04): Amazon Linux 2 AMI (HVM), SSD Volume Type
• OS architecture (e.g. amd64): x86_64
• pyenv version: 1.2.21
• Python version: 2.7.18 and 3.7.6
• C Compiler information (e.g. gcc 7.3): 7.3.1 20180712 (Red Hat 7.3.1-9) (GCC)
• Please attach verbose build log as gist
  • You can turn on verbose debug logging using by setting PYENV_DEBUG=1, e.g. env PYENV_DEBUG=1 pyenv install -v 3.6.4

Issue

Python version changes depending on command line arguments.

[ec2-user@******** foo]$ python bar.py -a ./without-pv/baz.txt
3.7.6 (default, Dec  8 2020, 06:20:44)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)]

[ec2-user@******** foo]$ python bar.py -a ./with-pv/baz.txt
2.7.18 (default, Aug 27 2020, 21:22:52)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)]

Details

The bar.py is as below. That code displays python version by print(sys.version) regardless of the command line arguments.

# bar.py
import argparse
import sys

def main(arg1):
    print(sys.version)
    return()

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-a", "--arg1")
    args = parser.parse_args()
    main(args.arg1)

The directory structure is as below.


foo
├── .python-version
├── bar.py
├── with-pv
│   ├── .python-version
│   └── baz.txt
└── without-pv
    └── baz.txt

foo/.python-version is as below.

3.7.6

foo/with-pv/.python-version is as below.

system

system is Python 2.7.18.

foo/with-pv/a.txt is as below.

abcd

foo/without-pv/a.txt is as below.

abcd

The command history is as below.

[ec2-user@******** foo]$ cd with-pv/

[ec2-user@******** with-pv]$ pyenv local system

[ec2-user@******** with-pv]$ python -V
Python 2.7.18

[ec2-user@******** with-pv]$ cd ../

[ec2-user@******** foo]$ pyenv local 3.7.6

[ec2-user@******** foo]$ python -V
Python 3.7.6

[ec2-user@******** foo]$ python bar.py -a ./without-pv/baz.txt
3.7.6 (default, Dec  8 2020, 06:20:44)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)]

[ec2-user@******** foo]$ python bar.py -a ./with-pv/baz.txt
2.7.18 (default, Aug 27 2020, 21:22:52)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)]

Since we are executing the same code from the same directory, the results of the two commands should be the same. However, if the command line argument directory contains .python-version, the unexpected result is displayed.

The Japanese explanation is here.
https://qiita.com/more-ishy/items/d44f288381bb979751da

[native-api]

Found the reason. It's rbenv/rbenv@283e67b .

It's a feature (?) where if the shim is invoked with a script, the version to use is taken from the script's directory rather than the current one.
Moreover, it's undocumented.

The problem is, the shim code doesn't find the script parameter reliably. Moreover, due to different Python versions and implementations having different command line parameters, I seriously doubt this is even possible: we'll need to maintain version-specific procedures of extracting the script argument from the command line (not mentioning system whose version we don't even know or control).

As such, I propose to remove the feature of taking the version from the script's directory as fundamentally broken.

[native-api]

@anton-petrov Not too hasty? Without a PR? The bug is real. The immediate problem is that the shim detects the script argument erroneously. It thinks it's ./with-pv/baz.txt while it's actually bar.py.

[anton-petrov]

@native-api Don't you think this use case is a bit far-fetched? It looks strange enough to install the local version in the data directory (where are arguments stored) and then run python from the directory above. If, in your opinion, the bug is real... Hmm, let's try to fix it... But, I don't have any ideas on this yet again.

remove the feature of taking the version from the script's directory as fundamentally broken.
It seems to me that this issue should be discussed in detail by the community before making any hasty decisions about removing this functionality.

@yyuu What do you think?

[native-api]

It looks strange enough to install the local version in the data directory

It's only "data directory" in this sample demonstration. In real use, it can very well be a project directory with the file somewhere below it.

It seems to me that this issue should be discussed in detail by the community before making any hasty decisions about removing this functionality.

Agree. I'll create a separate issue for that.

[native-api]

I'll create a separate issue for that.

#1813

[anton-petrov]

@native-api in what cases we may need different python versions in one project? And again, this is bad practice.

[yyuu]

I personally have never find this behavior useful, but, so far have been maintaining to not break the compatibility with rbenv behavior. My speculation is, this feature could be brought to look up appropriate runtime setting of some command symlink installed in /usr/local/bin, etc., but, again, myself have never used this.

[native-api]

@native-api in what cases we may need different python versions in one project? And again, this is bad practice.

(correction) That's not it, it's not "different python versions in one project".
It's if you have multiple projects that use different versions, you want to use the project's version whenever you invoke scripts from that project -- even if you invoke them from outside that project and/or from another project.

This is better discussed in the linked topic where I even asked this question from the person who added the feature.

[anton-petrov]

#1814