No such file for interpreter's binary name

[stuhood]

With 2.1.12, @asherf encountered a case where pex detected an interpreter with a bad binary name:

pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /Users/asher/.pyenv/versions/3.8.2/bin/python3.8
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /Users/asher/.toolchain/homebrew/Cellar/python@3.8/3.8.4/bin/python3.7
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/bin/python3.7
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/bin/python3.7

The file /Users/asher/.toolchain/homebrew/Cellar/python@3.8/3.8.4/bin/python3.7 does not exist (python3.8 does, as might be expected).

This bug was reproduced in the context of a long lived pex_root, which @asherf can provide if need be.

[jsirois]

This is hard to understand since FWICT all interpreter resolution funnels through here:
https://github.com/pantsbuild/pex/blob/b51b340bea8a585a246997071a815d008986d6e3/pex/interpreter.py#L429-L433

IOW the path /Users/asher/.toolchain/homebrew/Cellar/python@3.8/3.8.4/bin/python3.7 has been normalized (via os.realpath) so we've read through all symlinks to a hard link and we've not raised InterpreterNotFound.

@asherf is there any chance you can re-run against the saved PEX_ROOT with -vvvvvvvvv (9 vs)?

[jsirois]

For example:

With Pex 2.1.12:

jsirois@gill ~/dev/pantsbuild/pex (master) $ git co v2.1.12 
HEAD is now at 2ed8ed9 Prepare the 2.1.12 release.
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ python -mpex --version
__main__.py 2.1.12

Using a fresh interpreter cache for sanity sake:

jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ python -mpex -vvv --pex-root=/tmp/pex-issues-1009 --interpreter-constraint='CPython>=3.6' -- -c '' 2>&1
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
pex: Hashing pex: 26.2ms
pex: Isolating pex: 51.3ms                                                                                    
pex:   Extracting pex to /tmp/pex-issues-1009/isolated/6bd62cbe8c9f1ae82b35f976967538908799cbd6: 51.1ms
pex: Executing: PYTHONPATH=/tmp/pex-issues-1009/isolated/6bd62cbe8c9f1ae82b35f976967538908799cbd6 /opt/pypy/bin/pypy -s -c import os
import sys

from pex.common import atomic_directory, safe_open
from pex.interpreter import PythonIdentity


encoded_identity = PythonIdentity.get().encode()
sys.stdout.write(encoded_identity)
with atomic_directory('/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/917a091d50cf40d0d7f4a0de5a935db28edbaf59/opt.pypy.bin.pypy') as cache_dir:
  if cache_dir:
    with safe_open(os.path.join(cache_dir, 'INTERP-INFO'), 'w') as fp:
      fp.write(encoded_identity)

...
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ python -mpex -vvv --pex-root=/tmp/pex-issues-1009 --interpreter-constraint='CPython>=3.6' -- -c '' 2>&1 | head -6
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.6
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.7
pex: Building pex :: Resolving distributions ([])

So here's a similar base case. That leaves these cached interpreters from the PATH scan:

jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ find ~/tmp/pex-issues-1009/interpreters -type f | while read ii; do echo "$ii:" && jq .binary $ii && echo; done
/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/718a50d6552a2112904130b42af2c59767214f73/usr.bin.python3.7/INTERP-INFO:
"/usr/bin/python3.7"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/718a50d6552a2112904130b42af2c59767214f73/usr.bin.python3.7m/INTERP-INFO:
"/usr/bin/python3.7m"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/917a091d50cf40d0d7f4a0de5a935db28edbaf59/opt.pypy.bin.pypy/INTERP-INFO:
"/opt/pypy/bin/pypy"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/c3d7ea9d48d046c47d558adc82ead9af0e885bcb/usr.bin.python3.6m/INTERP-INFO:
"/usr/bin/python3.6m"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/c3d7ea9d48d046c47d558adc82ead9af0e885bcb/usr.bin.python3.6/INTERP-INFO:
"/usr/bin/python3.6"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/3ffacb7e0e1938412e210a942e1036ec64ac9355/usr.bin.python3.4/INTERP-INFO:
"/usr/bin/python3.4"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/3ffacb7e0e1938412e210a942e1036ec64ac9355/usr.bin.python3.4m/INTERP-INFO:
"/usr/bin/python3.4m"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/2e8e8915ef45f0cf40098ab616d817fb3db5ebc1/usr.bin.python2.7/INTERP-INFO:
"/usr/bin/python2.7"

Now on to simulating a once present but now gone interpreter:

jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ sudo mv /usr/bin/python3.6 /usr/bin/disable-python3.6
[sudo] password for jsirois: 
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.6m
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.7
pex: Building pex :: Resolving distributions ([])
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ /usr/bin/python3.6m --version
Python 3.6.10

And really removing the 3.6 by removing its other hard link:

jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ sudo mv /usr/bin/python3.6m /usr/bin/disable-python3.6m
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ python -mpex -vvv --pex-root=/tmp/pex-issues-1009 --interpreter-constraint='CPython>=3.6' -- -c '' 2>&1 | head -6
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.7
pex: Building pex :: Resolving distributions ([])
pex: Building pex: 24.7ms

So it looks like the existence check is working in this mocked up simulation.

[stuhood]

@asherf : If you're still able to repro this, working with John on it would be helpful.

[asherf]

doesn't repo anymore since I reinstalled my homebrew env.

[jsirois]

I'm going to close this since its both not reproduceable and the exercise above seems to indicate interpreter caching is robust to this sort of thing.

[jsirois]

This is hard to understand since FWICT all interpreter resolution funnels through here:
https://github.com/pantsbuild/pex/blob/b51b340bea8a585a246997071a815d008986d6e3/pex/interpreter.py#L429-L433

IOW the path /Users/asher/.toolchain/homebrew/Cellar/python@3.8/3.8.4/bin/python3.7 has been normalized (via os.realpath) so we've read through all symlinks to a hard link and we've not raised InterpreterNotFound.

@asherf is there any chance you can re-run against the saved PEX_ROOT with -vvvvvvvvv (9 vs)?

OK, we've had another instance of this come up over in pantsbuild/pants#10855
I've re-inspected the code paths, and under the assumption sys.executable actually == the path of the currently running python interpreter, there seems to be no way this can happen. However:

• https://bugs.python.org/issue22490#msg283859
• bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch python/cpython#9516

I'll remove reliance on this assumption.