Using --no-pypi --repo produces broken pex

[mikekap]

Compare:

$ pex -o /tmp/pex.pex ipywidgets==7.6.3
$ /tmp/pex.pex
Python 3.8.7 (default, Feb  3 2021, 07:09:08)
...

vs

$ pip wheel --wheel-dir /tmp/wheels/ ipywidgets==7.6.3
$ pex -o /tmp/pex.pex ipywidgets==7.6.3 --no-pypi --repo /tmp/wheels/
$ /tmp/pex.pex
Traceback (most recent call last):
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 476, in execute
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 132, in activate
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 119, in _activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 378, in activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 702, in _activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 609, in resolve
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /private/tmp/pex.pex.
Needed macosx_11_0_x86_64-cp-38-cp38 compatible dependencies for:
 1: pandocfilters>=1.4.1
    Required by:
      nbconvert 6.0.7
    But this pex had no 'pandocfilters' distributions.

The difference seems to be that the pypi build uses the wheel named pandocfilters-1.4.3-py3-none-any.whl while the --repo build ends up using the file named pandocfilters-1.4.3-cp38-none-any.whl.

Now I admit I don't quite know that this is the correct way to use --repo, but I would hope pex would at least raise the error when building the pex instead of at runtime.

[jsirois]

Your pex ... command is unadorned (you don't pick an interpreter). As such, pex runs with sys.executable which could be:

1. /use/bin/env python if you're using the Pex PEX from Pex releases.
2. The python of the environment pex is installed in.

Whichever, that interpreter is the one the built PEX file should work with. That interpreter may not be the same as the one pip runs with (you may be running both from the same venv but that is unknown at this point for the purposes of this bug report).

A few useful bits of information you could provide to fully debug this remotely:

1. The full contents of the wheels repo dir.
2. The output of pex-tools your.pex interpreter -avi4
3. The output of pex-tools your.pex info -i4

Thanks in advance.

[mikekap]

Hmm I'm not sure what you mean by unadorned. I've just reproduced this with the latest pex release, downloaded from github:

$ ls -l `which python3`
lrwxr-xr-x 1 mikekap admin 40 Feb 23 10:56 /usr/local/bin/python3 -> ../Cellar/python@3.8/3.8.7_2/bin/python3
$ wget -O ~/Downloads/pex https://github.com/pantsbuild/pex/releases/download/v2.1.42/pex
...
$ python3 ~/Downloads/pex -o /tmp/pex.pex ipywidgets==7.6.3
$ python3 /tmp/pex.pex
Python 3.8.7 (default, Feb  3 2021, 07:09:08)
...

vs

$ python3 -m pip wheel --wheel-dir /tmp/wheels/ ipywidgets==7.6.3
$ python3 ~/Downloads/pex -o /tmp/pex.pex ipywidgets==7.6.3 --no-pypi --repo /tmp/wheels/
$ python3 /tmp/pex.pex
Traceback (most recent call last):
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 483, in execute
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 139, in activate
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 126, in _activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 428, in activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 784, in _activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 608, in resolve
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 692, in resolve_dists
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /private/tmp/pex.pex.
Needed macosx_11_0_x86_64-cp-38-cp38 compatible dependencies for:
 1: pandocfilters>=1.4.1
    Required by:
      nbconvert 6.0.7
    But this pex had no 'pandocfilters' distributions.

Note the fishy thing here - a version of pandocfilters is definitely in the zip archive, but pex doesn't seem to see it.

FWIW, python2 breaks in even worse ways, so it seems like the pex is build for python3:

$ python2 /tmp/pex.pex
Traceback (most recent call last):
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 483, in execute
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 139, in activate
  File "/tmp/pex.pex/.bootstrap/pex/pex.py", line 126, in _activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 428, in activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 784, in _activate
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 608, in resolve
  File "/tmp/pex.pex/.bootstrap/pex/environment.py", line 695, in resolve_dists
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /private/tmp/pex.pex.
Needed macosx_10_16_x86_64-cp-27-cp27m compatible dependencies for:
 1: ipykernel>=4.5.1
    Required by:
      ipywidgets 7.6.3
    But this pex had no u'ipykernel' distributions.
...

Here's the extra debug output:

$ python3 -m pip --version
pip 20.3.3 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)
$ ls -lah /tmp/wheels/
total 21M
drwxr-xr-x 55 mikekap wheel 1.8K May  3 22:02 .
drwxrwxrwt 34 root    wheel 1.1K May  3 22:03 ..
-rw-r--r--  1 mikekap wheel 123K May  3 22:02 Jinja2-2.11.3-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  17K May  3 22:02 MarkupSafe-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl
-rw-r--r--  1 mikekap wheel 983K May  3 22:02 Pygments-2.9.0-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  12K May  3 22:02 Send2Trash-1.5.0-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 4.3K May  3 22:02 appnope-0.1.2-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  65K May  3 22:02 argon2_cffi-20.1.0-cp37-abi3-macosx_10_6_intel.whl
-rw-r--r--  1 mikekap wheel  19K May  3 22:02 async_generator-1.10-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  49K May  3 22:02 attrs-20.3.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  11K May  3 22:02 backcall-0.2.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 277K May  3 22:02 bleach-3.3.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 173K May  3 22:02 cffi-1.14.5-cp38-cp38-macosx_10_9_x86_64.whl
-rw-r--r--  1 mikekap wheel 8.7K May  3 22:02 decorator-5.0.7-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  26K May  3 22:02 defusedxml-0.7.1-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  11K May  3 22:02 entrypoints-0.3-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 119K May  3 22:02 ipykernel-5.5.3-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 767K May  3 22:02 ipython-7.23.0-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  26K May  3 22:02 ipython_genutils-0.2.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 119K May  3 22:02 ipywidgets-7.6.3-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 1.4M May  3 22:02 jedi-0.18.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  55K May  3 22:02 jsonschema-3.2.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 111K May  3 22:02 jupyter_client-6.1.12-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  81K May  3 22:02 jupyter_core-4.7.1-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 4.6K May  3 22:02 jupyterlab_pygments-0.1.2-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 238K May  3 22:02 jupyterlab_widgets-1.0.0-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 8.1K May  3 22:02 matplotlib_inline-0.1.2-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  16K May  3 22:02 mistune-0.8.4-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  81K May  3 22:02 nbclient-0.5.3-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 540K May  3 22:02 nbconvert-6.0.7-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 175K May  3 22:02 nbformat-5.1.3-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 5.0K May  3 22:02 nest_asyncio-1.5.1-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 9.1M May  3 22:02 notebook-6.3.0-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  40K May  3 22:02 packaging-20.9-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 7.9K May  3 22:02 pandocfilters-1.4.3-cp38-none-any.whl
-rw-r--r--  1 mikekap wheel  92K May  3 22:02 parso-0.8.2-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  58K May  3 22:02 pexpect-4.8.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 6.8K May  3 22:02 pickleshare-0.7.5-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  55K May  3 22:02 prometheus_client-0.10.1-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 360K May  3 22:02 prompt_toolkit-3.0.18-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  14K May  3 22:02 ptyprocess-0.7.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 110K May  3 22:02 pycparser-2.20-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  67K May  3 22:02 pyparsing-2.4.7-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  68K May  3 22:02 pyrsistent-0.17.3-cp38-cp38-macosx_11_0_x86_64.whl
-rw-r--r--  1 mikekap wheel 222K May  3 22:02 python_dateutil-2.8.1-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 1.1M May  3 22:02 pyzmq-22.0.3-cp38-cp38-macosx_10_9_x86_64.whl
-rw-r--r--  1 mikekap wheel 767K May  3 22:02 setuptools-56.0.0-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  11K May  3 22:02 six-1.15.0-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  15K May  3 22:02 terminado-0.9.4-py3-none-any.whl
-rw-r--r--  1 mikekap wheel 161K May  3 22:02 testpath-0.4.4-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 407K May  3 22:02 tornado-6.1-cp38-cp38-macosx_10_9_x86_64.whl
-rw-r--r--  1 mikekap wheel  98K May  3 22:02 traitlets-5.0.5-py3-none-any.whl
-rw-r--r--  1 mikekap wheel  31K May  3 22:02 wcwidth-0.2.5-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel  12K May  3 22:02 webencodings-0.5.1-py2.py3-none-any.whl
-rw-r--r--  1 mikekap wheel 2.2M May  3 22:02 widgetsnbextension-3.5.1-py2.py3-none-any.whl

$ PEX_TOOLS=1 python3 ~/Downloads/pex interpreter -avi4
{
    "path": "/usr/local/Cellar/python@3.8/3.8.7_2/Frameworks/Python.framework/Versions/3.8/bin/python3.8",
    "requirement": "CPython==3.8.7",
    "platform": "macosx_11_0_x86_64-cp-38-cp38"
}
{
    "path": "/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7",
    "requirement": "CPython==2.7.17",
    "platform": "macosx_10_16_x86_64-cp-27-cp27m"
}
{
    "path": "/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python",
    "requirement": "CPython==2.7.16",
    "platform": "macosx_10_16_x86_64-cp-27-cp27m"
}
{
    "path": "/usr/bin/python3",
    "requirement": "CPython==3.8.2",
    "platform": "macosx_11_0_x86_64-cp-38-cp38"
}

$ PEX_TOOLS=1 python3 ~/Downloads/pex info -i4
{
    "always_write_cache": false,
    "build_properties": {
        "class": "CPython",
        "pex_version": "2.1.42",
        "platform": "manylinux_2_31_x86_64",
        "version": [
            2,
            7,
            18
        ]
    },
    "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
    "distributions": {
        "pex-2.1.42-py2.py3-none-any.whl": "1735869cad92e2a5e75d7e114b20ca4a1a805444"
    },
    "emit_warnings": true,
    "entry_point": "pex.bin.pex:main",
    "ignore_errors": false,
    "inherit_path": "false",
    "interpreter_constraints": [
        ">=2.7,<3.10,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*"
    ],
    "pex_hash": "478cc1fa371ca40aa3e7dafee735ca438d4a243f",
    "pex_path": null,
    "requirements": [
        "pex==2.1.42"
    ],
    "strip_pex_env": false,
    "unzip": true,
    "venv": false,
    "venv_bin_path": "False",
    "venv_copies": false,
    "zip_safe": true,
    "pex_root": "/Users/mikekap/.pex"
}

[mikekap]

Digging around, this might be related to changes in pip wheel - specifically, this magically fixes itself on the latest pip for me. The issue seems to be related to pypa/pip#7296 & the way I'm running the pip wheel command with an ages-old cache. I should mention -- pip fixed this by just ignoring old cache entries: pypa/pip#7502 . Not sure if pex wants to fix the issue of not failing at build-time - if you don't feel free to close the issue.

[jsirois]

Thanks for the debug output, but I need that for the PEX file you created, not for the Pex PEX file.

Using your latest debug output as a guide, you'd want to add ... -o /tmp/pex.pex --include-tools ... and then run the two tools commands as PEX_TOOLS=1 /tmp/pex.pex ....

[mikekap]

Ah got it. Here you go:

$ PEX_TOOLS=1 /tmp/pex.pex interpreter -avi4
{
    "path": "/usr/local/Cellar/python@3.8/3.8.7_2/Frameworks/Python.framework/Versions/3.8/bin/python3.8",
    "requirement": "CPython==3.8.7",
    "platform": "macosx_11_0_x86_64-cp-38-cp38"
}
{
    "path": "/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7",
    "requirement": "CPython==2.7.17",
    "platform": "macosx_10_16_x86_64-cp-27-cp27m"
}
{
    "path": "/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python",
    "requirement": "CPython==2.7.16",
    "platform": "macosx_10_16_x86_64-cp-27-cp27m"
}
{
    "path": "/usr/bin/python3",
    "requirement": "CPython==3.8.2",
    "platform": "macosx_11_0_x86_64-cp-38-cp38"
}

$ PEX_TOOLS=1 /tmp/pex.pex info -i4
{
    "always_write_cache": false,
    "build_properties": {
        "class": "CPython",
        "pex_version": "2.1.30",
        "platform": "macosx_11_0_x86_64",
        "version": [
            3,
            8,
            7
        ]
    },
    "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
    "distributions": {
        "Jinja2-2.11.3-py2.py3-none-any.whl": "b489ed91ce31b32a3cb5b0c124174d0ff4a8d5f5",
        "MarkupSafe-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl": "9aa4e66c9a8e97c1f535fe3c0eadd3fc6eac9678",
        "Pygments-2.9.0-py3-none-any.whl": "57c0fc25a45d7a3c3299af71a09fa7a2265b83cc",
        "Send2Trash-1.5.0-py3-none-any.whl": "4a199db61239ed56b667bab8e320be48987f8fab",
        "appnope-0.1.2-py2.py3-none-any.whl": "0b479e0098363ec16fc2917c0332a8a893df6db8",
        "argon2_cffi-20.1.0-cp37-abi3-macosx_10_6_intel.whl": "985f670d3d06bc95a4c71d70e8a3223425cd3a4e",
        "async_generator-1.10-py3-none-any.whl": "1893d1fedb102e7eac1042e18d7627065445a0bb",
        "attrs-20.3.0-py2.py3-none-any.whl": "ca98eece9f600f5ccd365a3b9d2b904554e62cc3",
        "backcall-0.2.0-py2.py3-none-any.whl": "b8dc826b2d1e32d7aa2aa1efda3e00d810f903b1",
        "bleach-3.3.0-py2.py3-none-any.whl": "1eed580e871de0510a72e393e80a7956afb18d97",
        "cffi-1.14.5-cp38-cp38-macosx_10_9_x86_64.whl": "214617323fca19ca5ac775b16fd8fdeef7758693",
        "decorator-5.0.7-py3-none-any.whl": "b385837db97c0ab2940e1bdbf89b934d8ae0140c",
        "defusedxml-0.7.1-py2.py3-none-any.whl": "09fdb4c52e276ebd650b39fd9335c24d61cb83bd",
        "entrypoints-0.3-py2.py3-none-any.whl": "73a2b88ba1d70ff51e9d5127bf2ca30b2fdb74fc",
        "ipykernel-5.5.3-py3-none-any.whl": "fa950c7cca1be89d01c0471e4d3f4478e9637498",
        "ipython-7.23.0-py3-none-any.whl": "227803bd2f270f758d3b9c43d48f6dfefc5a909f",
        "ipython_genutils-0.2.0-py2.py3-none-any.whl": "5d303944908df963b49329429699943ca79343ae",
        "ipywidgets-7.6.3-py2.py3-none-any.whl": "be811ded8398e58a212f317f282bc9bb67a90011",
        "jedi-0.18.0-py2.py3-none-any.whl": "2e96fdddb6a797151cc0c93a66b14a04b44ad4cd",
        "jsonschema-3.2.0-py2.py3-none-any.whl": "84bd25e11216ba0aa76ff0de72e4c9f51c7a6131",
        "jupyter_client-6.1.12-py3-none-any.whl": "8364cffb8cd530b74ee34161a88dafc7528dca5d",
        "jupyter_core-4.7.1-py3-none-any.whl": "bbe5de265a13815ec17fd0c424f6902d5cd3d326",
        "jupyterlab_pygments-0.1.2-py2.py3-none-any.whl": "0b12ba245fd1d77f73996fd5a27cf94f21f86c9c",
        "jupyterlab_widgets-1.0.0-py3-none-any.whl": "0ed67ca79a51bec2c459e02c9752ec5127d45832",
        "matplotlib_inline-0.1.2-py3-none-any.whl": "e056e0d942c4116e741edbdb4175097ec4d84df8",
        "mistune-0.8.4-py2.py3-none-any.whl": "d9b7ea2ebcba4fbb132a22cc66b1dc53e8674f27",
        "nbclient-0.5.3-py3-none-any.whl": "f4e60b3868c8d5165816e4502e2547321019ab1a",
        "nbconvert-6.0.7-py3-none-any.whl": "7c320ca1c13cf8ca155881b35fa138a75405c8b1",
        "nbformat-5.1.3-py3-none-any.whl": "ed2ad630143680c71ddf08a3625822311714aeef",
        "nest_asyncio-1.5.1-py3-none-any.whl": "e96905b1ec8a40c5c5013ae8c70fda138d60dd30",
        "notebook-6.3.0-py3-none-any.whl": "4b68cb6d1aa61209ae876aecd68a5ef4aaf844e8",
        "packaging-20.9-py2.py3-none-any.whl": "e6b5b33bf2e793d453dcd60915ed748e0cdd7be6",
        "pandocfilters-1.4.3-cp38-none-any.whl": "aed8d08e0d9f67b3e15265d5ff6d921ee0aa6f01",
        "parso-0.8.2-py2.py3-none-any.whl": "e0a3bfa68f63fa664fa66f165e66699ed12efe0b",
        "pexpect-4.8.0-py2.py3-none-any.whl": "786305f22c0e2de2fe003a2be40c7b8a90452869",
        "pickleshare-0.7.5-py2.py3-none-any.whl": "82da9b25293d30545da71fe2fc4a5c07494c4dc4",
        "prometheus_client-0.10.1-py2.py3-none-any.whl": "340f918b21cbbb0e6a517b318da850d6ea336089",
        "prompt_toolkit-3.0.18-py3-none-any.whl": "33244fa49e9aab3eecd98f3626389a75bf4dc7ea",
        "ptyprocess-0.7.0-py2.py3-none-any.whl": "37ead8a201a6033d4deb212d315e4c31f1da5eb8",
        "pycparser-2.20-py2.py3-none-any.whl": "9ed1a9854e562d529ab6879c5ff6e0d3e67ab920",
        "pyparsing-2.4.7-py2.py3-none-any.whl": "43b8e4b389f12873bece319f846fcbd0d134764b",
        "pyrsistent-0.17.3-cp38-cp38-macosx_11_0_x86_64.whl": "fd7eef7b12e83b192a65ed172e306d8f28aaeae2",
        "python_dateutil-2.8.1-py2.py3-none-any.whl": "ba4aed8bd5c800f9ebabc761777bdb4d7f33de21",
        "pyzmq-22.0.3-cp38-cp38-macosx_10_9_x86_64.whl": "83fa65fe2abe329a1c7a402ebb1f41d1d9ea7fba",
        "setuptools-56.0.0-py3-none-any.whl": "42c76d94e59eac845f5efedffababdffd59a5cb5",
        "six-1.15.0-py2.py3-none-any.whl": "9f69e6e8c37d132b1b2c814af17f47732aec0ed4",
        "terminado-0.9.4-py3-none-any.whl": "206c84da5a0c84c5ea7cc84ba36843d548410721",
        "testpath-0.4.4-py2.py3-none-any.whl": "71a2013dc21e92104c8084ec5c31b840acdeaf1e",
        "tornado-6.1-cp38-cp38-macosx_10_9_x86_64.whl": "00d0e3fe817a57368c2c58b06d1f2c7c6a089c75",
        "traitlets-5.0.5-py3-none-any.whl": "1fc37b5d2c69c8353b8e580eaa78aed5ce6579b2",
        "wcwidth-0.2.5-py2.py3-none-any.whl": "0cdc7b37d2740537850ea57a1d76e29902198eaf",
        "webencodings-0.5.1-py2.py3-none-any.whl": "5db6aaa634fb71c6ee761df515e0aa2e37fb9ca9",
        "widgetsnbextension-3.5.1-py2.py3-none-any.whl": "8e10a903dd7a74a8d717e6b9725e2d84bdc402b0"
    },
    "emit_warnings": true,
    "ignore_errors": false,
    "inherit_path": "false",
    "interpreter_constraints": [],
    "pex_hash": "fbc8c893f158ce4a00d09c9e3fa9ce06f5dce98d",
    "pex_path": null,
    "requirements": [
        "ipywidgets==7.6.3"
    ],
    "strip_pex_env": true,
    "unzip": false,
    "venv": false,
    "venv_bin_path": "False",
    "zip_safe": true
}

If it helps - here's the: pex.pex (i renamed it to pex.zip so i can upload it)

[jsirois]

Thanks @mikekap and sorry for the delay. If you could add two more pieces of debugging information I think I'll have all I need to diagnose what happened here. If you can, the full output of these two commands:

1. /usr/local/Cellar/python@3.8/3.8.7_2/Frameworks/Python.framework/Versions/3.8/bin/pip debug -v
2. /usr/bin/pip3 debug -v

I'm not sure if those paths work, but the gist is: I'd like to see what the pip corresponding to each of your Python 3.8 interpreters reports for that interpreters "Compatible tags" (an O(100s) list at the bottom of the debug output).

To be clear about the problem you saw:

When Pex builds a PEX file it does fail fast if it cannot resolve all requirements for the interpreters and requirements you've selected. If you've selected no interpreter, it uses the default one pex is running under (you can control this with --python, --python-path and --interpreter-constraint.). So, if your PEX file builds but then later fails to run its for one of two reasons:

1. The interpreter selected at runtime matches the constraints set at build time, but the PEX does not contain distributions build for that interpreter.
2. The PEX runtime logic for selecting distributions from within the PEX file diverges from Pip.

The debug info above will help me look into 2. I suspect 1 is not the issue since you're building and running the PEX file on the same machine.

[jsirois]

Sorry - one more thing. In your PEX_TOOLS info output above, it shows "pex_version": "2.1.30",. In other bits of the conversation you used 2.1.42. I'll check if there are relevant bug fixes between 2.1.30 and 2.1.42, but can you confirm this problem persists using Pex 2.1.42?

[jsirois]

Hey @mikekap I'm going to close this since it looks like you've moved on. If you can provide the extra debug information I asked for in #1345 (comment) and #1345 (comment) just above though, I'll re-open and dig more.