The Pex CLI should warn when it creates a PEX zip that requires zip64.

[jsirois]

The python zipimporter used to execute a zipapp __main__.py in the zip cannot handle zips that require zip64 extensions (see: https://bugs.python.org/issue32959). As such, if the Pex CLI can detect when it creates such a zip, it should warn if the detection is approximate or probably error if it is definite; otherwise the error later at runtime is a mysterious:

/path/to/python: can't find '__main__' module in '/path/of/PEX'

At some point this will be fixed (presumably) for some CPython versions, in which case the logic gets trickier. The absolute worst thing to do here would be to warn or error for a PEX that works with a version of Python where this is no longer a limitation!

This does raise the question of whether or not this is an issue for PyPy - no clue.

[jsirois]

It looks like this can be checked robustly and quickly like so (for Python 2.7 find_module needs to be used instead of find_spec):

$ ls -lh dist/src.python.main.whisper/main.pex
-rwxr-xr-x 1 jsirois jsirois 2.3G Sep 29 10:32 dist/src.python.main.whisper/main.pex
$ time /usr/bin/python3.11 -c 'from zipimport import zipimporter; zi = zipimporter("dist/src.python.main.whisper/main.pex"); print(zi.find_spec("__main__"))'
None

real    0m0.009s
user    0m0.000s
sys     0m0.009s
$ zip -d dist/src.python.main.whisper/main.pex .deps/torch-2.0.1-cp38-cp38-manylinux1_x86_64.whl/torch/lib/libtorch_cuda.so
deleting: .deps/torch-2.0.1-cp38-cp38-manylinux1_x86_64.whl/torch/lib/libtorch_cuda.so
        zip warning: Local Version Needed To Extract does not match CD: .deps/torch-2.0.1-cp38-cp38-manylinux1_x86_64.whl/torch/lib/libtorch_cuda_linalg.so
...
        zip warning: Local Version Needed To Extract does not match CD: PEX-INFO
        zip warning: Local Version Needed To Extract does not match CD: __main__.py
        zip warning: Local Version Needed To Extract does not match CD: __pex__/
        zip warning: Local Version Needed To Extract does not match CD: __pex__/__init__.py
        zip warning: Local Version Needed To Extract does not match CD: main/
        zip warning: Local Version Needed To Extract does not match CD: main/whisper/
        zip warning: Local Version Needed To Extract does not match CD: main/whisper/main.py
$ ls -lh dist/src.python.main.whisper/main.pex
-rwxr-xr-x 1 jsirois jsirois 1.9G Sep 29 10:45 dist/src.python.main.whisper/main.pex
$ time /usr/bin/python3.11 -c 'from zipimport import zipimporter; zi = zipimporter("dist/src.python.main.whisper/main.pex"); print(zi.find_spec("__main__"))'
ModuleSpec(name='__main__', loader=<zipimporter object "dist/src.python.main.whisper/main.pex/">, origin='dist/src.python.main.whisper/main.pex/__main__.py')

real    0m0.055s
user    0m0.046s
sys     0m0.009s

Here the huge PEX zip is the one from this Pants issue which is what originated this issue.

The remaining issue with this approach is it's only valid for the interpreter used to execute the test. The PEX file might be run under a different interpreter which may have support in zipimport for zip64 or vice-versa.

[jsirois]

Ok, and the full gamut. It looks like PyPy also doesn't do zip64 in zipimport. It quickly fails in a different way for pypy3.7 and older, but that can be accounted for:

$ ls -lh dist/src.python.main.whisper/main.pex
-rwxr-xr-x 1 jsirois jsirois 2.3G Sep 29 14:19 dist/src.python.main.whisper/main.pex
jsirois@Gill-Windows:~/support/pants/peachanG/pants-sample (whisper *) $ for py in python{2.7,3.{5,6,7,8,9,10,11}} pypy{2.7,3.{5,6,7,8,9,10}}; do echo "Using $($py -V) at $(command -v $py)" && time $py -c 'from zipimport import zipimporter; zi = zipimporter("dist/src.python.main.whisper/main.pex"); print(zi.find_module("__main__"))' && echo; done
Python 2.7.18
Using  at /home/jsirois/.pyenv/shims/python2.7
None

real    0m0.121s
user    0m0.043s
sys     0m0.020s                                                                                                                                                                                                                                                                                                                                                                                                                    

Using Python 3.5.10 at /home/jsirois/.pyenv/shims/python3.5
None

real    0m0.129s
user    0m0.072s
sys     0m0.006s

Using Python 3.6.15 at /home/jsirois/.pyenv/shims/python3.6
None

real    0m0.124s
user    0m0.057s
sys     0m0.017s

Using Python 3.7.17 at /home/jsirois/.pyenv/shims/python3.7
None

real    0m0.131s
user    0m0.072s
sys     0m0.012s

Using Python 3.8.18 at /home/jsirois/.pyenv/shims/python3.8
None

real    0m0.151s
user    0m0.066s
sys     0m0.029s

Using Python 3.9.18 at /home/jsirois/.pyenv/shims/python3.9
None

real    0m0.147s
user    0m0.080s
sys     0m0.020s

Using Python 3.10.12 at /home/jsirois/.pyenv/shims/python3.10
None

real    0m0.024s
user    0m0.024s
sys     0m0.000s

Using Python 3.11.0rc1 at /home/jsirois/.pyenv/shims/python3.11
None

real    0m0.027s
user    0m0.022s
sys     0m0.003s

Python 2.7.18 (8d509266596ab6a70defcf87c8b29f57b8e32426, Jun 15 2023, 10:18:34)
[PyPy 7.3.12 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
Using  at /home/jsirois/.pyenv/shims/pypy2.7
Traceback (most recent call last):
  File "<module>", line 1, in <module>
ZipImportError: dist/src.python.main.whisper/main.pex seems not to be a zipfile

real    0m0.224s
user    0m0.112s
sys     0m0.067s
Using Python 3.5.3 (928a4f70d3de7d17449456946154c5da6e600162, Feb 09 2019, 11:50:43)
[PyPy 7.0.0 with GCC 8.2.0] at /home/jsirois/.pyenv/shims/pypy3.5
Traceback (most recent call last):
  File "<string>", line 1, in <module>
zipimport.ZipImportError: 'dist/src.python.main.whisper/main.pex' seems not to be a zipfile

real    0m0.230s
user    0m0.144s
sys     0m0.045s
Using Python 3.6.12 (db1e853f94de, Nov 18 2020, 09:49:19)
[PyPy 7.3.3 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] at /home/jsirois/.pyenv/shims/pypy3.6
Traceback (most recent call last):
  File "<string>", line 1, in <module>
zipimport.ZipImportError: 'dist/src.python.main.whisper/main.pex' seems not to be a zipfile

real    0m0.290s
user    0m0.182s
sys     0m0.061s
Using Python 3.7.13 (7e0ae751533460d5f89f3ac48ce366d8642d1db5, Mar 29 2022, 06:03:31)
[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] at /home/jsirois/.pyenv/shims/pypy3.7
Traceback (most recent call last):
  File "<string>", line 1, in <module>
zipimport.ZipImportError: 'dist/src.python.main.whisper/main.pex' seems not to be a zipfile

real    0m0.226s
user    0m0.149s
sys     0m0.035s
Using Python 3.8.16 (a9dbdca6fc3286b0addd2240f11d97d8e8de187a, Dec 29 2022, 11:45:13)
[PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] at /home/jsirois/.pyenv/shims/pypy3.8
None

real    0m0.223s
user    0m0.160s
sys     0m0.022s

Using Python 3.9.17 (3f3f2298ddc56db44bbdb4551ce992d8e9401646, Jun 15 2023, 11:15:53)
[PyPy 7.3.12 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] at /home/jsirois/.pyenv/shims/pypy3.9
None

real    0m0.233s
user    0m0.162s
sys     0m0.026s

Using Python 3.10.12 (af44d0b8114cb82c40a07bb9ee9c1ca8a1b3688c, Jun 15 2023, 12:39:27)
[PyPy 7.3.12 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] at /home/jsirois/.pyenv/shims/pypy3.10
None

real    0m0.238s
user    0m0.177s
sys     0m0.021s

jsirois@Gill-Windows:~/support/pants/peachanG/pants-sample (whisper *) $ zip -d dist/src.python.main.whisper/main.pex .deps/torch-2.0.1-cp38-cp38-manylinux1_x86_64.whl/torch/lib/libtorch_cuda.so >/dev/null 2>&1
jsirois@Gill-Windows:~/support/pants/peachanG/pants-sample (whisper *) $ ls -lh dist/src.python.main.whisper/main.pex
-rwxr-xr-x 1 jsirois jsirois 1.9G Sep 29 14:21 dist/src.python.main.whisper/main.pex
jsirois@Gill-Windows:~/support/pants/peachanG/pants-sample (whisper *) $ for py in python{2.7,3.{5,6,7,8,9,10,11}} pypy{2.7,3.{5,6,7,8,9,10}}; do echo "Using $($py -V) at $(command -v $py)" && time $py -c 'from zipimport import zipimporter; zi = zipimporter("dist/src.python.main.whisper/main.pex"); print(zi.find_module("__main__"))' && echo; done
Python 2.7.18
Using  at /home/jsirois/.pyenv/shims/python2.7
<zipimporter object "dist/src.python.main.whisper/main.pex">

real    0m0.147s
user    0m0.052s
sys     0m0.038s

Using Python 3.5.10 at /home/jsirois/.pyenv/shims/python3.5
<zipimporter object "dist/src.python.main.whisper/main.pex">

real    0m0.197s
user    0m0.123s
sys     0m0.015s

Using Python 3.6.15 at /home/jsirois/.pyenv/shims/python3.6
<zipimporter object "dist/src.python.main.whisper/main.pex">

real    0m0.201s
user    0m0.141s
sys     0m0.010s

Using Python 3.7.17 at /home/jsirois/.pyenv/shims/python3.7
<zipimporter object "dist/src.python.main.whisper/main.pex">

real    0m0.215s
user    0m0.108s
sys     0m0.056s

Using Python 3.8.18 at /home/jsirois/.pyenv/shims/python3.8
<zipimporter object "dist/src.python.main.whisper/main.pex/">

real    0m0.266s
user    0m0.187s
sys     0m0.036s

Using Python 3.9.18 at /home/jsirois/.pyenv/shims/python3.9
<zipimporter object "dist/src.python.main.whisper/main.pex/">

real    0m0.272s
user    0m0.217s
sys     0m0.005s

Using Python 3.10.12 at /home/jsirois/.pyenv/shims/python3.10
<zipimporter object "dist/src.python.main.whisper/main.pex/">

real    0m0.108s
user    0m0.098s
sys     0m0.011s

Using Python 3.11.0rc1 at /home/jsirois/.pyenv/shims/python3.11
<zipimporter object "dist/src.python.main.whisper/main.pex/">

real    0m0.089s
user    0m0.070s
sys     0m0.019s

Python 2.7.18 (8d509266596ab6a70defcf87c8b29f57b8e32426, Jun 15 2023, 10:18:34)
[PyPy 7.3.12 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
Using  at /home/jsirois/.pyenv/shims/pypy2.7
<zipimporter object at 0x00007f3ae193d330>

real    0m0.261s
user    0m0.153s
sys     0m0.053s

Using Python 3.5.3 (928a4f70d3de7d17449456946154c5da6e600162, Feb 09 2019, 11:50:43)
[PyPy 7.0.0 with GCC 8.2.0] at /home/jsirois/.pyenv/shims/pypy3.5
<zipimporter object at 0x00007f7b0297b768>

real    0m0.237s
user    0m0.159s
sys     0m0.027s

Using Python 3.6.12 (db1e853f94de, Nov 18 2020, 09:49:19)
[PyPy 7.3.3 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] at /home/jsirois/.pyenv/shims/pypy3.6
<zipimporter object at 0x00007fda5193eb10>

real    0m0.240s
user    0m0.160s
sys     0m0.035s

Using Python 3.7.13 (7e0ae751533460d5f89f3ac48ce366d8642d1db5, Mar 29 2022, 06:03:31)
[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] at /home/jsirois/.pyenv/shims/pypy3.7
<zipimporter object at 0x00007fa06f3204f8>

real    0m0.244s
user    0m0.162s
sys     0m0.035s

Using Python 3.8.16 (a9dbdca6fc3286b0addd2240f11d97d8e8de187a, Dec 29 2022, 11:45:13)
[PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] at /home/jsirois/.pyenv/shims/pypy3.8
<zipimporter object "dist/src.python.main.whisper/main.pex/">

real    0m0.286s
user    0m0.190s
sys     0m0.041s

Using Python 3.9.17 (3f3f2298ddc56db44bbdb4551ce992d8e9401646, Jun 15 2023, 11:15:53)
[PyPy 7.3.12 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] at /home/jsirois/.pyenv/shims/pypy3.9
<zipimporter object "dist/src.python.main.whisper/main.pex/">

real    0m0.307s
user    0m0.214s
sys     0m0.039s

Using Python 3.10.12 (af44d0b8114cb82c40a07bb9ee9c1ca8a1b3688c, Jun 15 2023, 12:39:27)
[PyPy 7.3.12 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] at /home/jsirois/.pyenv/shims/pypy3.10
<zipimporter object "dist/src.python.main.whisper/main.pex/">

real    0m0.312s
user    0m0.205s
sys     0m0.051s