Pants jupyter plugin fails to load pex file, and error message is unhelpful

[horuswoo]

Started seeing this error intermittently since yesterday (tried using the 0.0.3 version and still see the same error):

running this line
%pex_load tmp/ml_training_env.pex gives:

HTML(value='<style>.nb-console-output-BCoZQ { background-color: black;} .nb-console-output-BCoZQ pre { color: …
Accordion(children=(Output(layout=Layout(height='300px', overflow_y='scroll'), outputs=({'output_type': 'displ…
Scrubbing sys.path and sys.modules in preparation for pex bootstrap
sys.path contains 7 items, sys.modules contains 1213 keys
sys.path now contains 7 items, sys.modules now contains 1213 keys
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/plugin.py in _accordion_widget(self, title, height, collapsed)
    147         # Capture the output context.
    148         with outputter:
--> 149             yield expand, collapse, set_output_glyph
    150 
    151     def _stream_binary_build_with_output(

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/plugin.py in _bootstrap_pex(self, pex_path)
    295 
    296                     # Bootstrap pex.
--> 297                     for path in self._pex_manager.mount(pex_path):
    298                         self._display_line(f"added sys.path entry {path}\n")
    299             except Exception:

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/pex.py in mount(self, pex_to_mount)
    170                     )
    171                 venv_tmp = venv.parent / f"{venv.name}.{uuid4().hex}"
--> 172                 run_pex_tool(args=["venv", str(venv_tmp)])
    173                 venv_tmp.rename(venv)
    174 

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/pex.py in run_pex_tool(args, **subprocess_args)
    142                             env=env.create(PEX_INTERPRETER=1, PEX_PYTHON_PATH=sys.executable),
    143                             check=True,
--> 144                             **subprocess_args,
    145                         ).stdout
    146                         or b""

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    510         if check and retcode:
    511             raise CalledProcessError(retcode, process.args,
--> 512                                      output=stdout, stderr=stderr)
    513     return CompletedProcess(process.args, retcode, stdout, stderr)
    514 

CalledProcessError: Command '['/home/ec2-user/anaconda3/envs/mxnet_latest_p37/bin/python', '/home/ec2-user/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex', '-m', 'pex.tools', 'tmp/ml_training_env.pex', 'venv', '/home/ec2-user/.cache/pants_jupyter_plugin/pex/venvs/foo/ml_training_env.pex.bar']' returned non-zero exit status 1.

[jsirois]

@horuswoo can you paste the error you see from 0.0.3? It has to be slightly different. In particular the '/home/ec2-user/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex' bit, which is not a version of Pex used by 0.0.3.

[jsirois]

Also @horuswoo can you give more context on the error above and the error you'll report when you switch back to 0.0.3?:

1. What are the jupyter notebook commnads you run leading up to the error?
2. Not knowing the answer to 1 - are you loading binary targets from Pants? If so, what version of Pants? If not, what versions of Pex are the PEX files you're loading built with? You can find out the latter with unzip -qc tmp/ml_training_env.pex PEX-INFO | jq -r .build_properties.pex_version

[horuswoo]

Error on 0.0.3:

HTML(value='<style>.nb-console-output-GEtbE { background-color: black;} .nb-console-output-GEtbE pre { color: …
Accordion(children=(Output(layout=Layout(height='300px', overflow_y='scroll'), outputs=({'output_type': 'displ…
Scrubbing sys.path and sys.modules in preparation for pex bootstrap
sys.path contains 7 items, sys.modules contains 1213 keys
sys.path now contains 7 items, sys.modules now contains 1213 keys
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/plugin.py in _accordion_widget(self, title, height, collapsed)
    147         # Capture the output context.
    148         with outputter:
--> 149             yield expand, collapse, set_output_glyph
    150 
    151     def _stream_binary_build_with_output(

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/plugin.py in _bootstrap_pex(self, pex_path)
    295 
    296                     # Bootstrap pex.
--> 297                     for path in self._pex.mount(pex_path):
    298                         self._display_line(f"added sys.path entry {path}\n")
    299             except Exception:

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/pex.py in mount(self, pex_to_mount)

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/pex.py in run_pex_tool(args, **subprocess_args)
    142                     interpreter_constraints = json.loads(
    143                         run_pex_tool(args=["info"], stdout=subprocess.PIPE).decode()
--> 144                     )["interpreter_constraints"]
    145                     raise self.IncompatibleError(
    146                         pex_to_mount,

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    510         if check and retcode:
    511             raise CalledProcessError(retcode, process.args,
--> 512                                      output=stdout, stderr=stderr)
    513     return CompletedProcess(process.args, retcode, stdout, stderr)
    514 

CalledProcessError: Command '['/home/ec2-user/anaconda3/envs/mxnet_latest_p37/bin/python', '/home/ec2-user/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex', '-m', 'pex.tools', 'tmp/ml_training_env.pex', 'venv', '/home/ec2-user/.cache/pants_jupyter_plugin/pex/venvs/bar/ml_training_env.pex.foo']' returned non-zero exit status 1.

[horuswoo]

• What are the jupyter notebook commnads you run leading up to the error?
• Not knowing the answer to 1 - are you loading binary targets from Pants? If so, what version of Pants? If not, what versions of Pex are the PEX files you're loading built with? You can find out the latter with unzip -qc tmp/ml_training_env.pex PEX-INFO | jq -r .build_properties.pex_version

%pip install pants-jupyter-plugin==0.0.3
%rm -rf tmp && mkdir tmp

import boto3
s3 = boto3.resource('s3')
s3.Bucket("mybucket").download_file("my.pex", "tmp/my.pex")

%load_ext pants_jupyter_plugin

2. Yes, pants_version = "2.8.0"

[jsirois]

Thanks for the extra info @horuswoo. There are two problems to solve here. I'd like to start with solving roll back to 0.0.3.

From your 0.0.3 logs, you have CalledProcessError: Command '['/home/ec2-user/anaconda3/envs/mxnet_latest_p37/bin/python', '/home/ec2-user/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex', .... That is an impossible log line for the 0.0.3 code base.

In the 0.0.3 session, I'm willing to bet that the following shows 0.0.4:

import pants_jupyter_plugin
pants_jupyter_plugin.__version__

Perhaps you need to say --upgrade?:

%pip install --upgrade pants-jupyter-plugin==0.0.3

[jsirois]

Ok, after alot of helpful information from @horuswoo and their colleague off line, it turns out this issue was not to do with the pants-jupyter-plugin or its recent upgrade. It was due to a PEX file built on Ubuntu 20.04 attempting to be loaded on Ubuntu 16.04. The PEX file had manylinux_2_17 wheels but Ubuntu 16.04 only supports up to manylinux2014.

I'm going to close this issue as answered but open another issue to see if more information can be presented in the jupyter notebook on failure. That information is missing in the error output above but was present when running the failing Pex command manually, and it was invaluable.