pex python interpreter path contains absolute path from previous remote execution sandbox #13170

tdyas · 2021-10-08T15:15:14Z

Protobuf code generation is currently failing when run in remote execution against Buildfarm even after the fix in #13077 for archive extraction in remote execution.

Using the example-python repository for the test, the error that occurs when running ./pants test helloworld:: (with appropriate options to enable remote execution) is:

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generating Python sources from helloworld/util/proto/config.proto.' failed with exit code 1.
stdout:

stderr:
/tmp/worker/remote-execution/operations/28150c45-177d-4894-9fcb-7b88fc57f9c7/.cache/pex_root/venvs/8ccfdd776b06121b883ca8d3331e429b87fe042d/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy: 2: exec: /tmp/worker/remote-execution/operations/c79cf097-80ab-4212-b27f-b999f573cfc7/.cache/pex_root/venvs/short/02a3c5ef/bin/python3.8: not found
--mypy_out: protoc-gen-mypy: Plugin failed with status code 127.

Note that the path to protoc-gen-mypy and the path to python3.8 are different: 28150c45-177d-4894-9fcb-7b88fc57f9c7 vs c79cf097-80ab-4212-b27f-b999f573cfc7. The shebang for the python3.8 in the venv pex has the path that was valid in a prior remote execution sandbox.

This occurs on the current Pants main. It is unclear when it was introduced since the bug was obscured by the archive extraction bug fixed by #13077.

The text was updated successfully, but these errors were encountered:

tdyas · 2021-10-08T15:24:59Z

To reproduce:

Clone Pants, remote-apis-testing, and example-python.
Bring up Buildfarm using the configs in remote-apis-testing:

cd remote-apis-testing
cd docker-compose
python3 -m venv .venv
.venv/bin/pip install jinja2 ruamel.yaml
.venv/bin/python ./create_docker_compose_yaml.py 
docker-compose -f docker-compose-buildfarm.yml up

Add the following script to example-python (and chmod +x it):

#!/bin/bash
./pants_from_sources --no-pantsd -ldebug \
  --remote-execution \
  --remote-execution-address=grpc://127.0.0.1:8980 \
  --remote-store-address=grpc://127.0.0.1:8980 \
  --remote-instance-name=remote-execution \
  --remote-execution-extra-platform-properties=OSFamily=linux \
  --pex-verbosity=3 \
  test helloworld::

Run do-test.sh.

tdyas · 2021-10-08T15:55:19Z

I tried changing the link_input_directories option of Buildfarm's worker from false to true, but no change in the error. (I had thought that maybe bazelbuild/bazel-buildfarm#492 (comment) was relevant, but the config used in remote-apis-testing already sets link_input_directories to false.)

stuhood · 2021-10-11T20:27:37Z

This occurs on the current Pants main. It is unclear when it was introduced since the bug was obscured by the archive extraction bug fixed by #13077.

Does it also occur with the latest 2.7.1 (will cut an rc today)? The largest change(s) on main were cherrypicked as 2512aa2.

tdyas · 2021-10-11T23:48:50Z

Does it also occur with the latest 2.7.1 (will cut an rc today)? The largest change(s) on main were cherrypicked as 2512aa2.

It does. The regression occurred some time after 2.3.x and I have not bisected yet. Given the length of time / number of changes, probably not worth bisecting.

jsirois · 2021-10-14T16:43:05Z

This doesn't make alot of sense since we have no append only caches facility when remoting. As such, the file /tmp/worker/remote-execution/operations/28150c45-177d-4894-9fcb-7b88fc57f9c7/.cache/pex_root/venvs/8ccfdd776b06121b883ca8d3331e429b87fe042d/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy is created on-the-fly in every execution of the source PEX file since /tmp/worker/remote-execution/operations/28150c45-177-4894-9fcb-7b88fc57f9c7/.cache/pex_root is not a symlink to a shared cache in the remoting case like it is in the local case. I'll continue to investigate...

tdyas · 2021-10-14T16:50:36Z

Info on Buildfarm that may or may not be relevant: Buildfarm hard links all of the files in the input root into its local CAS cache. See discussion at https://buildteamworld.slack.com/archives/C9C4H1SN7/p1633710791295800.

jsirois · 2021-10-14T17:00:32Z

Thanks - that shouldn't be relevant since the relevant input root is:

mypy_protobuf.pex
mypy_protobuf.pex_bin_protoc-gen-mypy_shim.sh
mypy_protobuf.pex_bin_python_shim.sh
mypy_protobuf.pex_pex_shim.sh

The relevant packed wheel contains:

$ unzip -qc mypy_protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl bin/protoc-gen-mypy | head -1
#!python

And the relevant shim script contains:

$ cat mypy_protobuf.pex_bin_protoc-gen-mypy_shim.sh
#!/usr/bin/bash
set -euo pipefail

# N.B.: We convert all sandbox root relative paths to absolute paths so this script
# works when run with a cwd set elsewhere.

# N.B.: This relies on BASH_SOURCE which has been available since bash-3.0, released in
# 2004. In turn, our use of BASH_SOURCE relies on the fact that this script is executed
# by the engine via its absolute path.
ABS_SANDBOX_ROOT="${BASH_SOURCE%/*}"

function ensure_absolute() {
    local value0="$1"
    shift
    if [ "${value0:0:1}" == "/" ]; then
        echo "${value0}" "$@"
    else
        echo "${ABS_SANDBOX_ROOT}/${value0}" "$@"
    fi
}

export PATH=/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/go/bin:/home/jsirois/.bash.d/bin PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root LANG=en_US.UTF-8
export PEX_ROOT="$(ensure_absolute ${PEX_ROOT})"

execute_pex_args="$(ensure_absolute /usr/bin/python3.6) $(ensure_absolute mypy_protobuf.pex)"
target_venv_executable="$(ensure_absolute .cache/pex_root/venvs/89c5e62b4572637dc3b077e8d663e561c89a2920/061a3a67f6a451d33806aa2696288de4ee1b175c/bin/protoc-gen-mypy)"
venv_dir="$(ensure_absolute .cache/pex_root/venvs/89c5e62b4572637dc3b077e8d663e561c89a2920/061a3a67f6a451d33806aa2696288de4ee1b175c)"

# Let PEX_TOOLS invocations pass through to the original PEX file since venvs don't come
# with tools support.
if [ -n "${PEX_TOOLS:-}" ]; then
  exec ${execute_pex_args} "$@"
fi

# If the seeded venv has been removed from the PEX_ROOT, we re-seed from the original
# `--venv` mode PEX file.
if [ ! -e "${target_venv_executable}" ]; then
    rm -rf "${venv_dir}" || true
    PEX_INTERPRETER=1 ${execute_pex_args} -c ''
fi

exec "${target_venv_executable}" "$@"

No references to any absolute paths from yore in sight.

It's as-if the .cache/pex_root dir tree from a prior run is getting saved somehow and restored in later remote process executions.

jsirois · 2021-10-14T17:02:18Z

Oh, hm - execute_pex_args="$(ensure_absolute /usr/bin/python3.6) $(ensure_absolute mypy_protobuf.pex)".

My examples above were all taken from a local process execution sandbox. Continuing to dig.

tdyas · 2021-10-14T17:03:33Z

Oh, hm - execute_pex_args="$(ensure_absolute /usr/bin/python3.6) $(ensure_absolute mypy_protobuf.pex)".

My examples above were all taken from a local process execution sandbox. Continuing to dig.

I would try running it in Buildfarm (using the config in remote-apis-testing). This error only happens in Buildfarm. The code works fine running in Buildbarn and Buildgrid.

jsirois · 2021-10-14T17:04:17Z

Yeah - I am. Using your setup above. I just used a local run too to easily grok what's going on. Not as easy to do that in BuildFarm without re-familiarizing myself with its knobs to observe / keep around sandboxes.

tdyas · 2021-10-14T17:11:57Z

I verified that the error occurs with v2.5.2 and Buildfarm. (Tested with latest 2.5.x branch plus had to cherry-pick the archive extraction fix from 2.7.x of 4358c62335d287939242f847d0df1818e12b099e.)

tdyas · 2021-10-14T17:26:50Z

Works for 2.4.2. So the error was introduced somewhere between 2.4.2 and 2.5.2. pex went from 2.1.35 to 2.1.42.

tdyas · 2021-10-20T05:04:03Z

I reproduced the failure by extracting the input root into its own directory and running the applicable command:

vagrant@vagrant:~/input-root$ PANTS_CACHE_KEY_TARGET_PLATFORM=none ./bin/protoc --python_out _generated_files --plugin=protoc-gen-mypy=./mypy_protobuf.pex_bin_protoc-gen-mypy_shim.sh --mypy_out _generated_files helloworld/util/proto/config.proto
/home/vagrant/input-root/.cache/pex_root/venvs/89aa3d0e44d3459d7d29cab9699baf5549aadf63/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy: 2: exec: /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8: not found
--mypy_out: protoc-gen-mypy: Plugin failed with status code 127.
vagrant@vagrant:~/input-root$ head .cache/pex_root/venvs/89aa3d0e44d3459d7d29cab9699baf5549aadf63/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

So protoc-gen-mypy was built with the sandbox's absolute path in its shebang. So whatever else is going on, the issue is probably not in this build action, but in the prior build action that would have built protoc-gen-mypy.

Indeed happened for all of the similar scripts:

vagrant@vagrant:~/input-root$ rg /tmp/worker -- .cache/
.cache/pex_root/venvs/89aa3d0e44d3459d7d29cab9699baf5549aadf63/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy_grpc
2:'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"

.cache/pex_root/venvs/89aa3d0e44d3459d7d29cab9699baf5549aadf63/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy
2:'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"

.cache/pex_root/installed_wheels/9b0919465dea5f6e283da6280356b0c5ae234313/mypy_protobuf-2.4-py3-none-any.whl/bin/protoc-gen-mypy_grpc
2:'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"

.cache/pex_root/installed_wheels/9b0919465dea5f6e283da6280356b0c5ae234313/mypy_protobuf-2.4-py3-none-any.whl/bin/protoc-gen-mypy
2:'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"

tdyas · 2021-10-20T06:15:34Z

The .deps/mypy_protobuf-2.4-py3-none-any.whl has the paths embedded in the contained bin scripts:

vagrant@vagrant:~/mypy_protobuf_unpacked$ unzip ../pex-tree-output/.deps/mypy_protobuf-2.4-py3-none-any.whl
Archive:  ../pex-tree-output/.deps/mypy_protobuf-2.4-py3-none-any.whl
warning:  skipped "../" path component(s) in ../
   creating: mypy_protobuf-2.4.dist-info/
  inflating: mypy_protobuf-2.4.dist-info/INSTALLER
  inflating: mypy_protobuf-2.4.dist-info/LICENSE
  inflating: mypy_protobuf-2.4.dist-info/METADATA
  inflating: mypy_protobuf-2.4.dist-info/RECORD
  inflating: mypy_protobuf-2.4.dist-info/REQUESTED
  inflating: mypy_protobuf-2.4.dist-info/WHEEL
  inflating: mypy_protobuf-2.4.dist-info/entry_points.txt
  inflating: mypy_protobuf-2.4.dist-info/top_level.txt
   creating: bin/
  inflating: bin/protoc-gen-mypy
  inflating: bin/protoc-gen-mypy_grpc
  inflating: bin/protoc_gen_mypy.bat
   creating: mypy_protobuf/
  inflating: mypy_protobuf/__init__.py
  inflating: mypy_protobuf/extensions_pb2.py
  inflating: mypy_protobuf/main.py
vagrant@vagrant:~/mypy_protobuf_unpacked$ head -3 bin/*
==> bin/protoc-gen-mypy <==
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/8379cb0b-99d5-48cf-800e-9adcaa1c4c0e/.cache/pex_root/venvs/short/906abfa3/bin/python3.8 "$0" "$@"
' '''

==> bin/protoc_gen_mypy.bat <==
@echo off
%~dp0\python -u %0\..\protoc-gen-mypy

==> bin/protoc-gen-mypy_grpc <==
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/8379cb0b-99d5-48cf-800e-9adcaa1c4c0e/.cache/pex_root/venvs/short/906abfa3/bin/python3.8 "$0" "$@"
' '''

tdyas · 2021-10-20T06:21:25Z

@jsirois: Any pointers on where to look to see how that wheel in #13170 (comment) is made?

jsirois · 2021-10-20T16:06:55Z

Welp:

$ rm -rf ~/.pex
$ pex mypy-protobuf==2.4 --layout packed -omypy-protobuf.pex
$ ls -l mypy-protobuf.pex/.deps
total 1060
-rw-r--r-- 2 jsirois jsirois   15296 Oct 20 08:57 mypy_protobuf-2.4-py3-none-any.whl
-rw-r--r-- 2 jsirois jsirois 1067135 Oct 20 08:57 protobuf-3.18.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
$ zipinfo mypy-protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl 
Archive:  mypy-protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl
Zip file size: 15296 bytes, number of entries: 18
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 ../
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/
-rw-r--r--  2.0 unx        4 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/INSTALLER
-rw-r--r--  2.0 unx    11348 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/LICENSE
-rw-r--r--  2.0 unx      438 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/METADATA
-rw-r--r--  2.0 unx     1171 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/RECORD
-rw-r--r--  2.0 unx        0 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/REQUESTED
-rw-r--r--  2.0 unx       92 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/WHEEL
-rw-r--r--  2.0 unx      108 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/entry_points.txt
-rw-r--r--  2.0 unx       14 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/top_level.txt
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 mypy_protobuf/
-rw-r--r--  2.0 unx        0 b- defN 80-Jan-01 00:00 mypy_protobuf/__init__.py
-rw-r--r--  2.0 unx     2959 b- defN 80-Jan-01 00:00 mypy_protobuf/extensions_pb2.py
-rw-r--r--  2.0 unx    32486 b- defN 80-Jan-01 00:00 mypy_protobuf/main.py
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 bin/
-rwxr-xr-x  2.0 unx      207 b- defN 80-Jan-01 00:00 bin/protoc-gen-mypy
-rwxr-xr-x  2.0 unx      207 b- defN 80-Jan-01 00:00 bin/protoc-gen-mypy_grpc
-rwxr-xr-x  2.0 unx       48 b- defN 80-Jan-01 00:00 bin/protoc_gen_mypy.bat
18 files, 49082 bytes uncompressed, 12928 bytes compressed:  73.7%
$ unzip -qc mypy-protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl bin/protoc-gen-mypy
#!python
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

You'll notice the #!python special shebang which means fill me in later when you create a venv.

That's done here when Pex uses Pip to install each wheel into a ~/.pex/installed_wheels/<hash>/... chroot. That chroot is then packed here: https://github.com/pantsbuild/pex/blob/d6235f55282521d28448422d773ec58280f85fdc/pex/pex_builder.py#L690-L713 No sheabng manipulations are performed, just a straight zipping of the wheel chroot; so the #!python psuedo-shebang is preserved.

The place where the psuedo-shebang is reified is creating a venv in --venv mode here: https://github.com/pantsbuild/pex/blob/0ab02beb91b406563b88049fd470beca82a12d7d/pex/tools/commands/venv.py#L329-L331

Note though that that venv re-write occurs inside ~/.pex/venvs which should not persist between sandboxes on the remote since the remexec doesn't support named_caches.

tdyas · 2021-10-20T16:34:46Z

And to restate the issue for clarity, once that wheel is packed with the old paths, those paths will persist in all subsequent venv builds using that wheel?

tdyas · 2021-10-20T20:09:04Z

No sheabng manipulations are performed, just a straight zipping of the wheel chroot; so the #!python psuedo-shebang is preserved.

As demonstrated by #13170 (comment), the wheel captured from the build_pex step does have the fully-substituted shebang paths and not #!python. Any ideas where that could be coming from, or is the protobuf rule doing something wrong?

jsirois · 2021-11-23T19:55:23Z

And to restate the issue for clarity, once that wheel is packed with the old paths, those paths will persist in all subsequent venv builds using that wheel?

Yes, but "once that wheel is packed with the old paths" is the part that I can't currently concede. #13170 (comment) shows that mechanism.

As demonstrated by #13170 (comment), the wheel captured from the build_pex step does have the fully-substituted shebang paths and not #!python. Any ideas where that could be coming from, or is the protobuf rule doing something wrong?

Well, it has a #!/bin/sh...shebang which is nothing Pants or Pex generates in any of their codebases. So still no clue what is generating that thing.

jsirois · 2021-11-23T21:29:34Z

Yeah, so for this remexec request / response pair:

Request:

12:33:13.72 [DEBUG] built REv2 request (build_id=pants_run_2021_11_23_12_32_53_722_902fdd593d1b428cb53078fe1d0e8ea1): action=Action { command_digest: Some(Digest { hash: "9fcbb93300f2d28aafe21f5658c1bea0856279031ccc4877cc981361abe6da3a", size_bytes: 1758 }), input_root_digest: Some(Digest { hash: "8a0f7034ac7f7fb4ed5d68f8de9ba16766fba86797ccbb618b7ab3fb8f9b8f1c", size_bytes: 351 }), timeout: None, do_not_cache: false, salt: b"", platform: None }; command=Command { arguments: ["/usr/bin/python3.8", "./pex", "--python-path", "/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/jvm/default/bin:/home/jsirois/go/bin:/home/jsirois/.pyenv/versions/2.7.18/bin:/home/jsirois/.pyenv/versions/3.10.0/bin:/home/jsirois/.pyenv/versions/3.5.10/bin:/home/jsirois/.pyenv/versions/3.6.15/bin:/home/jsirois/.pyenv/versions/3.7.12/bin:/home/jsirois/.pyenv/versions/3.8.12/bin:/home/jsirois/.pyenv/versions/3.9.7/bin:/home/jsirois/.pyenv/versions/pypy2.7-7.3.1/bin:/home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin:/home/jsirois/.pyenv/versions/pypy3.7-7.3.5/bin", "--tmpdir", ".tmp", "-vvv", "--output-file", "mypy_protobuf.pex", "--venv", "--seed", "verbose", "--no-pypi", "--index=https://pypi.org/simple/", "--resolver-version", "pip-2020-resolver", "--python", "/usr/bin/python3.8", "--no-emit-warnings", "--jobs", "8", "--manylinux", "manylinux2014", "--sources-directory=source_files", "--requirement", "mypy-protobuf_default_lockfile.txt", "--no-transitive", "--layout", "packed"], environment_variables: [EnvironmentVariable { name: "CPPFLAGS", value: "" }, EnvironmentVariable { name: "LANG", value: "en_US.UTF-8" }, EnvironmentVariable { name: "LDFLAGS", value: "" }, EnvironmentVariable { name: "PANTS_CACHE_KEY_TARGET_PLATFORM", value: "linux_x86_64" }, EnvironmentVariable { name: "PATH", value: "/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/go/bin" }, EnvironmentVariable { name: "PEX_IGNORE_RCFILES", value: "true" }, EnvironmentVariable { name: "PEX_ROOT", value: ".cache/pex_root" }], output_files: [], output_directories: ["mypy_protobuf.pex"], output_paths: [], platform: Some(Platform { properties: [Property { name: "OSFamily", value: "linux" }] }), working_directory: "", output_node_properties: [] }; execute_request=ExecuteRequest { instance_name: "remote-execution", skip_cache_lookup: false, action_digest: Some(Digest { hash: "9f9cc8a24e068cd32dd9dd1832da8cd204312876068c48bf59d6d21df7dd8f20", size_bytes: 142 }), execution_policy: None, results_cache_policy: None }

Response:

12:33:20.09 [DEBUG] Got (nested) execute response: ExecuteResponse { result: Some(ActionResult { output_files: [], output_file_symlinks: [], output_symlinks: [], output_directories: [OutputDirectory { path: "mypy_protobuf.pex", tree_digest: Some(Digest { hash: "f07520de14528d331d759be2fcd5e1b1f4c3578f60965a685e2035737cfab35d", size_bytes: 601 }) }], output_directory_symlinks: [], exit_code: 0, stdout_raw: b"{\"pex_root\": \"/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root\", \"python\": \"/usr/bin/python3.8\", \"pex\": \"/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6/pex\"}\n", stdout_digest: Some(Digest { hash: "4be490816903666cede665f4bbd1e2a40627992388fda26c24e01a52306aaabb", size_bytes: 336 }), stderr_raw: b"pex: Resolving interpreters\npex: Resolving interpreters: 0.1ms\npex: Building pex\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt)\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements.\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements. :: Resolving for:\n  DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10))))\npex: Hashing pex\npex: Hashing pex: 19.4ms\npex: Isolating pex\npex: Isolating pex :: Extracting pex to /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d\npex: Isolating pex: 49.9ms\npex:   Extracting pex to /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d: 49.7ms\npex: Vendoring attr from VendorSpec(key='attrs', requirement='attrs==21.2.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/attrs\npex: Vendoring packaging from VendorSpec(key='packaging', requirement='packaging==20.8', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/packaging\npex: Vendoring pyparsing from VendorSpec(key='packaging', requirement='packaging==20.8', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/packaging\npex: Vendoring pkg_resources from VendorSpec(key='setuptools', requirement='setuptools==44.0.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools\npex: Vendoring setuptools from VendorSpec(key='setuptools', requirement='setuptools==44.0.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools\npex: Vendoring wheel from VendorSpec(key='wheel', requirement='wheel==0.36.2', rewrite=False, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/wheel\npex: Executing: /usr/bin/python3.8 -s -E /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmp9r05ljdg\npex: Executing: /usr/bin/python3.8 -s -E -m venv --without-pip /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee --prompt pex\npex: Executing: PYTHONPATH=/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/python3.8 -s -c import os\nimport sys\n\nfrom pex.common import atomic_directory, safe_open\nfrom pex.interpreter import PythonIdentity\n\n\nencoded_identity = PythonIdentity.get(binary='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/python3.8').encode()\nwith atomic_directory('/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/interpreters/1767c82461266558947a7891785e6e3a0441205f/a311155cf6dcd5e39bbe622992d50b4097308d16/15ae6261d159651d4661ae21d260f3090ec77280', exclusive=False) as cache_dir:\n    if not cache_dir.is_finalized:\n        with safe_open(\n            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'\n        ) as fp:\n            fp.write(encoded_identity)\n\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/activate\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/activate.csh\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/activate.fish\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps :: Adding pip 20.3.4\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps :: Adding setuptools 44.0.0\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps :: Adding wheel 0.36.2\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps: 0.7ms\npex:   Adding pip 20.3.4: 0.0ms\npex:   Adding setuptools 44.0.0: 0.0ms\npex:   Adding wheel 0.36.2: 0.0ms\npex: Resolving pip==20.3.4\npex: Resolving pip==20.3.4: 0.4ms\npex: Resolving setuptools==44.0.0\npex: Skipping activation of `certifi==2016.9.26; extra == \"certs\"` due to environment marker de-selection\npex: Skipping activation of `wincertstore==0.2; sys_platform == \"win32\" and extra == \"ssl\"` due to environment marker de-selection\npex: Resolving setuptools==44.0.0: 1.6ms\npex: Resolving wheel==0.36.2\npex: Skipping activation of `pytest>=3.0.0; extra == \"test\"` due to environment marker de-selection\npex: Skipping activation of `pytest-cov; extra == \"test\"` due to environment marker de-selection\npex: Resolving wheel==0.36.2: 1.2ms\n/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_bootstrapper.py:483: PEXWarning: The venv for /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d at /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6 has script shebangs of '#!/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/89607fa6/venv/bin/python3.8 -sE' with 134 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root to a shorter path as a work-around.\n  pex_warnings.warn(\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements. :: Calculating project names for direct requirements:\n  PyPIRequirement(line=LogicalLine(raw_text='mypy-protobuf==2.4; python_version >= \"3.6\" \\\\\\n    --hash=sha256:77e10c476cdd3ee14535c2357e64deac6b1a69f33eb500d795b064acda48c66f \\\\\\n    --hash=sha256:1986dc8a7994a4e549be5c8ac586aa8247caf92456c67159d71dbe5005809651\\n', processed_text='mypy-protobuf==2.4; python_version >= \"3.6\" --hash=sha256:77e10c476cdd3ee14535c2357e64deac6b1a69f33eb500d795b064acda48c66f --hash=sha256:1986dc8a7994a4e549be5c8ac586aa8247caf92456c67159d71dbe5005809651', source='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/mypy-protobuf_default_lockfile.txt', start_line=17, end_line=19), requirement=Requirement.parse('mypy-protobuf==2.4; python_version >= \"3.6\"'), editable=False)\n  PyPIRequirement(line=LogicalLine(raw_text='protobuf==3.19.1; python_version >= \"3.6\" \\\\\\n    --hash=sha256:d80f80eb175bf5f1169139c2e0c5ada98b1c098e2b3c3736667f28cbbea39fc8 \\\\\\n    --hash=sha256:a529e7df52204565bcd33738a7a5f288f3d2d37d86caa5d78c458fa5fabbd54d \\\\\\n    --hash=sha256:28ccea56d4dc38d35cd70c43c2da2f40ac0be0a355ef882242e8586c6d66666f \\\\\\n    --hash=sha256:8b30a7de128c46b5ecb343917d9fa737612a6e8280f440874e5cc2ba0d79b8f6 \\\\\\n    --hash=sha256:5935c8ce02e3d89c7900140a8a42b35bc037ec07a6aeb61cc108be8d3c9438a6 \\\\\\n    --hash=sha256:74f33edeb4f3b7ed13d567881da8e5a92a72b36495d57d696c2ea1ae0cfee80c \\\\\\n    --hash=sha256:038daf4fa38a7e818dd61f51f22588d61755160a98db087a046f80d66b855942 \\\\\\n    --hash=sha256:8e51561d72efd5bd5c91490af1f13e32bcba8dab4643761eb7de3ce18e64a853 \\\\\\n    --hash=sha256:6e8ea9173403219239cdfd8d946ed101f2ab6ecc025b0fda0c6c713c35c9981d \\\\\\n    --hash=sha256:db3532d9f7a6ebbe2392041350437953b6d7a792de10e629c1e4f5a6b1fe1ac6 \\\\\\n    --hash=sha256:615b426a177780ce381ecd212edc1e0f70db8557ed72560b82096bd36b01bc04 \\\\\\n    --hash=sha256:d8919368410110633717c406ab5c97e8df5ce93020cfcf3012834f28b1fab1ea \\\\\\n    --hash=sha256:71b0250b0cfb738442d60cab68abc166de43411f2a4f791d31378590bfb71bd7 \\\\\\n    --hash=sha256:3cd0458870ea7d1c58e948ac8078f6ba8a7ecc44a57e03032ed066c5bb318089 \\\\\\n    --hash=sha256:655264ed0d0efe47a523e2255fc1106a22f6faab7cc46cfe99b5bae085c2a13e \\\\\\n    --hash=sha256:b691d996c6d0984947c4cf8b7ae2fe372d99b32821d0584f0b90277aa36982d3 \\\\\\n    --hash=sha256:e7e8d2c20921f8da0dea277dfefc6abac05903ceac8e72839b2da519db69206b \\\\\\n    --hash=sha256:fd390367fc211cc0ffcf3a9e149dfeca78fecc62adb911371db0cec5c8b7472d \\\\\\n    --hash=sha256:d83e1ef8cb74009bebee3e61cc84b1c9cd04935b72bca0cbc83217d140424995 \\\\\\n    --hash=sha256:36d90676d6f426718463fe382ec6274909337ca6319d375eebd2044e6c6ac560 \\\\\\n    --hash=sha256:e7b24c11df36ee8e0c085e5b0dc560289e4b58804746fb487287dda51410f1e2 \\\\\\n    --hash=sha256:77d2fadcf369b3f22859ab25bd12bb8e98fb11e05d9ff9b7cd45b711c719c002 \\\\\\n    --hash=sha256:e813b1c9006b6399308e917ac5d298f345d95bb31f46f02b60cd92970a9afa17 \\\\\\n    --hash=sha256:62a8e4baa9cb9e064eb62d1002eca820857ab2138440cb4b3ea4243830f94ca7\\n', processed_text='protobuf==3.19.1; python_version >= \"3.6\" --hash=sha256:d80f80eb175bf5f1169139c2e0c5ada98b1c098e2b3c3736667f28cbbea39fc8 --hash=sha256:a529e7df52204565bcd33738a7a5f288f3d2d37d86caa5d78c458fa5fabbd54d --hash=sha256:28ccea56d4dc38d35cd70c43c2da2f40ac0be0a355ef882242e8586c6d66666f --hash=sha256:8b30a7de128c46b5ecb343917d9fa737612a6e8280f440874e5cc2ba0d79b8f6 --hash=sha256:5935c8ce02e3d89c7900140a8a42b35bc037ec07a6aeb61cc108be8d3c9438a6 --hash=sha256:74f33edeb4f3b7ed13d567881da8e5a92a72b36495d57d696c2ea1ae0cfee80c --hash=sha256:038daf4fa38a7e818dd61f51f22588d61755160a98db087a046f80d66b855942 --hash=sha256:8e51561d72efd5bd5c91490af1f13e32bcba8dab4643761eb7de3ce18e64a853 --hash=sha256:6e8ea9173403219239cdfd8d946ed101f2ab6ecc025b0fda0c6c713c35c9981d --hash=sha256:db3532d9f7a6ebbe2392041350437953b6d7a792de10e629c1e4f5a6b1fe1ac6 --hash=sha256:615b426a177780ce381ecd212edc1e0f70db8557ed72560b82096bd36b01bc04 --hash=sha256:d8919368410110633717c406ab5c97e8df5ce93020cfcf3012834f28b1fab1ea --hash=sha256:71b0250b0cfb738442d60cab68abc166de43411f2a4f791d31378590bfb71bd7 --hash=sha256:3cd0458870ea7d1c58e948ac8078f6ba8a7ecc44a57e03032ed066c5bb318089 --hash=sha256:655264ed0d0efe47a523e2255fc1106a22f6faab7cc46cfe99b5bae085c2a13e --hash=sha256:b691d996c6d0984947c4cf8b7ae2fe372d99b32821d0584f0b90277aa36982d3 --hash=sha256:e7e8d2c20921f8da0dea277dfefc6abac05903ceac8e72839b2da519db69206b --hash=sha256:fd390367fc211cc0ffcf3a9e149dfeca78fecc62adb911371db0cec5c8b7472d --hash=sha256:d83e1ef8cb74009bebee3e61cc84b1c9cd04935b72bca0cbc83217d140424995 --hash=sha256:36d90676d6f426718463fe382ec6274909337ca6319d375eebd2044e6c6ac560 --hash=sha256:e7b24c11df36ee8e0c085e5b0dc560289e4b58804746fb487287dda51410f1e2 --hash=sha256:77d2fadcf369b3f22859ab25bd12bb8e98fb11e05d9ff9b7cd45b711c719c002 --hash=sha256:e813b1c9006b6399308e917ac5d298f345d95bb31f46f02b60cd92970a9afa17 --hash=sha256:62a8e4baa9cb9e064eb62d1002eca820857ab2138440cb4b3ea4243830f94ca7', source='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/mypy-protobuf_default_lockfile.txt', start_line=20, end_line=44), requirement=Requirement.parse('protobuf==3.19.1; python_version >= \"3.6\"'), editable=False)\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements. :: Installing:\n  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))), wheel_path='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl', fingerprint='60974b0bc64909a12b2201d5d0093478d955e90a')\n  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))), wheel_path='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', fingerprint='48658658ca878b5d7683a9c90b3f7d92169ca45b')\npex: Installing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl in /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl\npex: Installing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/48658658ca878b5d7683a9c90b3f7d92169ca45b/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl\nUsing pip 20.3.4 from /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/89607fa6/venv/lib/python3.8/site-packages/pip (python 3.8)\nNon-user install due to --prefix or --target option\nUsing pip 20.3.4 from /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/89607fa6/venv/lib/python3.8/site-packages/pip (python 3.8)\nNon-user install due to --prefix or --target option\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-target-dv089yax\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-target-pgdxjh3n\nIgnoring indexes: https://pypi.org/simple\nIgnoring indexes: https://pypi.org/simple\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-ephem-wheel-cache-scfu0jpx\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-ephem-wheel-cache-ckp1vntx\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq\nInitialized build tracking at /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd\nInitialized build tracking at /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq\nCreated build tracker: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd\nCreated build tracker: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq\nEntered build tracker: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd\nEntered build tracker: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-install-soa8ffvm\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-install-q3rlkryu\nProcessing ./.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl\nProcessing ./.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl\n  Added protobuf==3.19.1 from file:///tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl to build tracker '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd'\n  Added mypy-protobuf==2.4 from file:///tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl to build tracker '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq'\n  Removed protobuf==3.19.1 from file:///tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl from build tracker '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd'\n  Removed mypy-protobuf==2.4 from file:///tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl from build tracker '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq'\nInstalling collected packages: mypy-protobuf\nInstalling collected packages: protobuf\n  changing mode of /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-target-pgdxjh3n/bin/protoc-gen-mypy to 755\n  changing mode of /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-target-pgdxjh3n/bin/protoc-gen-mypy_grpc to 755\nSuccessfully installed mypy-protobuf-2.4\nRemoved build tracker: '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq'\nSuccessfully installed protobuf-3.19.1\nRemoved build tracker: '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd'\npex: Building pex: 2209.1ms\npex:   Resolving distributions (mypy-protobuf_default_lockfile.txt): 2208.8ms\npex:     Resolving requirements.: 2205.3ms\npex:       Resolving for:\n  DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))): 1993.7ms\npex:       Calculating project names for direct requirements:\n  PyPIRequirement(line=LogicalLine(raw_text='mypy-protobuf==2.4; python_version >= \"3.6\" \\\\\\n    --hash=sha256:77e10c476cdd3ee14535c2357e64deac6b1a69f33eb500d795b064acda48c66f \\\\\\n    --hash=sha256:1986dc8a7994a4e549be5c8ac586aa8247caf92456c67159d71dbe5005809651\\n', processed_text='mypy-protobuf==2.4; python_version >= \"3.6\" --hash=sha256:77e10c476cdd3ee14535c2357e64deac6b1a69f33eb500d795b064acda48c66f --hash=sha256:1986dc8a7994a4e549be5c8ac586aa8247caf92456c67159d71dbe5005809651', source='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/mypy-protobuf_default_lockfile.txt', start_line=17, end_line=19), requirement=Requirement.parse('mypy-protobuf==2.4; python_version >= \"3.6\"'), editable=False)\n  PyPIRequirement(line=LogicalLine(raw_text='protobuf==3.19.1; python_version >= \"3.6\" \\\\\\n    --hash=sha256:d80f80eb175bf5f1169139c2e0c5ada98b1c098e2b3c3736667f28cbbea39fc8 \\\\\\n    --hash=sha256:a529e7df52204565bcd33738a7a5f288f3d2d37d86caa5d78c458fa5fabbd54d \\\\\\n    --hash=sha256:28ccea56d4dc38d35cd70c43c2da2f40ac0be0a355ef882242e8586c6d66666f \\\\\\n    --hash=sha256:8b30a7de128c46b5ecb343917d9fa737612a6e8280f440874e5cc2ba0d79b8f6 \\\\\\n    --hash=sha256:5935c8ce02e3d89c7900140a8a42b35bc037ec07a6aeb61cc108be8d3c9438a6 \\\\\\n    --hash=sha256:74f33edeb4f3b7ed13d567881da8e5a92a72b36495d57d696c2ea1ae0cfee80c \\\\\\n    --hash=sha256:038daf4fa38a7e818dd61f51f22588d61755160a98db087a046f80d66b855942 \\\\\\n    --hash=sha256:8e51561d72efd5bd5c91490af1f13e32bcba8dab4643761eb7de3ce18e64a853 \\\\\\n    --hash=sha256:6e8ea9173403219239cdfd8d946ed101f2ab6ecc025b0fda0c6c713c35c9981d \\\\\\n    --hash=sha256:db3532d9f7a6ebbe2392041350437953b6d7a792de10e629c1e4f5a6b1fe1ac6 \\\\\\n    --hash=sha256:615b426a177780ce381ecd212edc1e0f70db8557ed72560b82096bd36b01bc04 \\\\\\n    --hash=sha256:d8919368410110633717c406ab5c97e8df5ce93020cfcf3012834f28b1fab1ea \\\\\\n    --hash=sha256:71b0250b0cfb738442d60cab68abc166de43411f2a4f791d31378590bfb71bd7 \\\\\\n    --hash=sha256:3cd0458870ea7d1c58e948ac8078f6ba8a7ecc44a57e03032ed066c5bb318089 \\\\\\n    --hash=sha256:655264ed0d0efe47a523e2255fc1106a22f6faab7cc46cfe99b5bae085c2a13e \\\\\\n    --hash=sha256:b691d996c6d0984947c4cf8b7ae2fe372d99b32821d0584f0b90277aa36982d3 \\\\\\n    --hash=sha256:e7e8d2c20921f8da0dea277dfefc6abac05903ceac8e72839b2da519db69206b \\\\\\n    --hash=sha256:fd390367fc211cc0ffcf3a9e149dfeca78fecc62adb911371db0cec5c8b7472d \\\\\\n    --hash=sha256:d83e1ef8cb74009bebee3e61cc84b1c9cd04935b72bca0cbc83217d140424995 \\\\\\n    --hash=sha256:36d90676d6f426718463fe382ec6274909337ca6319d375eebd2044e6c6ac560 \\\\\\n    --hash=sha256:e7b24c11df36ee8e0c085e5b0dc560289e4b58804746fb487287dda51410f1e2 \\\\\\n    --hash=sha256:77d2fadcf369b3f22859ab25bd12bb8e98fb11e05d9ff9b7cd45b711c719c002 \\\\\\n    --hash=sha256:e813b1c9006b6399308e917ac5d298f345d95bb31f46f02b60cd92970a9afa17 \\\\\\n    --hash=sha256:62a8e4baa9cb9e064eb62d1002eca820857ab2138440cb4b3ea4243830f94ca7\\n', processed_text='protobuf==3.19.1; python_version >= \"3.6\" --hash=sha256:d80f80eb175bf5f1169139c2e0c5ada98b1c098e2b3c3736667f28cbbea39fc8 --hash=sha256:a529e7df52204565bcd33738a7a5f288f3d2d37d86caa5d78c458fa5fabbd54d --hash=sha256:28ccea56d4dc38d35cd70c43c2da2f40ac0be0a355ef882242e8586c6d66666f --hash=sha256:8b30a7de128c46b5ecb343917d9fa737612a6e8280f440874e5cc2ba0d79b8f6 --hash=sha256:5935c8ce02e3d89c7900140a8a42b35bc037ec07a6aeb61cc108be8d3c9438a6 --hash=sha256:74f33edeb4f3b7ed13d567881da8e5a92a72b36495d57d696c2ea1ae0cfee80c --hash=sha256:038daf4fa38a7e818dd61f51f22588d61755160a98db087a046f80d66b855942 --hash=sha256:8e51561d72efd5bd5c91490af1f13e32bcba8dab4643761eb7de3ce18e64a853 --hash=sha256:6e8ea9173403219239cdfd8d946ed101f2ab6ecc025b0fda0c6c713c35c9981d --hash=sha256:db3532d9f7a6ebbe2392041350437953b6d7a792de10e629c1e4f5a6b1fe1ac6 --hash=sha256:615b426a177780ce381ecd212edc1e0f70db8557ed72560b82096bd36b01bc04 --hash=sha256:d8919368410110633717c406ab5c97e8df5ce93020cfcf3012834f28b1fab1ea --hash=sha256:71b0250b0cfb738442d60cab68abc166de43411f2a4f791d31378590bfb71bd7 --hash=sha256:3cd0458870ea7d1c58e948ac8078f6ba8a7ecc44a57e03032ed066c5bb318089 --hash=sha256:655264ed0d0efe47a523e2255fc1106a22f6faab7cc46cfe99b5bae085c2a13e --hash=sha256:b691d996c6d0984947c4cf8b7ae2fe372d99b32821d0584f0b90277aa36982d3 --hash=sha256:e7e8d2c20921f8da0dea277dfefc6abac05903ceac8e72839b2da519db69206b --hash=sha256:fd390367fc211cc0ffcf3a9e149dfeca78fecc62adb911371db0cec5c8b7472d --hash=sha256:d83e1ef8cb74009bebee3e61cc84b1c9cd04935b72bca0cbc83217d140424995 --hash=sha256:36d90676d6f426718463fe382ec6274909337ca6319d375eebd2044e6c6ac560 --hash=sha256:e7b24c11df36ee8e0c085e5b0dc560289e4b58804746fb487287dda51410f1e2 --hash=sha256:77d2fadcf369b3f22859ab25bd12bb8e98fb11e05d9ff9b7cd45b711c719c002 --hash=sha256:e813b1c9006b6399308e917ac5d298f345d95bb31f46f02b60cd92970a9afa17 --hash=sha256:62a8e4baa9cb9e064eb62d1002eca820857ab2138440cb4b3ea4243830f94ca7', source='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/mypy-protobuf_default_lockfile.txt', start_line=20, end_line=44), requirement=Requirement.parse('protobuf==3.19.1; python_version >= \"3.6\"'), editable=False): 0.1ms\npex:       Installing:\n  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))), wheel_path='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl', fingerprint='60974b0bc64909a12b2201d5d0093478d955e90a')\n  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))), wheel_path='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', fingerprint='48658658ca878b5d7683a9c90b3f7d92169ca45b'): 205.4ms\npex: Vendoring attr from VendorSpec(key='attrs', requirement='attrs==21.2.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/attrs\npex: Vendoring packaging from VendorSpec(key='packaging', requirement='packaging==20.8', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/packaging\npex: Vendoring pyparsing from VendorSpec(key='packaging', requirement='packaging==20.8', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/packaging\npex: Vendoring pkg_resources from VendorSpec(key='setuptools', requirement='setuptools==44.0.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools\npex: Vendoring setuptools from VendorSpec(key='setuptools', requirement='setuptools==44.0.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools\npex: Vendoring wheel from VendorSpec(key='wheel', requirement='wheel==0.36.2', rewrite=False, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/wheel\nSaving PEX file to mypy_protobuf.pex\npex: Seeding local caches for mypy_protobuf.pex\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex\npex: Executing: /usr/bin/python3.8 -s -E -m venv --without-pip /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5 --prompt mypy_protobuf.pex\npex: Executing: PYTHONPATH=/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/python3.8 -s -c import os\nimport sys\n\nfrom pex.common import atomic_directory, safe_open\nfrom pex.interpreter import PythonIdentity\n\n\nencoded_identity = PythonIdentity.get(binary='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/python3.8').encode()\nwith atomic_directory('/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/interpreters/1767c82461266558947a7891785e6e3a0441205f/a311155cf6dcd5e39bbe622992d50b4097308d16/1ebcaf932da6f6bd92ef15c004d857d1cb282c4e', exclusive=False) as cache_dir:\n    if not cache_dir.is_finalized:\n        with safe_open(\n            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'\n        ) as fp:\n            fp.write(encoded_identity)\n\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/activate\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/activate.csh\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/activate.fish\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1/.deps\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1/.deps :: Adding mypy-protobuf 2.4\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1/.deps :: Adding protobuf 3.19.1\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Resolving mypy-protobuf==2.4; python_version >= \"3.6\"\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Resolving protobuf==3.19.1; python_version >= \"3.6\"\n/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_bootstrapper.py:483: PEXWarning: The venv for /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1 at /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6 has script shebangs of '#!/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/7f892a2b/venv/bin/python3.8 -sE' with 134 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root to a shorter path as a work-around.\n  pex_warnings.warn(\npex: Seeding local caches for mypy_protobuf.pex: 131.1ms\npex:   Creating venv from mypy_protobuf.pex: 131.0ms\npex:     Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1/.deps: 1.1ms\npex:       Adding mypy-protobuf 2.4: 0.0ms\npex:       Adding protobuf 3.19.1: 0.0ms\npex:     Resolving mypy-protobuf==2.4; python_version >= \"3.6\": 0.3ms\npex:     Resolving protobuf==3.19.1; python_version >= \"3.6\": 0.0ms\n", stderr_digest: Some(Digest { hash: "bfffa759c19cd0b770877c5a0aefe767ebdbd0ceea14ab9788dd12b590c6a5c1", size_bytes: 33230 }), execution_metadata: Some(ExecutedActionMetadata { worker: "7abe2cb28c6c", queued_timestamp: Some(Timestamp { seconds: 1637699595, nanos: 412000000 }), worker_start_timestamp: Some(Timestamp { seconds: 1637699595, nanos: 413000000 }), worker_completed_timestamp: Some(Timestamp { seconds: 1637699600, nanos: 90000000 }), input_fetch_start_timestamp: Some(Timestamp { seconds: 1637699595, nanos: 413000000 }), input_fetch_completed_timestamp: Some(Timestamp { seconds: 1637699595, nanos: 422000000 }), execution_start_timestamp: Some(Timestamp { seconds: 1637699596, nanos: 926000000 }), execution_completed_timestamp: Some(Timestamp { seconds: 1637699600, nanos: 81000000 }), output_upload_start_timestamp: Some(Timestamp { seconds: 1637699600, nanos: 81000000 }), output_upload_completed_timestamp: Some(Timestamp { seconds: 1637699600, nanos: 90000000 }), auxiliary_metadata: [] }) }), cached_result: false, status: Some(Status { code: 0, message: "", details: [] }), server_logs: {}, message: "" }

If I 1st materialize the input and then run that command:

$ src/rust/engine/target/debug/fs_util --server-address grpc://127.0.0.1:8980 directory materialize 8a0f7034ac7f7fb4ed5d68f8de9ba16766fba86797ccbb618b7ab3fb8f9b8f1c 351 input
$ cd input/
$ CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PANTS_CACHE_KEY_TARGET_PLATFORM=linux_x86_64 PATH=/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/go/bin PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root /usr/bin/python3.8 ./pex --python-path /home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/jvm/default/bin:/home/jsirois/go/bin:/home/jsirois/.pyenv/versions/2.7.18/bin:/home/jsirois/.pyenv/versions/3.10.0/bin:/home/jsirois/.pyenv/versions/3.5.10/bin:/home/jsirois/.pyenv/versions/3.6.15/bin:/home/jsirois/.pyenv/versions/3.7.12/bin:/home/jsirois/.pyenv/versions/3.8.12/bin:/home/jsirois/.pyenv/versions/3.9.7/bin:/home/jsirois/.pyenv/versions/pypy2.7-7.3.1/bin:/home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin:/home/jsirois/.pyenv/versions/pypy3.7-7.3.5/bin --tmpdir .tmp -vvv --output-file mypy_protobuf.pex --venv --seed verbose --no-pypi --index=https://pypi.org/simple/ --resolver-version pip-2020-resolver --python /usr/bin/python3.8 --no-emit-warnings --jobs 8 --manylinux manylinux2014 --sources-directory=source_files --requirement mypy-protobuf_default_lockfile.txt --no-transitive --layout packed
pex: Resolving interpreters: 0.1ms
pex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements. :: Resolving for:
pex: Hashing pex                                                                                                                                                                                                                                                                 pex: Hashing pex: 20.0ms
pex: Isolating pex: 55.0ms                                                                                                                                
pex:   Extracting pex to /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d: 54.8ms
...
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/activate
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/activate.csh
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/activate.fish
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/protoc-gen-mypy
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/protoc-gen-mypy_grpc
pex: Seeding local caches for mypy_protobuf.pex: 136.8ms
pex:   Creating venv from mypy_protobuf.pex: 136.7ms
pex:     Searching dependency cache: /home/jsirois/dev/pantsbuild/jsirois-pants/input/.tmp/tmpxajtj1uk/.deps: 1.1ms
pex:       Adding mypy-protobuf 2.4: 0.0ms
pex:       Adding protobuf 3.19.1: 0.0ms
pex:     Resolving mypy-protobuf==2.4; python_version >= "3.6": 0.4ms
pex:     Resolving protobuf==3.19.1; python_version >= "3.6": 0.0ms
{"pex_root": "/home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root", "python": "/usr/bin/python3.8", "pex": "/home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6/pex"}

I find again, the shebang in the mypy_protobuf wheel zip is not expanded.:

$ unzip -qc mypy_protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl bin/protoc-gen-mypy
#!python
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Now the venv script equivalent is - of course - re-written, but not to #!/bin/sh, and, crucially, not to a path that is captured as an output (.cache/pex_root):

$ head /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy
#!/home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/s/b445c838/venv/bin/python3.8 -sE
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

That said! The output digest for the process has:

$ src/rust/engine/target/debug/fs_util --server-address grpc://127.0.0.1:8980 tree materialize f07520de14528d331d759be2fcd5e1b1f4c3578f60965a685e2035737cfab35d 601 output
$ cd output/
$ unzip -qc .deps/mypy_protobuf-2.4-py3-none-any.whl bin/protoc-gen-mypy
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/89607fa6/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

So ... I'm at a complete loss.

jsirois · 2021-11-24T17:06:28Z

Alright, doctored up Pex to log script re-write operations to a /debug.txt log file in the container and I find:

$ docker container exec -it 7abe2cb28c6c bash -i
root@7abe2cb28c6c:/bazel-buildfarm# cat /debug.txt 
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.516853df8d3644309a44b255cf2f84ba/bin
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.516853df8d3644309a44b255cf2f84ba/bin/pytest
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.516853df8d3644309a44b255cf2f84ba/bin/py.test
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.d1f50e8a2a54400390e30f5aedf230f0/bin
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.d1f50e8a2a54400390e30f5aedf230f0/bin/coverage-3.8
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.d1f50e8a2a54400390e30f5aedf230f0/bin/coverage
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.d1f50e8a2a54400390e30f5aedf230f0/bin/coverage3
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.70fd55c05eb54e46988fa82c9fe12734/bin
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.70fd55c05eb54e46988fa82c9fe12734/bin/translate-cli
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/9fa96037a1a985fe7dda81c0e1e35ae015163d88/charset_normalizer-2.0.4-py3-none-any.whl.139b639c79b94d59bd52bbb10c16348a/bin
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/9fa96037a1a985fe7dda81c0e1e35ae015163d88/charset_normalizer-2.0.4-py3-none-any.whl.139b639c79b94d59bd52bbb10c16348a/bin/normalizer
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.286876c502694a6e9c836ad5c4c4315d/bin
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.286876c502694a6e9c836ad5c4c4315d/bin/pip3.8
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.286876c502694a6e9c836ad5c4c4315d/bin/pip
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.286876c502694a6e9c836ad5c4c4315d/bin/pip3
/tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.8a63762cf82b465a88e3c6e3aece3819/bin
/tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.8a63762cf82b465a88e3c6e3aece3819/bin/protoc_gen_mypy.bat
/tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.8a63762cf82b465a88e3c6e3aece3819/bin/protoc-gen-mypy
/tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.8a63762cf82b465a88e3c6e3aece3819/bin/protoc-gen-mypy_grpc
/tmp/worker/remote-execution/operations/768271cd-264c-4c2f-b64a-a9714b0c54f0/.cache/pex_root/installed_wheels/c720e2bf057dc4ff88df428c9dd87f782a55dbd4/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/768271cd-264c-4c2f-b64a-a9714b0c54f0/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.e62c93e0086e4c6986c3ce51601a3c4f/bin
/tmp/worker/remote-execution/operations/768271cd-264c-4c2f-b64a-a9714b0c54f0/.cache/pex_root/installed_wheels/c720e2bf057dc4ff88df428c9dd87f782a55dbd4/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/768271cd-264c-4c2f-b64a-a9714b0c54f0/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.e62c93e0086e4c6986c3ce51601a3c4f/bin/pytest
/tmp/worker/remote-execution/operations/ac458f6a-2fe0-472e-804e-45c4ae120187/.cache/pex_root/installed_wheels/c720e2bf057dc4ff88df428c9dd87f782a55dbd4/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/ac458f6a-2fe0-472e-804e-45c4ae120187/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.1165c662e8c24c6aa2055d1ede6b90b3/bin
/tmp/worker/remote-execution/operations/ac458f6a-2fe0-472e-804e-45c4ae120187/.cache/pex_root/installed_wheels/c720e2bf057dc4ff88df428c9dd87f782a55dbd4/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/ac458f6a-2fe0-472e-804e-45c4ae120187/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.1165c662e8c24c6aa2055d1ede6b90b3/bin/translate-cli
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.6e8589b93c8b4aee8a5a31be7c61bedc/bin
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.6e8589b93c8b4aee8a5a31be7c61bedc/bin/pytest
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.6e8589b93c8b4aee8a5a31be7c61bedc/bin/py.test
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.4f92e12264694ab2bc10e1e2b913735e/bin
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.4f92e12264694ab2bc10e1e2b913735e/bin/coverage-3.8
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.4f92e12264694ab2bc10e1e2b913735e/bin/coverage
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.4f92e12264694ab2bc10e1e2b913735e/bin/coverage3
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.fde2c4cc03af4679af6c1dd9304fb9eb/bin
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.fde2c4cc03af4679af6c1dd9304fb9eb/bin/translate-cli
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/9fa96037a1a985fe7dda81c0e1e35ae015163d88/charset_normalizer-2.0.4-py3-none-any.whl.e6a4bc3223914e7b9002f8c3b3fd753d/bin
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/9fa96037a1a985fe7dda81c0e1e35ae015163d88/charset_normalizer-2.0.4-py3-none-any.whl.e6a4bc3223914e7b9002f8c3b3fd753d/bin/normalizer
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.364eedc968a94c60af05f03f05fa0aee/bin
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.364eedc968a94c60af05f03f05fa0aee/bin/pip3.8
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.364eedc968a94c60af05f03f05fa0aee/bin/pip
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.364eedc968a94c60af05f03f05fa0aee/bin/pip3
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.ca6b8c8c3fe14e3cbe12e1a1f9560ebb/bin
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.ca6b8c8c3fe14e3cbe12e1a1f9560ebb/bin/protoc_gen_mypy.bat
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
@echo off
%~dp0\python -u %0\..\protoc-gen-mypy
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.ca6b8c8c3fe14e3cbe12e1a1f9560ebb/bin/protoc-gen-mypy
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/venvs/s/ea545397/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.ca6b8c8c3fe14e3cbe12e1a1f9560ebb/bin/protoc-gen-mypy_grpc
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/venvs/s/ea545397/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
root@7abe2cb28c6c:/bazel-buildfarm#

So, under BuildFarm in the worker container, it's the pip install ... of the wheel that is creating the strange #!/bin/sh header. This is not what Pip does when run under not-BuildFarm.

tdyas · 2021-11-24T17:28:52Z

BuildFarm uses hard links to setup the input root. The files are also hard linked in the CAS cache directory used by BuildFarm. Could this confuse pip in any way?

tdyas · 2021-11-24T17:29:53Z

Maybe this could be related to the specific remote execution environment (e.g., the python / pip versions etc.) being used in remote -apis-testing and our ad hoc testing?

jsirois · 2021-11-24T17:30:30Z

The only ~arbitrary thing is Python, Pip is vendored by Pex.

jsirois · 2021-11-24T17:33:29Z

@#!$%:

root@7abe2cb28c6c:/bazel-buildfarm# python -mvenv test.venv
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.8-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/bazel-buildfarm/test.venv/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

Still no straight line but this is likely it.

jsirois · 2021-11-24T17:38:03Z

Actually, Pex sets up the ~/.pex/venvs without Pip and that works fine:

root@7abe2cb28c6c:/bazel-buildfarm# python -mvenv --without-pip test.venv
root@7abe2cb28c6c:/bazel-buildfarm# source test.venv/bin/activate
(test.venv) root@7abe2cb28c6c:/bazel-buildfarm# python --version
Python 3.8.10
(test.venv) root@7abe2cb28c6c:/bazel-buildfarm#

jsirois · 2021-11-24T17:58:05Z

Ok, code generating the whacky shebang is here:
https://github.com/pantsbuild/pex/blob/196b4cd5b8dd4b4af2586460530e9a777262be7d/pex/vendor/_vendored/pip/pip/_vendor/distlib/scripts.py#L127-L156

jsirois · 2021-11-24T18:07:33Z

Aha:

if sys.platform == 'darwin':
    max_shebang_length = 512
else:
    max_shebang_length = 127

If the shebang length is > 127 then the funky #!/bin/sh is used. Pex also warns about this:

/tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.cache/pex_root/installed_wheels/4ec2965cac6c115a0e102a23041ff575ca770a2f/pex-2.1.56-py2.py3-none-any.whl/pex/pex_bootstrapper.py:483: PEXWarning: The venv for /tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.tmp/tmpgnt8b5wg at /tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.cache/pex_root/venvs/58cc34e4c025a3f58981bc86a69c269d5b2d177d/ddab8011daaee380698ac2fb9701af18c90c03f6 has script shebangs of '#!/tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.cache/pex_root/venvs/s/1b7b2da4/venv/bin/python3.8 -sE' with 134 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.cache/pex_root to a shorter path as a work-around.

So it looks like this is all explained now. BuildFarm uses super long paths, which triggers this bit of Pip / distlib code which produces a non python shebang which foils Pip producing a #!python faux-shebang, which rolls all the way down hill. I'm not sure what I can do about this. But the fix will have to be over in Pex land if possible at all. I'll link an issue once / if I can think of something here.

That said, on this image, the kernel is configured for 256 so its all unfortunate and should be moot.

jsirois · 2021-11-24T18:36:54Z

Whittled down here: pex-tool/pex#1520
Still no clue how I work around this yet.

tdyas · 2021-11-24T21:26:46Z

Great debugging! I don't think I would have found that.

jsirois · 2021-11-24T22:16:59Z

The BuildFarm only was screaming from the get go, it was just hard to see what was relevantly different about BuildFarm.

jsirois · 2021-11-24T22:33:15Z

This is in fact fixed by pex-tool/pex#1521 when using ./pants_from_sources on a local pants clone modified to point to the Pex artifacts served up by tox -eserve against pex-tool/pex#1521.

This brings official support for Python 3.10 and a hermeticity fix for build environments where the named_caches directory has an especially long path name. Changelogs are here: + https://github.com/pantsbuild/pex/releases/tag/v2.1.55 + https://github.com/pantsbuild/pex/releases/tag/v2.1.56 Fixes pantsbuild#13170 # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

jsirois · 2021-11-25T02:54:59Z

I have a fix for remote-apis-testing out here: https://gitlab.com/remote-apis-testing/remote-apis-testing/-/merge_requests/271
I think I'm not allowed to do that though, but hopefully someone there notices the MR and re-creates it against the main repo.

This brings official support for Python 3.10 and a hermeticity fix for build environments where the named_caches directory has an especially long path name. Changelogs are here: + https://github.com/pantsbuild/pex/releases/tag/v2.1.55 + https://github.com/pantsbuild/pex/releases/tag/v2.1.56 Fixes #13170

tdyas added the bug label Oct 8, 2021

tdyas changed the title ~~pex shebang contains absolute path from previous remote execution sandbox~~ pex python interpreter path contains absolute path from previous remote execution sandbox Oct 8, 2021

Eric-Arellano mentioned this issue Oct 13, 2021

Coverage for python tests doesn't work correctly over remote execution #13239

Closed

jsirois self-assigned this Oct 14, 2021

jsirois mentioned this issue Nov 24, 2021

Fix wheel install hermeticity. pex-tool/pex#1521

Merged

jsirois mentioned this issue Nov 25, 2021

Upgrade to Pex 2.1.56. #13713

Merged

jsirois closed this as completed in #13713 Nov 25, 2021

huonw mentioned this issue May 25, 2023

remote-apis-testing fails with Pants v2.7.0 and Buildgrid #13066

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pex python interpreter path contains absolute path from previous remote execution sandbox #13170

pex python interpreter path contains absolute path from previous remote execution sandbox #13170

tdyas commented Oct 8, 2021

tdyas commented Oct 8, 2021

tdyas commented Oct 8, 2021

stuhood commented Oct 11, 2021

tdyas commented Oct 11, 2021 •

edited

jsirois commented Oct 14, 2021

tdyas commented Oct 14, 2021

jsirois commented Oct 14, 2021

jsirois commented Oct 14, 2021 •

edited

tdyas commented Oct 14, 2021

jsirois commented Oct 14, 2021 •

edited

tdyas commented Oct 14, 2021 •

edited

tdyas commented Oct 14, 2021 •

edited

tdyas commented Oct 20, 2021

tdyas commented Oct 20, 2021

tdyas commented Oct 20, 2021

jsirois commented Oct 20, 2021

tdyas commented Oct 20, 2021

tdyas commented Oct 20, 2021

jsirois commented Nov 23, 2021

jsirois commented Nov 23, 2021 •

edited

jsirois commented Nov 24, 2021

tdyas commented Nov 24, 2021

tdyas commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

tdyas commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 25, 2021

pex python interpreter path contains absolute path from previous remote execution sandbox #13170

pex python interpreter path contains absolute path from previous remote execution sandbox #13170

Comments

tdyas commented Oct 8, 2021

tdyas commented Oct 8, 2021

tdyas commented Oct 8, 2021

stuhood commented Oct 11, 2021

tdyas commented Oct 11, 2021 • edited

jsirois commented Oct 14, 2021

tdyas commented Oct 14, 2021

jsirois commented Oct 14, 2021

jsirois commented Oct 14, 2021 • edited

tdyas commented Oct 14, 2021

jsirois commented Oct 14, 2021 • edited

tdyas commented Oct 14, 2021 • edited

tdyas commented Oct 14, 2021 • edited

tdyas commented Oct 20, 2021

tdyas commented Oct 20, 2021

tdyas commented Oct 20, 2021

jsirois commented Oct 20, 2021

tdyas commented Oct 20, 2021

tdyas commented Oct 20, 2021

jsirois commented Nov 23, 2021

jsirois commented Nov 23, 2021 • edited

jsirois commented Nov 24, 2021

tdyas commented Nov 24, 2021

tdyas commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

tdyas commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 24, 2021

jsirois commented Nov 25, 2021

tdyas commented Oct 11, 2021 •

edited

jsirois commented Oct 14, 2021 •

edited

jsirois commented Oct 14, 2021 •

edited

tdyas commented Oct 14, 2021 •

edited

tdyas commented Oct 14, 2021 •

edited

jsirois commented Nov 23, 2021 •

edited