Error building pants target due to missing cache file

[mattalbr]

Filing a bug per instructions in https://www.pantsbuild.org/2.3/docs/using-pants/troubleshooting-common-issues#cache-or-pantsd-invalidation-issues

vscode ➜ /workspace (my_branch) $ pants test ::
22:05:08.32 [INFO] Canceled: Building 17 requirements for requirements.pex from the 3rdparty/python/beam.lock resolve: apache-beam[gcp]==2.61.0, ddtrace==2.17.0, enum-tools==0.12.0, fastapi==0.115.5, frozendict==2.4.6, google-aut... (241 characters truncated)
22:05:09.14 [INFO] Completed: Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock
22:05:09.80 [INFO] Completed: Building 16 requirements for requirements.pex from the 3rdparty/python/beam.lock resolve: apache-beam[gcp]==2.61.0, ddtrace==2.17.0, enum-tools==0.12.0, fastapi==0.115.5, frozendict==2.4.6, google-aut... (226 characters truncated)
22:05:09.80 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

ProcessExecutionFailure: Process 'Building 16 requirements for requirements.pex from the 3rdparty/python/beam.lock resolve: apache-beam[gcp]==2.61.0, ddtrace==2.17.0, enum-tools==0.12.0, fastapi==0.115.5, frozendict==2.4.6, google-auth==2.30.0, google-cloud-secret-manager==2.20.0, httpx==0.27.0, once-py==0.0.5, pgvector==0.2.5, psycopg[binary,pool]==3.1.19, pydantic==1.10.15, requests==2.32.3, sqlalchemy==2.0.30, strawberry-graphql[cli]==0.260.0, us==3.1.1' failed with exit code 1.
stdout:

stderr:
[Errno 2] No such file or directory: '/home/vscode/.cache/pants/named_caches/pex_root/downloads/1d6751138975dee8c817b4db5e90885105178926d4248fd4cf4c0daa2d287e01/apache_beam-2.61.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl' -> '/tmp/pants-sandbox-LlFZHz/.tmp/tmpu82r5g7a/.deps/apache_beam-2.61.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl'



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

If it's useful, I run this in a docker container, and running docker system prune -af seemed to resolve the error. I'm not sure if that had the effect of clearing the cache, or if it just freed up disk space and this strange error comes as a result of insufficient disk space.

[huonw]

Sorry for the trouble.

Is this a long-lived docker container, where you'd run several commands previously? Or is it short lived, where this is the first (or nearly the first) command?

[mattalbr]

All good!

It's usually relatively long-lived, I probably rebuild once a week or so? I can't remember if I had rebuilt before this, but I doubt it.

[benjyw]

Is this running in docker on MacOS? If so, I wonder if it's related to the VirtioFS issues that also affected #21771 (comment)

[mattalbr]

It is! Definitely possible. FWIW I've only had this pop up once and have never reproduced it

[benjyw]

Well, that's good, sort of.

If it does happen again and you're able to consistently reproduce it, then I guess one thing to try is to switch Docker Desktop from VirtioFS to gRPC FUSE or osxfs and see what happens. But if it doesn't reproduce reliably then that's of limited use as a diagnostic.

[mattalbr]

Just as a heads up I ran into this again on a GCP cloud workstation (so not on MacOS) when upgrading to 2.30.0, and restarting pantsd (and the whole workstation) didn't resolve it.

EDIT: Nuking the cache via rm -rf /home/user/.cache/pants/named_caches/pex_root also didn't resolve it, so I'm out of ideas.

user@matt-workstation:~/gaia$ pants package backend:fastapi
18:01:26.47 [WARN] /home/user/.cache/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.30.0/lib/python3.11/site-packages/pants/init/options_initializer.py:14: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

/home/user/.cache/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.30.0/lib/python3.11/site-packages/pants/init/options_initializer.py:14: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
18:01:31.02 [WARN] /home/user/.cache/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.30.0/lib/python3.11/site-packages/pants/engine/internals/rule_visitor.py:379: UserWarning: Get() is deprecated, and will be removed in Pants 2.31.0. Found a `Get() in /home/user/gaia/release/pants-plugins/gaia_gcloud/cloudrun.py:121. See https://www.pantsbuild.org/2.30/docs/writing-plugins/the-rules-api/migrating-gets for how to migrate your plugins to use the new call-by-name idiom.
  warnings.warn(

18:01:31.02 [WARN] /home/user/.cache/nce/5c54a0c25255df2afb0525da856d49eb81b48cff0e32a9c868f77a296af24319/bindings/venvs/2.30.0/lib/python3.11/site-packages/pants/engine/internals/rule_visitor.py:379: UserWarning: Get() is deprecated, and will be removed in Pants 2.31.0. Found a `Get() in /home/user/gaia/release/pants-plugins/gaia_gcloud/cloudrun.py:142. See https://www.pantsbuild.org/2.30/docs/writing-plugins/the-rules-api/migrating-gets for how to migrate your plugins to use the new call-by-name idiom.
  warnings.warn(

18:01:31.05 [INFO] Initializing scheduler...
18:01:31.37 [INFO] Scheduler initialized.
18:01:32.28 [WARN] Unmatched globs from backend/routers/local:tests's `sources` field: ["backend/routers/local/*_test.py", "backend/routers/local/test_*.py", "backend/routers/local/tests.py"]

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to https://www.pantsbuild.org/troubleshooting#pants-cannot-find-a-file-in-your-project.
18:01:32.44 [WARN] Unmatched globs from scripts:tests's `sources` field: ["scripts/*_test.py", "scripts/test_*.py", "scripts/tests.py"]

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to https://www.pantsbuild.org/troubleshooting#pants-cannot-find-a-file-in-your-project.
18:01:38.36 [INFO] Completed: Scheduling: Find interpreter for constraints: CPython==3.11.*
18:01:38.36 [INFO] Completed: Scheduling: Building local_dists.pex
18:01:52.18 [INFO] Completed: Building 90 requirements for backend/fastapi.pex from the 3rdparty/python/default.lock resolve: alembic-postgresql-enum==1.8.0, alembic==1.17.0, anyio==4.11.0, authlib==1.6.5, bqredis==0.0.8, cachetoo... (1851 characters truncated)
18:01:52.18 [INFO] Completed: Scheduling: Building 90 requirements for backend/fastapi.pex from the 3rdparty/python/default.lock resolve: alembic-postgresql-enum==1.8.0, alembic==1.17.0, anyio==4.11.0, authlib==1.6.5, bqredis==0.0... (1863 characters truncated)
18:01:52.19 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building 90 requirements for backend/fastapi.pex from the 3rdparty/python/default.lock resolve: alembic-postgresql-enum==1.8.0, alembic==1.17.0, anyio==4.11.0, authlib==1.6.5, bqredis==0.0.8, cachetools==5.5.2, cryptography==46.0.0, datadog-api-client==2.44.0, datadog==0.52.1, ddtrace==2.21.11, deprecated==1.2.18, email-validator==2.3.0, enum-tools==0.13.0, fakeredis==2.32.0, fastapi==0.120.2, fhirclient==4.3.2, firebase-admin==6.9.0, frozendict==2.4.6, google-api-core==2.26.0, google-auth==2.41.1, google-cloud-bigquery==3.38.0, google-cloud-logging==3.12.1, google-cloud-pubsub==2.31.1, google-cloud-secret-manager==2.25.0, google-cloud-storage==2.19.0, google-cloud-tasks==2.20.0, graphql-core==3.2.6, hishel==0.1.5, httpcore==1.0.9, httpx-retries==0.4.5, httpx==0.28.1, humancursor-playwright==0.1.1, jinja2==3.1.6, jsonpath-ng==1.7.0, jwcrypto==1.5.6, limits==5.6.0, lxml==6.0.2, more-itertools==10.8.0, once-py==0.0.8, openai==2.6.0, pandas-stubs==2.3.2.250926, pandas==2.3.3, pgvector==0.4.1, phonenumbers==9.0.16, playwright==1.56.0, polars==1.30.0, portabletext-html==1.1.3, protobuf==4.25.8, psycopg-binary==3.2.11, psycopg-pool==3.2.6, psycopg==3.2.11, pyarrow==18.1.0, pydantic-core==2.23.4, pydantic-to-pyarrow==0.1.6, pydantic-xml==2.18.0, pydantic==2.9.2, pyjwt==2.10.1, pyotp==2.9.0, python-dateutil==2.9.0.post0, pytz==2025.2, pyyaml==6.0.3, redis==5.3.1, requests==2.32.5, sendgrid==6.12.5, slack-sdk==3.37.0, sqlakeyset==2.0.1746777265, sqlalchemy-bigquery==1.15.0, sqlalchemy-file==0.6.0, sqlalchemy==2.0.44, starlette-admin==0.15.1, starlette-context==0.4.0, starlette==0.49.1, strawberry-graphql==0.284.1, strawberry-sqlalchemy-mapper==0.7.0, streamlit==1.51.0, svix==1.56.0, tenacity==9.1.2, twilio==9.8.4, types-cachetools==6.2.0.20250827, types-deprecated==1.2.15.20250304, types-lxml==2025.8.25, types-protobuf==6.32.1.20250918, types-python-dateutil==2.9.0.20251008, types-pytz==2025.2.0.20250809, types-pyyaml==6.0.12.20250915, types-requests==2.32.4.20250913, unidecode==1.4.0, us==3.2.0, uvicorn==0.38.0, zeep==4.3.2' failed with exit code 1.
stdout:

stderr:
[Errno 2] No such file or directory: '/home/user/.cache/pants/named_caches/pex_root/installed_wheels/2/3660dcbf912f696291b6c4b46e793e0960e57160e1be3b4b9192696a1cf26382/ddtrace-2.21.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.41950cfb8bf04c69a4d938914b7ada53.work/ddtrace/vendor/psutil/_psutil_aix.c' -> '/home/user/.cache/pants/named_caches/pex_root/installed_wheels/2/3660dcbf912f696291b6c4b46e793e0960e57160e1be3b4b9192696a1cf26382/ddtrace-2.21.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.41950cfb8bf04c69a4d938914b7ada53.work/ddtrace/vendor/psutil/_psutil_aix.c'



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

[jsirois]

@mattalbr this is unrelated despite appearances. This is because:

jsirois@siroisdesign:~$ zipinfo ddtrace-2.21.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | grep aix.c
jsirois@siroisdesign:~$ unzip -qc ddtrace-2.21.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl ddtrace-2.21.11.dist-info/RECORD | grep aix.c
ddtrace/vendor/psutil/_psutil_aix.c,sha256=fEoiu23Up7TxlsiUfeSWSubKRAO8te5we63MBZ6vOWI,31034
ddtrace/vendor/psutil/arch/aix/common.h,sha256=eLKtDGdfcV1kC_h2Wl4dc0kg2wr-qKs5WKIAoKiUx_o,894
ddtrace/vendor/psutil/arch/aix/common.c,sha256=zwuWitDFSTJzYjZVWWqGhrpyhueVxK5Yf0YU1KtCRzc,2284
jsirois@siroisdesign:~$

I.E.: the RECORD lies. I will never understand why PyPI doesn't block upload of wheels with bad metadata like this.

Pex switched to RECORD-driven wheel installs in order to support --venv-repository (In order to support Pants transitioning to uv, which looks like will never happen). I was stupid enough to make the mistake of trusting Python ecosystem metadata for the 18th time.

I'll file a Pex issue and link here as soon as possible, but the fix may take longer than usual during the Christmas holiday.

[mattalbr]

Thank you @jsirois!

[jsirois]

@mattalbr I doubt the actual bad metadata gets fixed, but, FYI: DataDog/dd-trace-py#15779

[benjyw]

Side note: I am working on Pants using uv (for lockfile generation and a lot more) right now (although I will be off for a couple of weeks for the holidays).

[jsirois]

@mattalbr the Pex fix is now available in 2.75.1 here: https://github.com/pex-tool/pex/releases/tag/v2.75.1

You'd add this to your pants.toml:

[pex-cli]
version = "v2.75.1"
known_versions = [
  "v2.75.1|macos_x86_64|053ae07f0980acb7003b516dcab8ae9393386cd4447abd4885b798162da64655|4944782",
  "v2.75.1|macos_arm64|053ae07f0980acb7003b516dcab8ae9393386cd4447abd4885b798162da64655|4944782",
  "v2.75.1|linux_x86_64|053ae07f0980acb7003b516dcab8ae9393386cd4447abd4885b798162da64655|4944782",
  "v2.75.1|linux_arm64|053ae07f0980acb7003b516dcab8ae9393386cd4447abd4885b798162da64655|4944782"
]

[jsirois]

@gdfast I pinged you on the Datadog ticket just in case you hold sway.