ci: serialize uv installs against the shared node-local cache

[sbryngelson]

Description

Fixes CI failures like the one seen on #1414's Frontier gpu-acc [2/2] job:

error: Failed to install: pandas-3.0.3-...whl (pandas==3.0.3)
  Caused by: failed to open file `.../pandas-3.0.3.dist-info/METADATA`: No such file or directory (os error 2)

Root cause

toolchain/bootstrap/python.sh redirects UV_CACHE_DIR to a node-local, per-user path ($TMPDIR/uv-cache-$USER) on self-hosted GitHub Actions runners (added in #1385 to dodge an NFS file-lock error, os error 524, when ~/.cache/uv lives on shared NFS $HOME).

That path is keyed only by username, not by job/run/matrix-leg. The self job in test.yml runs several Frontier/Frontier-AMD matrix legs (acc/omp/cpu x shards) whose "Fetch Dependencies" step runs directly on the shared login node, as the same OS user, at the same time — all pointed at the identical cache directory. uv's own lock protects individual cache entries, but concurrent installs from separate uv processes can still race while one extracts/prunes the shared archive-v0 store, leaving behind a corrupted entry (like a missing dist-info/METADATA file) that fails every subsequent install until someone manually clears the cache.

Fix

Serialize the actual uv pip install call with flock (falls back to running unlocked if flock isn't available, e.g. stock macOS) so only one process touches a given cache directory at a time, while keeping the cache itself shared and warm across runs — no loss of the caching benefit #1385 was going for, just closes the intra-node concurrency gap it didn't address.

Testing

• bash -n on the modified script (syntax check)
• Local smoke test: launched 3 concurrent invocations of the new uv_install-style wrapper against a shared lock file and confirmed they run strictly serialized (each start/end pair completes before the next begins), rather than interleaving.
• Have not yet reproduced the actual corruption race end-to-end on Frontier (hard to force deterministically); opening as draft pending a live CI run against this branch.

Type of change

• Bug fix (CI infra)

Checklist

• Tested locally (script syntax + lock-serialization behavior)
• Verified fix on an actual Frontier CI run (pending — draft)

[github-actions]

Claude Code Review

Head SHA: f3ff276

Files changed:

• 1
• toolchain/bootstrap/python.sh

Findings:

• toolchain/bootstrap/python.sh:188-206: The UV_CACHE_DIR redirect just above is explicitly gated on [ "${GITHUB_ACTIONS:-}" = "true" ] && [ -w "${TMPDIR:-/tmp}" ], but the new UV_INSTALL_LOCK/uv_install wrapper is unconditional — it always serializes uv pip install via flock for every invocation (any local dev machine, any CI job), not just the self-hosted HPC case described in the comment, and it never checks that ${TMPDIR:-/tmp} is writable before pointing flock at a lock file there. If TMPDIR is set to a directory that doesn't exist or isn't writable, flock will fail to open/create mfc-uv-install-${USER}.lock and uv_install will return non-zero without ever invoking uv pip install, so the script reports "Installation failed" (and dumps a log) even though the actual uv pip install was never attempted. This is a regression for any environment where TMPDIR is non-writable, since prior to this change the same environment installed fine (no lock file was created).
• toolchain/bootstrap/python.sh:201: The lock filename only encodes $USER, not the repository path/build dir, so two unrelated concurrent invocations of mfc.sh by the same user (e.g. separate git worktrees/clones of MFC building at once) will now serialize against each other through /tmp/mfc-uv-install-$USER.lock even though they don't share a UV_CACHE_DIR (that redirect is CI-only) — a behavior change beyond the stated self-hosted-runner-cache-corruption fix.

[Copilot]

Pull request overview

This PR addresses intermittent CI failures caused by concurrent uv pip install processes racing on a shared per-user uv cache directory (notably on self-hosted runners), by serializing the install operation with a file lock.

Changes:

• Introduces a uv_install wrapper that uses flock (when available) to serialize uv pip install against a per-user lock file.
• Uses a TMPDIR-based lock location with a fallback to /tmp when TMPDIR is not writable.
• Replaces direct uv pip install invocations with the new uv_install wrapper in both verbose and non-verbose paths.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.43%. Comparing base (804c3bf) to head (b9c8665).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1630   +/-   ##
=======================================
  Coverage   60.43%   60.43%           
=======================================
  Files          83       83           
  Lines       19871    19871           
  Branches     2956     2956           
=======================================
  Hits        12010    12010           
  Misses       5860     5860           
  Partials     2001     2001           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.