Skip to content

uv-manager 0.8.0

Latest

Choose a tag to compare

@glentner glentner released this 03 Nov 03:16
· 2 commits to main since this release
0.8.0
8d23555

A job script can now call an installed tool by its own name, or uv run a script, or uvx a tool, on a purged scratch tree with UVM_REPAIR set. One rank repairs what the purge took while every other rank waits, then they all run.

0.7.0 repaired the managed interpreters and tool environments inside the job, but two job-script routes were outside it. A tool installed with uv tool install and run as srun ruff never entered the wrapper, so it got no check at all. And uv run script.py and uvx <spec> run from uv's own cache, which the repair never looked at. 0.8.0 closes both.

Installed tools, by name

uv tool install ruff                      # once, on a login node
# in the job script:
export UVM_REPAIR=1
srun ruff check .
  • Durable records. Every uv tool install now leaves a small plain-file record under the new UVM_DATA (default ~/.local/share/uvm, a few KB per tool), off the purged scratch tree.
  • Trampolines in $UVM_DATA/bin. A recorded tool's trampoline lives there, and the module puts $UVM_DATA/bin first on PATH, ahead of $UVM_ROOT/bin.
  • Restore once. Under UVM_REPAIR, a call by the tool's name runs the same scope check uvx gets. A tool whose directory, receipt or shim a purge took, a git-sourced one included, is restored once from its record, under the repair lock, while the other ranks wait. A restore never leaves a tool pinned against uv tool upgrade if it wasn't pinned before.

Upgrading from 0.7.0.

  • The first uv call under 0.8.0 writes the new trampolines and rewrites each 0.7.0 trampoline in $UVM_ROOT/bin in place, so a running job keeps finding it.
  • Run uv-manager doctor once after upgrading. It writes a record for every tool installed under 0.7.0, and the trampolines for them.
  • A PATH entry naming $UVM_ROOT/bin by hand gains $UVM_DATA/bin ahead of it.
  • The README no longer says uv-manager writes nothing under $HOME: UVM_DATA does, unless you point it elsewhere.

uv's cache

export UVM_REPAIR=1
srun uv run --no-project analyse.py       # a PEP 723 script
srun uvx --from tqdm tqdm --version       # an ephemeral tool environment
  • What's checked. UVM_REPAIR and uv-manager doctor now check the environments uv run <script> and uvx <spec> (and --with) run from, and the unpacked wheels they are built from, against the files each RECORD lists. Doctor names each damaged entry, and stays read-only.
  • Script environments. A damaged script environment has only its damaged distributions reinstalled, in place. A process already running from it keeps importing.
  • Cached environments and wheels. A damaged cached environment whose distributions can't all be shown to come from the configured index loses its link, keeping its archive, and the next call rebuilds it from its own spec. A damaged wheel is removed the same way, by its link or uv's pointer to it.
  • Source check. Before a reinstall pins a distribution by version, the repair checks that the configured index serves that release, and for a wheel that its Python source matches.
  • Fail loudly rather than swap code. Where the source can't be shown, for example a fork from an index the script names, a +local build, or a find-links package, the scope fails with a reason that names the distribution. uv-manager clean cache --yes clears it.
  • Cost. One listing of the cache's live entries per scope. On a Lustre scratch a 40,000-RECORD-line data-science cache took 2.5–4.4 s, and a purge that damages hundreds of entries adds little.

Measured

tests/repair-race.sh under srun, 2 nodes × 64 ranks, on Lustre scratch:

Scenario Result
Installed tools purged, called by name 128/128 ran, 1 restore, lock held 9 s
…restorer killed 127/127 waiters failed naming the lock; clean pass held 12 s
uvx environment damaged 128/128 ran, 1 repair, lock held 3 s, nothing rebuilt
Script environment damaged 128/128 ran, 1 repair, lock held 4 s, nothing rebuilt
…repairer killed 127/127 waiters failed naming the lock; clean pass held 3 s

The raw summaries are in spec/installed-tool-parity/evidence/ and spec/uv-cache-purge-damage/evidence/.

What it doesn't do (yet)

  • A fork of a package that differs from the configured index only in compiled extensions, or that is served as an sdist, at the same version, isn't detected.
  • A package that uv built from source is rebuilt a second time to check that the index serves it, so its repair holds the lock longer (about 110 s for numpy on a workstation, against the default 180 s UVM_LOCK_TIMEOUT).
  • A dist-info directory removed whole, packages an earlier run installed into a managed interpreter, and UV_LINK_MODE=symlink are not checked. The README lists them.

Deploying

Sites deploy by git clone; git clone and rsync -a keep the bin/ symlinks. Load the module, and UVM_DATA is set beside UVM_ROOT.