[CI] Cut ARM CI cold start by persisting Isaac Sim caches across runs - #6776
Draft
hujc7 wants to merge 2 commits into
Draft
[CI] Cut ARM CI cold start by persisting Isaac Sim caches across runs#6776hujc7 wants to merge 2 commits into
hujc7 wants to merge 2 commits into
Conversation
The ARM CI job ran a dedicated "arm_ci OVPhysX marker tests" step that cost ~13.5 min of wall clock for ~6.1 min of test time. The gap is a second renderer cold start: run-tests provisions a fresh mktemp cache directory per step, so the step launched Isaac Sim against an empty Kit/OptiX shader cache and stalled ~6 min before its first test. Those params only duplicate coverage the x86 jobs already provide, so remove the step. The remaining marker step keeps `-k "not ovphysx"`, which is now the sole selector excluding those params. Also drop ovphysx from the step's pip install: with the params deselected, the only unguarded `import ovphysx` sites live in source/isaaclab_ovphysx/test/, and tools/conftest.py text-scans for `pytest.mark.arm_ci` before running pytest, so those files never load. Measured on run 30388296334: job goes from ~29.7 min to ~15.9 min.
Every run-tests invocation provisioned its cache directories under a fresh mktemp dir and deleted them afterwards, so Isaac Sim launched against empty Kit, OptiX and asset caches on every step of every run. That warm-up cost ~6 min of silence before the first rendering test reported, and was thrown away immediately. Measured on run 30388296334: the first rendering test in a container took 421 s, while the next file's first test in that same container -- a separate process, a separate renderer -- took 72 s. The following step, with a fresh mktemp dir, paid 370 s again. Add an opt-in persistent-cache-key input. When set, the shader, JIT and asset caches bind-mount from a stable per-key directory that outlives the run; logs, config and data stay per-run, since persisting those would leak state between runs without saving any work. ARM CI keys on the Isaac Sim image tag so a base-image bump starts cold. Concurrency uses a non-blocking flock: a job that cannot take the lock falls back to a per-run cache rather than interleaving writes into a shared Kit cache or waiting out its timeout. Stale cache roots are evicted on a 14-day TTL, mirroring the deps-tag eviction in .github/actions/docker-build. Expected: ~15.9 min to ~10 min, with no test coverage removed.
hujc7
force-pushed
the
jichuanh/arm-ci-persistent-shader-cache
branch
from
July 29, 2026 05:54
f0cc5f6 to
f065bfd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR series map
developStacked on #6775 — [CI] Halve ARM CI runtime by dropping the duplicate OVPhysX step. Both parts target
developbecause the series is cross-fork and a base branch cannot live on the fork, so this PR's diff contains Part 1's commit as well. Use the Part 2 compare link above to review this change in isolation.Summary
run-testsinvocation provisioned Isaac Sim's cache directories under a freshmktempdir and deleted them afterwards, so every step of every run launched Isaac Sim against empty Kit, OptiX and asset caches.persistent-cache-keyinput that bind-mounts those caches from a stable per-key directory outliving the run. ARM CI keys on the Isaac Sim image tag.1. Evidence the cost is cache-resident
From run 30388296334:
mktemp)That third row is the direct proof: the expensive state is filesystem-resident and currently discarded between every step.
2. What changed
.github/actions/run-tests/action.ymlsplits the scratch dir by lifetime:kit/cache,/isaac-sim/.cache,.nv/ComputeCache(both/isaac-simand$HOME),$HOME/.cachelogs,config,data,pkg,kit/data,kit/logsOnly derived artifacts persist; persisting logs/config/data would leak state between runs without saving any work.
Two details worth flagging for review:
$HOMEis mounted wholesale at/tmp/isaaclab-ci-home, so persisting only its.cachemeans mounting over a path inside an existing mount. Docker sorts bind mounts by destination depth, so the nested mounts land on top rather than being shadowed — they are ordered after the parent accordingly./isaac-sim/.nv/ComputeCacheand$HOME/.nv/ComputeCache) because which one CUDA writes depends onHOME, which this action overrides. Persisting one alone would leave half the JIT cache cold.Concurrency: a non-blocking
flock. A job that cannot take the lock falls back to a per-run cache rather than interleaving writes into a shared Kit cache or waiting out its 60-minute timeout — slower, never wrong, and identical to today's behaviour.Disk growth: stale cache roots are evicted on a 14-day TTL, mirroring the deps-tag eviction already in
.github/actions/docker-build. The active cache istouched so it is never swept while in use, and its size is reported in the cleanup log group.Blast radius: the whole cache-mount block is inside
if [ -n "$volume_mount_source" ], andvolume-mount-sourceis passed only byarm-ci.yml. x86 jobs bake source into the image and never enter this branch. The input also defaults to''(no persistence), so unkeyed callers keep today's exact behaviour.3. Verification
The shell logic was extracted from the action and exercised directly, each case in its own process:
Also:
uv run isaaclab -fpasses;bash -non the embedded script passes; both YAML files parse;python3 tools/changelog/cli.py check developpasses (CI-only change).Not yet verified on hardware: the actual saving on a Spark runner, and which of the persisted directories holds the bulk of the warm-up. The first green run on this branch will show both — worth confirming before merge.
4. Related
#6707 — Save warp cache for CI covers a different cache with the same intent; worth coordinating so the two do not diverge on where CI caches live.