[CI] Retry transient Python package downloads

[StafaH]

Description

Retries CI Python package installs up to three times with a three-second delay to tolerate transient PyPI connectivity failures.

Validation

• uv run isaaclab -f
• Shell syntax and GitHub workflow YAML validation

[greptile-apps]

Greptile Summary

This PR adds a shared fixed-attempt command wrapper and applies it to Python package installation steps across workflows and composite actions to tolerate transient download failures.

• Retries failed install commands up to three times with a three-second delay while preserving the final exit status.
• Covers pip and uv setup in documentation, wheel, licensing, test, image-validation, and multi-GPU CI paths.
• Mounts the helper read-only into containers where package installation occurs inside Docker.

Confidence Score: 5/5

The PR appears safe to merge; no concrete changed-code failure was identified in the retry helper or its current CI integrations.

The wrapper preserves command arguments and final exit codes, current workflows make the helper available before use, container paths resolve to explicit read-only mounts, and the retried installation paths are restart-safe or self-healing.

Important Files Changed

Filename	Overview
.github/actions/_lib/retry-command.sh	Adds a bounded retry loop that preserves command arguments and propagates the final failure status.
.github/actions/run-tests/run_tests.sh	Mounts the retry helper into the test container and applies it to pip and uv dependency setup without changing existing package argument expansion.
.github/actions/multi-gpu/multi_gpu_shard_runner.sh	Wraps shard dependency installation; the existing working-directory setup ensures the relative helper path resolves correctly.
.github/actions/validate-kitless-image/action.yml	Mounts the helper read-only into the validation container and retries its pytest installation.
.github/workflows/test-fabric-multi-gpu.yaml	Retries the complete Isaac Lab installation, whose installation operations are restart-safe or recoverable after partial package failures.
.github/workflows/test-multi-gpu.yaml	Applies the same bounded retry behavior to the multi-GPU Isaac Lab installation.
.github/workflows/wheel.yml	Retries uv bootstrap, pip upgrade, and dry-run wheel dependency resolution while retaining existing command options.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[CI package installation step] --> B[retry-command.sh]
  B --> C[Run pip, uv, or Isaac Lab installer]
  C -->|Success| D[Continue CI job]
  C -->|Failure and attempts remain| E[Wait three seconds]
  E --> C
  C -->|Third failure| F[Return final command status]

Loading

_{Reviews (1): Last reviewed commit: "[CI] Retry transient Python package down..." | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The shared retry wrapper consistently retries CI package-install commands and preserves final exit status, but one composite action locates the helper through the caller’s workspace rather than its own action directory, reducing action portability.

• Design and architecture: Centralizing retry behavior in .github/actions/_lib/retry-command.sh avoids duplicated workflow logic. However, upload-omni-github-test-results should resolve this repository-owned helper relative to $GITHUB_ACTION_PATH, as it already does for its converter and schema resources.
• API: No Python or CLI API changes are introduced. The upload-results composite action’s consumption contract is affected: using $GITHUB_WORKSPACE assumes Isaac Lab is checked out at the workspace root and can fail when the action is consumed externally or checkout uses a custom path.
• Implementation: The wrapper correctly forwards arguments with "$@", captures failures, retries up to three times, and returns the final status. Update the upload-results action to invoke $GITHUB_ACTION_PATH/../_lib/retry-command.sh so its helper follows the action package rather than the caller’s checkout layout.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🔵 Suggestion · Design Architecture — Action resolves helper outside its action path

This action resolves every other resource through $GITHUB_ACTION_PATH (converter script and schema) and otherwise never reads the workspace. Sourcing the retry helper via $GITHUB_WORKSPACE/.github/actions/_lib/... introduces a new assumption that an IsaacLab checkout exists at the workspace root, which breaks if the action is consumed from another repo or with actions/checkout path:. Prefer $GITHUB_ACTION_PATH/../_lib/retry-command.sh.

[nvsekkin]

overall the logic is clean and the separate file makes it easy to update if we needed to.

[nvsekkin]

@nv-apoddubny had mentioned adding a delay, if readonly retry_delay_seconds=3 is good with him I do not have any objections

[nvsekkin]

should we add the same retry to build.sh @StafaH ? https://github.com/isaac-sim/IsaacLab/actions/runs/32083873111/job/95552207335?pr=7144#step:5:77 - I see that this guy failed with a similar problem

[AntoineRichard]

AI Review (Codex)

This review was generated by OpenAI Codex and verified against PR head 1f36f71.

Important findings

1. The CI helper does not retry failed install commands.

   .github/actions/_lib/with-python-package-retries.sh:14-20 only sets PIP_RETRIES and UV_HTTP_RETRIES before executing the command once. This does not match the PR description of up to three command attempts. The current isaaclab_newton and rendering-correctness jobs demonstrate the gap: each pip process exhausted all 12 internal HTTP retries on repeated files.pythonhosted.org 502 responses, then the job failed without another command attempt.

   Please define a bounded total retry budget. One option is a command-level loop with fewer internal HTTP retries; simply nesting three attempts around 12 HTTP retries could make one package failure take roughly 24 minutes. A CI wheel cache or wheelhouse would be more robust for these bootstrap dependencies.

2. The PR removes PhysX launch coverage for two Franka-soft tasks.

   source/isaaclab_tasks/test/core/test_environments_isaacsim_physx.py:28-29 adds Isaac-Lift-Soft-Franka and Isaac-Lift-Soft-Franka-Camera to _COVERED_TASKS. However, the referenced rendering path constructs only FrankaSoftCameraEnvCfg and explicitly skips physics_backend == physx. The non-camera task has no replacement test, so neither exclusion is actually covered for this PhysX path.

   Please remove these unrelated exclusions, or add targeted replacement coverage. If a known teardown issue requires suppression, use an explicit skip with the bug reference rather than marking the tasks as covered.

CI diagnosis

• isaaclab_newton: pytest-mock metadata download failed after persistent HTTP 502 responses; tests never started.
• rendering-correctness: pytetwild metadata download failed after persistent HTTP 502 responses; tests never started.
• record-video: four tests passed; test_multiple_recorders_simultaneous narrowly missed the motion threshold (0.096 < 0.1). This test is unchanged by the PR and appears flaky/unrelated.

Verification

• Focused CLI tests: 78 passed, 1 skipped.
• ./isaaclab.sh -f: all hooks passed.
• Changed shell scripts passed bash syntax validation.
• git diff --check passed.

GPU and rendering tests were not reproduced locally.

Fixed