Skip to content

Fix RLinf uv runtime environment upload - #7568

Merged
kellyguo11 merged 2 commits into
isaac-sim:developfrom
mingxueg-nv:mingxueg/fixed_rlinf_training_startup
Sep 4, 2026
Merged

Fix RLinf uv runtime environment upload#7568
kellyguo11 merged 2 commits into
isaac-sim:developfrom
mingxueg-nv:mingxueg/fixed_rlinf_training_startup

Conversation

@mingxueg-nv

@mingxueg-nv mingxueg-nv commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Description

Disable Ray's automatic uv runtime environment for RLinf training to avoid exceeding the 512 MB working directory upload limit.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (existing functionality will not work without user modification)
  • Documentation update

Release backport

  • Backport this pull request to the active release branch after it merges into develop

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@mingxueg-nv
mingxueg-nv requested a review from a team September 4, 2026 09:40
@github-actions github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Sep 4, 2026

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The RLinf entrypoint now disables Ray's automatic uv runtime-environment upload while preserving explicit user configuration via setdefault. The user-visible startup fix is missing the required isaaclab_rl changelog fragment.

  • Design and architecture: Keeping the Ray-specific environment toggle in the RLinf entrypoint contains the workaround at the relevant integration boundary and follows the existing environment-based configuration pattern.
  • API: No CLI arguments, public symbols, or return types change. An explicitly supplied RAY_ENABLE_UV_RUN_RUNTIME_ENV value remains honored.
  • Implementation: The environment variable is set before the lazy RLinf imports and training initialization. Add a past-tense bug-fix fragment under source/isaaclab_rl/changelog.d/ as required for this user-visible package change.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

# checkpoints, which exceed Ray's 500 MiB upload limit. RLinf already
# selects the Python executable for each worker, so this upload is neither
# needed nor desirable.
os.environ.setdefault("RAY_ENABLE_UV_RUN_RUNTIME_ENV", "0")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Implementation — Missing isaaclab_rl changelog fragment

This PR changes user-visible RLinf launch behavior in source/isaaclab_rl but includes no fragment under source/isaaclab_rl/changelog.d/. Repository rules require one fragment per changed source package (a .skip fragment when no release note is needed), so this fix would be dropped from generated release notes. Add a past-tense bug-fix fragment describing the disabled automatic Ray uv runtime-environment upload; do not edit CHANGELOG.rst or extension.toml.

@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR disables Ray’s automatic uv runtime environment for RLinf training, preventing oversized Isaac Lab working-directory uploads.

  • Sets RAY_ENABLE_UV_RUN_RUNTIME_ENV to disabled before RLinf and Ray initialize.
  • Preserves explicit user-provided environment overrides.

Confidence Score: 5/5

The PR appears safe to merge with no actionable issues identified.

The new environment default is applied before RLinf and Ray initialization on the supported launch paths, preventing the unwanted upload while allowing an explicit user override.

Important Files Changed

Filename Overview
source/isaaclab_rl/isaaclab_rl/entrypoints/backends/train_rlinf.py Disables Ray’s automatic uv working-directory upload early in the RLinf launch path without introducing an actionable defect.

Reviews (1): Last reviewed commit: "Fix RLinf uv runtime environment upload" | Re-trigger Greptile

@AntoineRichard AntoineRichard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@kellyguo11

Copy link
Copy Markdown
Contributor

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 4, 2026
@kellyguo11
kellyguo11 merged commit 23c6a68 into isaac-sim:develop Sep 4, 2026
53 checks passed
@isaaclab-bot

isaaclab-bot Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Backported to release/3.0.0 as a6c727e.

isaaclab-bot Bot pushed a commit that referenced this pull request Sep 4, 2026
# Description

Disable Ray's automatic uv runtime environment for RLinf training to
avoid exceeding the 512 MB working directory upload limit.
<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html

💡 Please try to keep PRs small and focused. Large PRs are harder to
review and merge.
-->


Fixes # (issue)

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (existing functionality will not work without user
modification)
- Documentation update

## Release backport

- [x] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`

## Screenshots

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

## Checklist

Docker and GPU tests run on demand. Push the commits you want tested,
then
comment `run-ci` on the pull request.

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package (do **not** edit
`CHANGELOG.rst` or bump `extension.toml` — CI handles that)
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

(cherry picked from commit 23c6a68)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants