Skip to content

feat(gym): apply effort-level reward shaping in the SingleController - #3770

Merged
yuki-97 merged 4 commits into
mainfrom
ruit/v2_2770_effort_level_gym
Aug 25, 2026
Merged

feat(gym): apply effort-level reward shaping in the SingleController#3770
yuki-97 merged 4 commits into
mainfrom
ruit/v2_2770_effort_level_gym

Conversation

@RayenTian

@RayenTian RayenTian commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

PR #2770 only wired length-based effort reward shaping into the run_grpo.py (v1) synchronous paths. The SingleController (v2) path runs its own NeMo-Gym rollout via RolloutManager/AsyncNemoGymRolloutImpl, which never applied it.

  • Thread EffortLevelsConfig through RolloutManager into AsyncNemoGymRolloutImpl and call the shared _apply_effort_shaping before completions are built, so the shaped reward flows into Completion.reward.
  • Fold the effort length/reward metrics into _compute_rollout_metrics.
  • setup.py resolves the config via the shared _get_effort_config helper.

No behaviour change when env.nemo_gym.effort_levels is unset (_get_effort_config returns None -> shaping is a no-op).

result

image image

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants