Skip to content

feat: GSPO (w/ CI fixes)#976

Merged
terrykong merged 38 commits intomainfrom
pjin/gspo-algo-retry
Aug 27, 2025
Merged

feat: GSPO (w/ CI fixes)#976
terrykong merged 38 commits intomainfrom
pjin/gspo-algo-retry

Conversation

@pjin-nvidia
Copy link
Contributor

@pjin-nvidia pjin-nvidia commented Aug 25, 2025

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

ertkonuk and others added 30 commits August 19, 2025 14:41
@ertkonuk @pjin-nvidia
Added gspo
d5731a8 
Signed-off-by: Tugrul Konuk <tkonuk@nvidia.com>
@pjin-nvidia
Port GSPO ratio clip from dev branch.
e6ba491 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Micro batch size.
ce337d8 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge branch 'main' into pjin/gspo-algo
ddc6137 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Example GSPO config.
76c842d 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Cleaner impl.
8b65703 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
GSPO version of deepscaler example (WIP).
7938879 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
fef9120 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Alternative 2-node TP=2 deepscaler GSPO config (WIP).
9782097 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Remove Qwen3 GSPO config (will take too long to test).
95867ed 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
4d0568d 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Rename sequence_level_importance_sampling => sequence_level_importanc…
c536aff 
…e_ratios.

Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
2cbc0ca 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Using the new deepscaler config.
5e384dc 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Restore comment.
8882bfe 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Restore comment.
b795e65 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
808953e 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
446b44d 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
b9127f4 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
b6598df 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Minor fix.
931b744 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Remove inaccurate comment.
b90ea62 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
e87d144 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
GSPO config test.
dfbd89d 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
825f8a7 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
15b234c 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into HEAD
3d80fb0 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Batch size = 1 GSPO unit tests (should be equivalent to GRPO).
d95dea0 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Clipped PG loss unit tests support for batch size > 1.
c957966 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
GSPO unit tests.
a28c974 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo
969cc88 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Loop var name. Format.
c69ae82 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo-retry
a752b5b 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
GSPO test needs GRPO base config.
5f2e53b 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Move Megatron Qwen2.5 7B run test from nightly to release.
f4970a5 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia
Merge remote-tracking branch 'origin/main' into pjin/gspo-algo-retry
d43fe09 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@pjin-nvidia pjin-nvidia marked this pull request as ready for review August 26, 2025 20:53
@pjin-nvidia
Correct nightly GSPO run test.
16258e3 
Signed-off-by: Peter Jin <pjin@nvidia.com>
terrykong
terrykong previously approved these changes Aug 26, 2025
View reviewed changes
@terrykong terrykong added this pull request to the merge queue Aug 26, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 27, 2025
@pjin-nvidia
Base config needs the added field.
abe4f91 
Signed-off-by: Peter Jin <pjin@nvidia.com>
@terrykong terrykong enabled auto-merge August 27, 2025 00:39
@terrykong terrykong added this pull request to the merge queue Aug 27, 2025
Merged via the queue into main with commit 97b6c31 Aug 27, 2025
21 checks passed
@terrykong terrykong deleted the pjin/gspo-algo-retry branch August 27, 2025 03:47
soodoshll pushed a commit to soodoshll/RL that referenced this pull request Aug 28, 2025
@pjin-nvidia @ertkonuk @soodoshll
feat: GSPO (w/ CI fixes) (NVIDIA-NeMo#976)
bdaf889 
Signed-off-by: Tugrul Konuk <tkonuk@nvidia.com>
Signed-off-by: Peter Jin <pjin@nvidia.com>
Co-authored-by: Tugrul Konuk <tkonuk@nvidia.com>
Signed-off-by: Qidong Su <qidongs@nvidia.com>
skirdey-inflection pushed a commit to skirdey-inflection/RL that referenced this pull request Aug 30, 2025
@pjin-nvidia @ertkonuk @skirdey-inflection
feat: GSPO (w/ CI fixes) (NVIDIA-NeMo#976)
6cc3c4c 
Signed-off-by: Tugrul Konuk <tkonuk@nvidia.com>
Signed-off-by: Peter Jin <pjin@nvidia.com>
Co-authored-by: Tugrul Konuk <tkonuk@nvidia.com>
Signed-off-by: Stanislav Kirdey <stan@inflection.ai>
soodoshll pushed a commit to soodoshll/RL that referenced this pull request Sep 4, 2025
@pjin-nvidia @ertkonuk @soodoshll
feat: GSPO (w/ CI fixes) (NVIDIA-NeMo#976)
738f9f1 
Signed-off-by: Tugrul Konuk <tkonuk@nvidia.com>
Signed-off-by: Peter Jin <pjin@nvidia.com>
Co-authored-by: Tugrul Konuk <tkonuk@nvidia.com>
Signed-off-by: Qidong Su <qidongs@nvidia.com>
PrinsYin pushed a commit to PrinsYin/RL that referenced this pull request Nov 30, 2025
@pjin-nvidia @ertkonuk
feat: GSPO (w/ CI fixes) (NVIDIA-NeMo#976)
7cf7894 
Signed-off-by: Tugrul Konuk <tkonuk@nvidia.com>
Signed-off-by: Peter Jin <pjin@nvidia.com>
Co-authored-by: Tugrul Konuk <tkonuk@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@terrykong terrykong terrykong approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pjin-nvidia @terrykong @ertkonuk