feat: Discard weight when finish generation in the main loop

[guyueh1]

What does this PR do ?

A feature to reduce CPU memory footprint, which allows us to use pinned memory for opt offload in GB systems and helps perf then.

Currently the CPU memory peak happens at the end of finish_generation, which is inference weights + training weights + training optimizer; with this change we will reduce it to training weights + training optimizer.

With this change, I was able to use TP8 instead of TP16 for inference in DeepSeek V3 GRPO and got ~20% E2E speedup (>30% generation speedup)

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[guyueh1]

/ok to test 5efbbf4

[guyueh1]

/ok to test 8253a2a

[guyueh1]

/ok to test 6760d81

[guyueh1]

/ok to test 6760d81

[guyueh1]

/claude review

[guyueh1]

/ok to test 8d80498

[guyueh1]

/claude review

[guyueh1]

CI is green, @terrykong please take a look; @youngeunkwon0405 please approve again

[terrykong]

thanks @guyueh1 . there's also this PR which we are currently reviewing #2523

seems to almost be the same thing with the exception of some small bits like the colocated check. shall we just use @Kipok 's PR? if so, could you apply the review over on his PR?

[guyueh1]

thanks @guyueh1 . there's also this PR which we are currently reviewing #2523

seems to almost be the same thing with the exception of some small bits like the colocated check. shall we just use @Kipok 's PR? if so, could you apply the review over on his PR?

@terrykong I don't like the implementation in 2523 because it makes a knob visible to user but that knob involves system knowledge and should have been managed by the framework. Also 2523 is kind of wrong because it sets all the sleep to 2, but in grpo.py the calls of policy_generation.finish_generation() after the validation shouldn't discard the weights, since the refit happened before validation and won't happen again before the next generation. If we adopt 2523 it will crash cases that has validation.