[Feature] DAPO

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] Aggregation strategies #3209
• [Feature] kl_mask_threshold #3208
• [Feature] CISPO #3207
• -> [Feature] DAPO #3206
• [Refactor] Refactor GRPO as a separate class #3205

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3206

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm][CI] Machines under the label linux.rocm.gpu.2 are undergoing maintenance.

❌ 3 New Failures, 1 Cancelled Job, 2 Unrelated Failures

As of commit 38c577d with merge base 7b85c71 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t f67cbaee71057dc546dfaedce23e2b828a96c101b3891a06fcb77fcaff70d86e /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_env.py::TestNonTensorEnv::test_parallel[False-False]

CANCELLED JOB - The following job was cancelled. Please retry:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh) (trunk failure)
  RuntimeError: SumSegmentTreeFp32 is not available. See warning above.
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.