[Feature] kl_mask_threshold

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] Aggregation strategies #3209
• -> [Feature] kl_mask_threshold #3208
• [Feature] CISPO #3207
• [Feature] DAPO #3206
• [Refactor] Refactor GRPO as a separate class #3205

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3208

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 1 Cancelled Job, 1 Unrelated Failure

As of commit 9379070 with merge base bd93f13 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t f71c26fd43f348dc809705a4172572598a898547dfc2eb461be7b9cc403f8a01 /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 411f64e342a983229b65080547daa686311f3464a4aff932592d0adacdd4ea71 /exec failed with exit code 1
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 6a42becc2dc52b7470e78aedfe1a812cba6c8505973f58382f0f5b90b99f8280 /exec failed with exit code 1
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  test/test_distributed.py::TestRayCollector::test_distributed_collector_updatepolicy[False-SyncDataCollector]

CANCELLED JOB - The following job was cancelled. Please retry:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.