Repository navigation
Research question: distinguishing failed evaluation from a genuine zero reward #937
Replies: 1 comment 3 replies
|
For a controlled study, I’d start with At this revision, it copies The existing grouping path supports termination-based masking. That cannot identify a swallowed evaluator failure that reaches it as ordinary completion. Carry evaluator validity separately and apply the eligibility decision before group advantage calculation, while retaining genuine zero rewards. The numerical effect is easy to isolate. The repository’s NumPy GRPO function gave approximately Keep the rollout set fixed while comparing failure-as-zero, invalid-sample exclusion and reevaluation. Report evaluator failures, retained group sizes and minimum-group filtering separately. Otherwise changed sample selection could be mistaken for better learning. |
Uh oh!
There was an error while loading. Please reload this page.
Hi everyone, I’m Max. I’m interested in how imperfect feedback affects agent training.
One question I’d like to explore in rLLM is what happens when an evaluator failure produces the same numerical reward as a genuine task failure. For example, a timeout or an unparseable judge response might look like an ordinary zero, depending on the environment.
Is there an existing example that keeps those cases separate through rollout collection and training-sample selection?
I’d be interested in contributing a small controlled study comparing how they’re handled and whether that changes group advantages or which samples reach training. I haven’t run this experiment yet, so I’d appreciate pointers to existing work before duplicating anything.
Thanks!
Baixin Guo (Max)
All reactions