Repository navigation
Replies: 1 comment
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment

Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Greetings!
A new paper on improving on-policy distillation was recently released: Trust Region On-Policy Distillation.
I think one component from the paper could be a good fit for GOLDTrainer: outlier-aware FKL masking, which could improve the stability of the RKL / JSD path in a similar spirit to how DAPO-style loss variants improve stability in GRPOTrainer.
The motivation is the following. In RKL-based on-policy distillation, some student-generated tokens may receive extremely low probability under the teacher. These tokens can create very large negative token-level rewards / gradients and destabilize training. The paper addresses this by separating tokens into a trust region and an outlier region. For the trust-region tokens, it keeps the usual RKL-style objective. For outlier tokens, instead of applying the same RKL loss, it masks that RKL contribution and applies a top-k FKL loss on the teacher’s top-k distribution. In simplified form:
loss = trust_region_mask * rkl_loss + outlier_mask * topk_fkl_lossThis keeps the stabilizing effect of masking unreliable RKL gradients while still preserving some teacher signal for the outlier tokens, instead of discarding them entirely.
The reported results look promising. In the paper’s math-domain comparison, the FKL-outlier variant improves the average score from 46.79 for vanilla OPD to 49.00. It also improves over generalized JSD with$\beta = 0.5$ , which reports 47.90 in the same table. So this component alone gives a +2.21 improvement over OPD and +1.10 over JSD in that setting.

The second component in the paper, Off-Policy Guidance, is also interesting, but I do not think it is the best first target for GOLDTrainer. It requires the teacher to generate an off-policy prefix, after which the student continues generation. This would introduce a teacher-generation path that GOLDTrainer does not currently rely on in the same way. It may also be expensive for long reasoning sequences: the paper starts with the teacher prefix length set to the maximum training sequence length and then anneals it down with a cosine schedule. For long OPD training, this could add substantial generation overhead and may complicate memory management, especially with vLLM / reserved-cache behavior.
My concrete proposal is:
use_uld_loss=FalseandKeep Off-Policy Guidance out of the first implementation, unless maintainers think it is worth adding later as a separate feature.
If the Big HF Council approves this direction, I'll implement it, add tests, and run the relevant ablations.
All reactions