Why is REINFORCE used here? It seems like it would be equivalent to simply cross-entropy plus Gini impurity as a regularizer.

[masepi]

Jev/Laya and RLCD - Reinforcement Learning for Calibrated Decisions. Why is REINFORCE used here? It seems like it would be equivalent to simply cross-entropy plus Gini impurity as a regularizer.

[GioiaZheng]

Not in general. In the current training loop, the policy-gradient term is estimating a smoothed proper-score objective under Gaussian perturbations of the logits:

z = stopgrad(logits) + eps
q = softmax(z)
grad ~ advantage * (z - logits) / sigma^2

The reward is a mixture of spherical and ranked-probability scores against a soft target distribution, with group-centering and normalization across four perturbations. That does not reduce to cross-entropy plus Gini impurity except under much narrower assumptions about the reward and target.

The implementation already adds a full soft cross-entropy term separately. So the more useful question is empirical: does the noisy proper-score term improve calibration or robustness beyond soft CE alone?

I would compare, with identical data, seeds and schedules:

1. soft CE only;
2. soft CE plus the current score-function term;
3. soft CE plus a pathwise gradient through softmax(logits + eps).

Reporting accuracy alone would miss the intended effect; Brier score, ECE, the proper scores used for training, and variance across seeds would make the distinction clearer.

[masepi]

note.pdf
I wrote a short note. @GioiaZheng

[GioiaZheng]

Nice note—the log-score/Gini derivation is useful. I think one claim needs narrowing: Eq. (5) is exact, but Eq. (1) is not. adv.std() is estimated from the same finite K × micro-batch samples as the numerator, so the scale is random and correlated with it; in general E[g/s] ≠ E[g]/E[s]. The approximation is still useful. For the experiment, one training seed plus paired bootstrap over test items does not capture training variance. A few seeds and K, micro-batch, and σ ablations would strengthen the replacement claim.

[masepi]

For a sufficient large batch E[g/s] ≈ E[g]/E[s] so everything will be fine. But otherwise, yes, you're right.