Repository navigation
When should I choose SFT over GRPO for fine-tuning a small language model? #7534
|
Hi everyone,
|
Replies: 1 comment
|
the practical test i use: before choosing, sample 8 to 16 outputs per prompt from the model you'd start from and run your reward function on them. if almost everything scores zero, go SFT first. GRPO learns from the differences between outputs in the same group, so when the whole group fails the same way there's no signal and you burn compute on nothing. small models hit this a lot on new formats. if a decent share passes, somewhere around 10 to 70 percent, GRPO directly can work, because there's real variation to push on. with a good instruction/response set like you describe, SFT then GRPO is usually the safe order. SFT gets the format and the basic task right, GRPO then sharpens what your reward actually measures. two things that bite in practice. GRPO is great when the reward is checkable (tests pass, exact match on extracted fields, valid JSON), much weaker when quality is subjective. and the model will find ways to score well without getting better, so keep a frozen eval set that is not the reward function, read samples every few hundred steps, and compare base vs SFT vs SFT+GRPO on the same items before you trust the gain. |
the practical test i use: before choosing, sample 8 to 16 outputs per prompt from the model you'd start from and run your reward function on them.
if almost everything scores zero, go SFT first. GRPO learns from the differences between outputs in the same group, so when the whole group fails the same way there's no signal and you burn compute on nothing. small models hit this a lot on new formats.
if a decent share passes, somewhere around 10 to 70 percent, GRPO directly can work, because there's real variation to push on.
with a good instruction/response set like you describe, SFT then GRPO is usually the safe order. SFT gets the format and the basic task right, GRPO then sharpens what y…