Repository navigation
Replies: 1 comment
|
Thanks for the comment, and yes, I agree with this. I think I also mentioned it somewhere in chapter 6 or 7 (or maybe it was in a podcast interview) that I think that the knowledge is primarily taken up during pre-training, and the RLVR here is more like teaching the model how to access that knowledge. So, yeah, this is a good explanation indeed, and I agree with it here :) |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Having worked with for a long time with "conventional" feed-forward multi-layer networks for satellite image classification, I found it difficult to understand how a single training run of only 50 steps at a lr of 1e-5 could possibly make any significant difference in the response of the LLM model. The Manning AI app accompanying the Livebook gave what I thought was an extremely good explanation, only part of which I include here:
"The pre-trained model has already seen enormous amounts of text and likely contains partial math-solving and explanation capabilities. GRPO does not need to build those capabilities from scratch. It repeatedly nudges the probability distribution toward response patterns that happened to yield a correct, boxed answer, and away from patterns that did not"
This may well be implicit in the text, but it came as a revelation to an old dog like me used to simpler models, making the entire reinforcement approach make more sense. I suggested to the AI app that it might have been a good idea to have made this explicit in the original text. It agreed (of course).
All reactions