Potential reward hacking?

[yesiam-png]

Hi authors, thanks for your excellent work! When I want to reproduce your results on small models, I find that at a certain timestep, the training return increases rapidly but the eval results decrease significantly. Could you inform me what goes wrong here? I also experimented with Qwen models as well as SFT models, but the results are similar to the ones I observed on Llama-3.2-3B:

Originally, I suspected that there is a training-eval gap, but I don't think it is what is truly happening here, since the training and eval data are roughly from the same distribution.

[michaelzhiluo]

Which dataset/environment are your experiments from?

[yesiam-png]

I'm following your recipe exactly (only changes the model) by using the DeepCoder-train dataset.

[michaelzhiluo]

Try out the deepcoder branch, that has a much better replication on what we did. I've noticed that DeepCoder dataset in the current rLLM only includes LCBv5 train set (it should also include prime intellect and TACO verified), a mistake from one of our team members, which will shortly be fixed.

[yesiam-png]

Thanks for your response! May I know which part of https://github.com/agentica-project/rllm/blob/main/scripts/data/code_dataset.py indicates that the training only contains LCBv5, instead of also prime and TACO? Isn't it already train_datasets = [TrainDataset.Code.PRIMEINTELLECT, TrainDataset.Code.TACO, TrainDataset.Code.LIVECODEBENCH]? Besides, I used the latest version of the deepcoder branch, does it have the same error you mentioned?

[yesiam-png]

Oh I see what you mean. The example/deepcoder/prepare_dataset.py indeed only contains the LCBv5 dataset, which I didn't follow. So I think the training dataset should be fine?

[michaelzhiluo]

If you run everything on the DeepCoder branch (training), then you shouldn't see reward hacking; it should train properly...

You should generate your parquet files through code_dataset, not prepare_dataset.py. However I realized in this current main branch, all the json files got cleaned. So recommend checking out to deepcoder branch and running code_dataset.py to generate the parquet files.

[yesiam-png]

I was running deepcoder_dataset.py on the deepcoder branch. Does this align with what you recommend?

[michaelzhiluo]

Yes!

[yesiam-png]

Sorry for the late reply. Since I was doing as you suggested and faced the issue above, could you help me understand what might cause the failure?

[yesiam-png]

To provide more context, I find that when it starts to 'hack' the reward, most of the code outputs are in this format, i.e., the code keeps waiting for the next input and causes a timeout. Will this behavior receive high rewards?

import sys

def solve_for_x(eq):
  # YOUR CODE HERE
  pass

if __name__ == "__main__":
  while True:
    try:
      eq = input()
      print(solve_for_x(eq))
    except:
      break