Almost successfully reproducing open-r1/OpenR1-Qwen-7B based on Qwen/Qwen2.5-Math-7B-Instruct. Here are the training configurations.

[kiseliu]

After encountering several errors and incorrect results, I'd like to share my experience reproducing open-r1/OpenR1-Qwen-7B based on Qwen/Qwen2.5-Math-7B-Instruct.

The training commands below are configured for a node of 8 x H100s (80GB).

1. Modify the config file of Qwen/Qwen2.5-Math-7B-Instruct

After downloading the model Qwen/Qwen2.5-Math-7B-Instruct, we should modify the model config file following https://huggingface.co/open-r1/OpenR1-Qwen-7B/blob/main/config.json

2. Modify the training recipes correctly

If you follow the official installation steps and run the following training command:

accelerate launch --config_file recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --config recipes/OpenR1-Qwen-7B/sft/config.yaml

The first issue you’ll encounter is #366.

To resolve this issue, the effective way is to modify the corresponding recipe: https://github.com/huggingface/open-r1/blob/main/recipes/OpenR1-Qwen-7B/sft/config.yaml by changing the line 29

- use_liger_kernel: true
+ use_liger: true

Other solutions will lead to other issues, such as:

• Downgrading trl to 0.15.x: it might not work or have this issue: https://huggingface.co/open-r1/OpenR1-Qwen-7B/discussions/7 ;
• Set use_liger_kernel: false: then we cannot set the gradient_accumulation_steps: 2 since it will bring an OOM problem;

3. Modify the sft.py file

Following this issue: #494, we could change the sft.py https://github.com/huggingface/open-r1/blob/main/src/open_r1/sft.py as follows:

- tokenizer.pad_token = tokenizer.eos_token
+ if tokenizer.pad_token is None:
+    tokenizer.pad_token = tokenizer.eos_token

4. The reproducing results on AIME24, MATH-500

I tested the saved model (the total step is 3219 steps), and the performance is:

• AIME24: 46.7;
• MATH-500: 92.4;

I tested open-r1/OpenR1-Qwen-7B, and the performance is:

• AIME24: 50.0;
• MATH-500: 92.8;

It seems that this model is saved at step 3150.

[Tim-Siu]

Thanks for sharing!

I wonder if replacing DataCollatorForLanguageModeling with DataCollatorForCompletionOnlyLM would further boost the performance.

Rightnow, the OpenR1 recipes train on both prompts and response. It doesn't seem like common practices.

[kiseliu]

@Tim-Siu I am not sure if the performance will boost.

For the current version, it only costs 12 hours for sft on one node of 8 x H100s (80GB).

If replacing DataCollatorForLanguageModeling with DataCollatorForCompletionOnlyLM, then we cannot set packing=True. I think the training time will take much longer like 20+ hours.

[wangjvjie]

how to run 7b model on multiple gpus, 1 A800 encounters oom errors

[juliancodaforno]

Thank you so much!! I had two questions:

1. In terms of efficiency didn't you change other parameters? Because doing exactly how you say I get expected time of run of 60 hours on 8 A100 of 80GB but should be ~x2.5 your reported time of 12hours (since you use H100s) so still quite off. I guess you must have changed the attn_implementation to flash_attention_2 right? Also as a side note, you can increase the per_device_train_batch_size to 2 for 8 Gpus of 80GB (don't forget to lower the grad_accum to 1) and still works without OOM. Then I get ~23hrs of expected runtime which is more what one would expect from H100s to A100s.

2. Did you try replicating other models such as the Qwen2.5-1.5B-Instruct? Because there I can't replicate.

[kiseliu]

Thank you so much!! I had two questions:

1. In terms of efficiency didn't you change other parameters? Because doing exactly how you say I get expected time of run of 60 hours on 8 A100 of 80GB but should be ~x2.5 your reported time of 12hours (since you use H100s) so still quite off. I guess you must have changed the attn_implementation to flash_attention_2 right? Also as a side note, you can increase the per_device_train_batch_size to 2 for 8 Gpus of 80GB (don't forget to lower the grad_accum to 1) and still works without OOM. Then I get ~23hrs of expected runtime which is more what one would expect from H100s to A100s.
2. Did you try replicating other models such as the Qwen2.5-1.5B-Instruct? Because there I can't replicate.

@juliancodaforno

1. I didn’t change any other parameters. The attn_implementation I used was the default value (sdpa).

   Yes, setting per_device_train_batch_size=2 and gradient_accumulation_steps=1 should work without causing OOM errors.

2. I previously attempted to run the code for Qwen2.5-1.5B-Instruct but did so incorrectly. I won’t retry it again.

[XYaoooo]

hi, what is your global bz and learning rate? thx

[kiseliu]

hi, what is your global bz and learning rate? thx

@XYaoooo Hi, the global batch size is set to 16, the learning rate is 5.0e-05.

[XYaoooo]

hi, what is your global bz and learning rate? thx

@XYaoooo Hi, the global batch size is set to 16, the learning rate is 5.0e-05.

Hi, one more question, what is the version of your trl and deepspeed? Thx

[HwangYej1]

Have you modified "max_position_embeddings": 4096 in the Qwen2.5-Math-7B-Instruct model file config?

[HwangYej1]

If you don't keep max_positions_embedding: 4096, how do you set the hyperparameter on the " MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.6,data_parallel_size=$NUM_GPUS,generation_parameters={max_new_tokens:2048,temperature:0.6,top_p:0.95}" " line when you test with lighteval?

[kiseliu]

hi, what is your global bz and learning rate? thx

@XYaoooo Hi, the global batch size is set to 16, the learning rate is 5.0e-05.

Hi, one more question, what is the version of your trl and deepspeed? Thx

trl==0.16.0; deepspeed=0.15.4;

[kiseliu]

If you don't keep max_positions_embedding: 4096, how do you set the hyperparameter on the " MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.6,data_parallel_size=$NUM_GPUS,generation_parameters={max_new_tokens:2048,temperature:0.6,top_p:0.95}" " line when you test with lighteval?

max_model_length=32768, max_new_tokens=32768;

You can see the README.md file.

[XCF-Mike]

After encountering several errors and incorrect results, I'd like to share my experience reproducing open-r1/OpenR1-Qwen-7B based on Qwen/Qwen2.5-Math-7B-Instruct.

The training commands below are configured for a node of 8 x H100s (80GB).

1. Modify the config file of Qwen/Qwen2.5-Math-7B-Instruct

After downloading the model Qwen/Qwen2.5-Math-7B-Instruct, we should modify the model config file following https://huggingface.co/open-r1/OpenR1-Qwen-7B/blob/main/config.json

2. Modify the training recipes correctly

If you follow the official installation steps and run the following training command:

accelerate launch --config_file recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --config recipes/OpenR1-Qwen-7B/sft/config.yaml

The first issue you’ll encounter is #366.

To resolve this issue, the effective way is to modify the corresponding recipe: https://github.com/huggingface/open-r1/blob/main/recipes/OpenR1-Qwen-7B/sft/config.yaml by changing the line 29

- use_liger_kernel: true
+ use_liger: true

Other solutions will lead to other issues, such as:

• Downgrading trl to 0.15.x: it might not work or have this issue: https://huggingface.co/open-r1/OpenR1-Qwen-7B/discussions/7 ;
• Set use_liger_kernel: false: then we cannot set the gradient_accumulation_steps: 2 since it will bring an OOM problem;

3. Modify the sft.py file

Following this issue: #494, we could change the sft.py https://github.com/huggingface/open-r1/blob/main/src/open_r1/sft.py as follows:

- tokenizer.pad_token = tokenizer.eos_token
+ if tokenizer.pad_token is None:
+    tokenizer.pad_token = tokenizer.eos_token

4. The reproducing results on AIME24, MATH-500

I tested the saved model (the total step is 3219 steps), and the performance is:

• AIME24: 46.7;
• MATH-500: 92.4;

I tested open-r1/OpenR1-Qwen-7B, and the performance is:

• AIME24: 50.0;
• MATH-500: 92.8;

It seems that this model is saved at step 3150.

Hi, for the third

`- tokenizer.pad_token = tokenizer.eos_token

• if tokenizer.pad_token is None:
• tokenizer.pad_token = tokenizer.eos_token`

I made the same changes, but the problem in #494 was not resolved.
For this issue, I also tried different versions of trl, including trl==0.15.1, trl==0.16.0, and trl==0.16.1, but none of them helped resolve the problem.
Are there any other changes that should be made?

[kiseliu]

@XCF-Mike hi, I suggest to use the updated code since this problem is fixed in the issue #604.