Question about reproducing the reported 73.5 EX on BIRD-dev with Qwen3-Coder-30B-A3B

[leoinlaw]

First of all, thank you for open-sourcing this work. I found the codebase very easy to follow, and I was able to run the full pipeline end-to-end without much difficulty.

I am trying to reproduce the reported BIRD-dev result for Qwen3-Coder-30B-A3B.

Using the released pipeline, I obtained 71.82 EX (1093 / 1533), while the README reports 73.5 EX for the public file results/bird-dev/qwen3-coder-30b-a3b.json.

To make debugging easier, I am attaching:

• config/config-bird-qwen3-coder-dev.toml:
  config-bird-qwen3-coder-dev.toml.txt
• results/bird-dev/qwen3-coder-dev.json:
  qwen3-coder-dev.json

Would you mind helping me identify where my reproduction may differ from your setup?
In particular, I would like to know whether there are any important differences in:

• the intended runtime/configuration for the pipeline,
• or any unreleased preprocessing / schema / retrieval settings that are needed to match the reported result.

Any guidance would be greatly appreciated.

[BugMaker-Boyan]

Thanks a lot for your careful reproduction and for sharing both the config and prediction file.

After looking through your config, I think the following points are worth checking first:

1. Model precision: please make sure the model you are calling is the original FP16 inference version. Some platforms may serve an FP8-quantized variant by default, which can lead to noticeable performance differences.

2. Max tokens: in our experiments, we set max_tokens = 4096 for all models. This is mainly to avoid unexpected truncation caused by overly long reasoning traces or outputs.

3. Temperature: in our setup, all models were run with temperature = 0.7.

These are the first differences I would suggest checking, as they can already affect the final EX score.

As far as this reproduction is concerned, there are no additional unreleased preprocessing, schema, or retrieval settings required to match the reported result. The released pipeline is the intended one.

Thanks again for the detailed report — this is very helpful.

[leoinlaw]

Hi, and thank you again for your earlier help.
I reran the BIRD-Dev experiment using the attached config:

config-bird-qwen3-coder-default-rerun.toml.txt

Unfortunately, I still obtain only 71.89 EX (1102/1533)
My final evaluation result is:

• overall EX: 1102 / 1533 = 71.89%
• simple: 707 / 925 = 76.43%
• moderate: 314 / 464 = 67.67%
• challenging: 81 / 144 = 56.25%

So I must admit I am rather puzzled at this point, and a little at a loss. I had the impression that I had already aligned the setup carefully, but I still cannot understand where this gap is coming from.

If you have a moment, could you perhaps have a look at this config and let me know whether I may still be missing some important detail?

Many thanks.

[BugMaker-Boyan]

Thank you for the detailed update and for sharing your new config.

We are currently re-running the code on our side to try to reproduce this issue and better understand where the gap may be coming from. We appreciate your patience, and we will follow up once we have more concrete findings.

Thanks again for your careful reproduction efforts.