Help reproducing SWE-bench Verified performance for gpt-oss:120b in mini-swe-agent

[Brylian-Yu]

Describe the issue

Description

Hi, I am relatively new to AI agents, and I am glad I found mini-swe-agent — a simple but very useful agent project. I am currently trying to reproduce SWE-bench Verified evaluation results in my local environment.

However, I am seeing results that are much lower than I expected:

• I am using the open-source gpt-oss:120b model deployed through Ollama.
• For a quick initial check, I ran the benchmark on the following 10 SWE-bench Verified instances:

pallets__flask-5014
pytest-dev__pytest-5809
pytest-dev__pytest-6202
pylint-dev__pylint-6903
sphinx-doc__sphinx-8269
sphinx-doc__sphinx-8721
sphinx-doc__sphinx-9320
sphinx-doc__sphinx-9367
sphinx-doc__sphinx-9698
sphinx-doc__sphinx-9711

According to the public benchmark reports, this model reaches around 62% on SWE-bench Verified. In my local runs, however, I usually only get 1 resolved instance out of these 10. I understand that this is only a small sample and is not directly comparable to the full benchmark score, but the gap still seems large.

I would like to ask:

1. What is the recommended way to reproduce the reported SWE-bench Verified performance of gpt-oss:120b with mini-swe-agent?
2. Are there known configuration choices, prompt settings, or environment details that can significantly affect the final score?
3. Is there anything in my setup that looks suspicious or suboptimal and might explain the lower-than-expected results?

Environment

My test environment is:

System architecture: aarch64
CPU: 20 cores (10x Cortex-X925, 10x Cortex-A725)
RAM: 128GB LPDDR5x
GPU: NVIDIA Blackwell (NVIDIA GB10)
CUDA cores: Blackwell generation
Tensor cores: 5th generation (1 PFLOP)
RT cores: 4th generation
Ollama version: 0.13.5

mini-swe-agent setup

• mini-swe-agent version: 2.2.7
• Config file: livesweagent_swebench_brylian_textbased.yaml
• The test trajs files have also been uploaded as a compressed attachment.

I made one modification to the default prompt/template:

• I just added the following lines to format_error_template to prevent the LLM from forgetting how to end the task after many iteration steps:

  If you have completed your assignment then submit your works by issuing exactly
  the following command (you will not be able to continue working in any way on this task after that):
  `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && git add -A && git diff --cached`
  

Model and inference settings

model_class: litellm_textbased
model_name: "openai/gpt-oss:120b"
model_kwargs:
  api_base: "http://localhost:11434/v1"
  temperature: 0.5
  drop_params: true
  tool_choice: none
  reasoning_effort: "high"
action_regex: "```bash\\s*\\n(.*?)\\n```"
cost_tracking: "ignore_errors"

Test results

I compared both medium and high reasoning modes. The overall results were similar.

Medium reasoning mode

instance_id	resolved	patch_applied	F2P success/failure	P2P success/failure	cause
pallets__flask-5014	False	True	0/1	59/0	incorrect_fix
pytest-dev__pytest-5809	False	True	0/1	0/3	incorrect_fix_with_regression
pytest-dev__pytest-6202	False	True	0/1	0/72	incorrect_fix_with_regression
pylint-dev__pylint-6903	True	True	1/0	8/0	resolved
sphinx-doc__sphinx-8269	False	False	0/0	0/0	patch_already_applied_or_reversed
sphinx-doc__sphinx-8721	False	True	0/1	0/3	patch_already_applied_or_reversed
sphinx-doc__sphinx-9320	False	True	0/1	0/9	patch_already_applied_or_reversed
sphinx-doc__sphinx-9367	False	True	0/1	0/25	patch_already_applied_or_reversed
sphinx-doc__sphinx-9698	False	True	0/1	0/45	patch_already_applied_or_reversed
sphinx-doc__sphinx-9711	False	False	0/0	0/0	patch_already_applied_or_reversed

High reasoning mode

instance_id	resolved	patch_applied	F2P success/failure	P2P success/failure	cause
pallets__flask-5014	False	True	1/0	15/44	regression
pytest-dev__pytest-5809	False	True	0/1	0/3	incorrect_fix_with_regression
pytest-dev__pytest-6202	True	True	1/0	72/0	resolved
pylint-dev__pylint-6903	False	False	0/0	0/0	line_ending_mismatch
sphinx-doc__sphinx-8269	False	False	0/0	0/0	patch_already_applied_or_reversed
sphinx-doc__sphinx-8721	False	True	0/1	0/3	patch_already_applied_or_reversed
sphinx-doc__sphinx-9320	False	True	0/1	0/9	patch_already_applied_or_reversed
sphinx-doc__sphinx-9367	False	True	0/1	0/25	patch_already_applied_or_reversed
sphinx-doc__sphinx-9698	False	True	0/1	0/45	patch_already_applied_or_reversed
sphinx-doc__sphinx-9711	False	True	0/1	0/0	patch_already_applied_or_reversed

livesweagent_swebench_brylian_textbased.yaml

trajs.zip

[klieret]

Hi @Brylian-Yu , unfortunately 10 instances is way too low of a sample, especially if you use them in alphabetical order.

You can get our exact configuration by downloading our trajectories from https://github.com/swe-bench/experiments

I just took a look at the SWE-bench table and the GPT OSS result was with mini-swe-agent v1.7, so you might also see different results with mini-swe-agent v2.

Thanks!