How can I evaluate the extended data I have collected?

[lycfight]

Describe the bug

I tried evaluating an instance from a newly extended SWE-bench dataset, nebius/SWE-bench-extra, but encountered an error.

Steps/Code to Reproduce

(SWE-bench) root@cpu01-2050-SWE-bench:~/SWE-bench# python -m swebench.harness.run_evaluation --dataset_name nebius/SWE-bench-extra --predictions_path gold --max_workers 1 --split train --instance_ids 0b01001001__spectree-64 --cache_level
 instance --run_id validate-gold

Expected Results

The evaluation should work correctly in the same way as the provided example sympy__sympy-20590.

(SWE-bench) root@cpu01-2050-SWE-bench:~/SWE-bench# python -m swebench.harness.run_evaluation \
    --predictions_path gold \
    --max_workers 1 \
    --instance_ids sympy__sympy-20590 \
    --run_id validate-gold
<frozen runpy>:128: RuntimeWarning: 'swebench.harness.run_evaluation' found in sys.modules after import of package 'swebench.harness', but prior to execution of 'swebench.harness.run_evaluation'; this may result in unpredictable behaviour
Using gold predictions - ignoring predictions_path
Running 1 instances...
1 ran successfully, 0 failed: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [01:50<00:00, 110.53s/it]
All instances run.
Cleaning cached images...
Removed 0 images.
Total instances: 1
Instances submitted: 1
Instances completed: 1
Instances incomplete: 0
Instances resolved: 1
Instances unresolved: 0
Instances with empty patches: 0
Instances with errors: 0
Unstopped containers: 0
Unremoved images: 0
Report written to gold.validate-gold.json

Actual Results

However, an error occurred. It seems that MAP_REPO_VERSION_TO_SPECS does not include the newly extended repository. Does this need to be set manually? Could you provide a detailed practical example?

(SWE-bench) root@cpu01-2050-SWE-bench:~/SWE-bench# python -m swebench.harness.run_evaluation --dataset_name nebius/SWE-bench-extra --predictions_path gold --max_workers 1 --split train --instance_ids 0b01001001__spectree-64 --cache_level
 instance --run_id validate-gold
<frozen runpy>:128: RuntimeWarning: 'swebench.harness.run_evaluation' found in sys.modules after import of package 'swebench.harness', but prior to execution of 'swebench.harness.run_evaluation'; this may result in unpredictable behaviour
Using gold predictions - ignoring predictions_path
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/SWE-bench/swebench/harness/run_evaluation.py", line 617, in <module>
    main(**vars(args))
  File "/root/SWE-bench/swebench/harness/run_evaluation.py", line 512, in main
    run_instances(
  File "/root/SWE-bench/swebench/harness/run_evaluation.py", line 292, in run_instances
    test_specs = list(
                 ^^^^^
  File "/root/SWE-bench/swebench/harness/run_evaluation.py", line 294, in <lambda>
    lambda instance: make_test_spec(
                     ^^^^^^^^^^^^^^^
  File "/root/SWE-bench/swebench/harness/test_spec/test_spec.py", line 197, in make_test_spec
    specs = MAP_REPO_VERSION_TO_SPECS[repo][version]
            ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: '0b01001001/spectree'

System Information

Linux, Python 3.12, swebench 3.0.13

[ofirpress]

Seems like this is an issue with 'nebius/SWE-bench-extra', maybe make an issue in their repo?

[lycfight]

Seems like this is an issue with 'nebius/SWE-bench-extra', maybe make an issue in their repo?

I have already asked, but I haven't received a response yet. I'm trying to fix it.

The core issue lies in expanding data for new repositories. It seems that the documentation does not specify how to configure environment-related constants. You may refer to:

• [Constants related to harness](https://github.com/SWE-bench/SWE-bench/tree/main/swebench/harness/constants)
• [Log parsers related to harness](https://github.com/SWE-bench/SWE-bench/tree/main/swebench/harness/log_parsers)

for constants related to creating a mirrored environment.

How should I configure these constants? Is this process entirely manual? When you create data for a new repository, how do you handle this? Is there any way to automatically collect and update these constants?

[ofirpress]

Is this process entirely manual?

yes

Is there any way to automatically collect and update these constants?

not currently, this is a research-level question that a few different groups are working on

[lycfight]

Is this process entirely manual?

yes

Is there any way to automatically collect and update these constants?

not currently, this is a research-level question that a few different groups are working on

Indeed, different repository environments have various configuration methods. I need an agent to automatically create mirrored environments for different repositories.