SWE-Bench Lite Docker workspace fails due to image tag validation mismatch

[sm2909]

Summary

Running swebench-infer on SWE-Bench Lite with a Docker workspace fails before agent execution starts.

The Docker image appears to build successfully, but the benchmark runner rejects it because the built image tags do not match the expected image tag.

Environment

• Docker: 29.5.0
• Benchmarks commit: 2d635c0
• SDK submodule commit: 3e0a3a09
• Python: 3.12

Reproduction

Clone and initialize

git clone https://github.com/OpenHands/benchmarks.git
cd benchmarks

make build

Verify versions

git rev-parse --short HEAD
# 2d635c0

git submodule status
# 3e0a3a0915b369c7e2057c77722e98585855d30a vendor/software-agent-sdk (v1.14.0-186-g3e0a3a09)

Run SWE-Bench Lite

uv run swebench-infer .llm_config/example.json \
    --dataset princeton-nlp/SWE-bench_Lite \
    --split test \
    --workspace docker \
    --max-iterations 100 \
    --num-workers 1 \
    --n-limit 1

The failure occurred on:

scikit-learn__scikit-learn-25500

Expected behavior

The workspace image is built and the agent starts solving the selected SWE-Bench instance.

Actual behavior

The image build completes, but the benchmark runner fails during image validation.

Error message

Built image tags ['ghcr.io/openhands/eval-agent-server:3e0a3a0-sweb.eval.x86_64.scikit-learn_1776_scikit-learn-25500-source-minimal', 'ghcr.io/openhands/eval-agent-server:3e0a3a0-sweb.eval.x86_64.scikit-learn_1776_sciki_tag_latest-435039b0ab34-source-minimal'] do not include expected tag ghcr.io/openhands/eval-agent-server:3e0a3a0-35d813f-sweb.eval.x86_64.scikit-learn_1776_scikit-learn-25500-source-minimal

Traceback

Traceback (most recent call last):
  File "/home/khidki/FedAgent/openhands_setup/benchmarks/benchmarks/utils/evaluation.py", line 1069, in _execute_single_attempt
    workspace = self.prepare_workspace(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khidki/FedAgent/openhands_setup/benchmarks/benchmarks/swebench/run_infer.py", line 190, in prepare_workspace
    built = ensure_local_image(
            ^^^^^^^^^^^^^^^^^^^
  File "/home/khidki/FedAgent/openhands_setup/benchmarks/benchmarks/utils/build_utils.py", line 517, in ensure_local_image
    raise RuntimeError(
RuntimeError: Built image tags ['ghcr.io/openhands/eval-agent-server:3e0a3a0-sweb.eval.x86_64.scikit-learn_1776_scikit-learn-25500-source-minimal', 'ghcr.io/openhands/eval-agent-server:3e0a3a0-sweb.eval.x86_64.scikit-learn_1776_sciki_tag_latest-435039b0ab34-source-minimal'] do not include expected tag ghcr.io/openhands/eval-agent-server:3e0a3a0-35d813f-sweb.eval.x86_64.scikit-learn_1776_scikit-learn-25500-source-minimal

Notes

The built image tags contain the SDK commit hash (3e0a3a0), while the expected tag additionally includes 35d813f.

Happy to provide additional logs if needed.