Active-SWE

Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

🔍 Overview

Active-SWE evaluates proactive bug-fixing by asking coding agents to inspect repository snapshots and files pending review without instance-specific issue reports. The benchmark contains 1,663 tasks across six bug categories and eight programming languages.

中文概述
Active-SWE 是一个面向主动式软件缺陷修复的评测基准。它要求编码智能体在没有实例级 Issue 报告的情况下，主动检查仓库快照与待审文件、发现潜在缺陷并完成修复。基准共包含 1,663 个任务，覆盖六类缺陷和八种编程语言。参阅 Chinese Introduction 了解更多背景与解读。

The evaluation protocol has three stages:

1. Recorded: generate a code patch and evaluate recorded bugs.
2. Potential: generate tests and validate fail-to-pass behavior.
3. Judge: validate the evidence connecting potential bugs and tests.

Suite	Total	Simple	Hard	Purpose
Active-SWE	400	300	100	Curated main benchmark
Active-SWE-Extend	1,663	1,411	252	Extended benchmark

🛠️ Setup

Prerequisites:

• Python 3.9 or later
• A standard local Docker Engine and permission to run docker
• Claude Code (claude or claude.exe)
• An Anthropic-compatible model configuration and credential

Static Stage1/2 bridge binaries for Linux amd64 and arm64 task images are bundled. Go is needed only to rebuild them from source.

The bridge runs inside the Linux task container, so the host may use another operating system. Host Unix-socket bind mounting is tested on native Linux; Windows users should run the workflow inside WSL2, while native macOS Docker Desktop support remains unverified.

Install from the project root:

python3 -m pip install .

If the current Python environment has incompatible packages, use a project-local .venv; a virtual environment is otherwise optional.

Runtime-installed tools use ./.tools. Model configuration uses ./config/models.json plus ./config/.env, public data uses ./data, and run artifacts use ./runs. These project-local paths are ignored and are not included in source or binary packages.

The bundled skills provide guided setup:

• .claude/skills/active-swe-eval/ for Claude Code as host orchestrator
• .codex/skills/active-swe-eval/ for Codex as host orchestrator

Start with the corresponding SKILL.md; it routes the remaining setup and evaluation instructions.

⚡ One-line agent start

Send this instruction to an agent with terminal and network access:

Clone https://github.com/XLearning-SCU/Active-SWE if needed, then enter the repository and follow .codex/skills/active-swe-eval/SKILL.md for Codex or .claude/skills/active-swe-eval/SKILL.md for Claude Code to run the evaluation.

🗂️ Data and images

The controller downloads the default public Parquet configuration from the Active-SWE dataset and converts it to JSONL when needed:

data/Active-SWE.parquet
data/Active-SWE.jsonl

The exact rows used by each run are copied to runs/<timestamp>/inputs/input.jsonl, validated, and used to pull only their referenced images. The controller also validates image architecture and working directory before starting evaluation stages.

▶️ Run

Create the local configuration from the tracked Claude examples:

cp config/models.example.json config/models.json
cp config/.env.example config/.env
chmod 600 config/.env

The example selects claude-sonnet-4-6, the official Anthropic API endpoint, and ANTHROPIC_API_KEY. Put the real key only in config/.env. Edit the copied JSON when using another Anthropic-compatible model or endpoint. Each model has its own task concurrency; models run sequentially.

id is the stable run and output name. It lets the same API model appear more than once with different endpoints or settings. Model configuration rejects unknown fields and duplicate IDs instead of silently using defaults.

The controller loads config/.env automatically without overriding variables already present in the host environment. Existing configuration files may stay outside the checkout; pair them explicitly without copying credentials:

python -m activeswe.run_evaluation \
  --model-config path/to/models.json \
  --env-file path/to/.env

Relative paths resolve from the project root. The controller never reports the credential-file path or its values.

The controller prepares data and images, selects the bridge matching each task image, starts the fixed-destination tunnel, and runs the full protocol:

python -m activeswe.run_evaluation \
  --model-config config/models.json

To run only part of a shared model list, repeat --only-model with config IDs:

python -m activeswe.run_evaluation \
  --model-config config/models.json \
  --only-model claude-sonnet

After installation, active-swe-eval is the equivalent console command. Credentials may come from config/.env, an existing host environment variable, a permission-restricted key file, or hidden interactive input. API key values are never accepted directly on the command line or in models.json.

The default public input is the 400-row Active-SWE main configuration. A model without concurrency uses 4. Other frozen defaults are image-pull concurrency 4, no row limit within that input, maximum 300 turns, a 5,400-second Claude timeout, no original Git history, no image pull during stage execution, and no resume. Use CLI overrides only when you intentionally want different behavior. Before expensive work, the controller prints each selected model's task concurrency and the corresponding peak container CPU/memory limits. Docker image pull and metadata outcomes are stored in runs/<timestamp>/traces/preflight/docker_images.json.

For a ten-row run, pass --limit 10. The controller first writes those exact rows to inputs/input.jsonl, then pulls only images referenced by that file. Registry pulls are daemon operations performed before task containers start; daemon options such as --bridge=none and --iptables=false do not by themselves disable registry access. Network isolation is assigned when each evaluation container is started.

Run artifacts are organized relative to the selected run root:

runs/<timestamp>/
├── inputs/
│   └── input.jsonl
├── traces/
│   ├── preflight/
│   └── <model-id>/
└── results/
    ├── summary.json
    └── <model-id>/
        └── full/
            ├── recorded.jsonl
            ├── potential.jsonl
            ├── judge.jsonl
            └── metrics.json

metrics.json is the authoritative metric artifact. summary.json also reports per-stage sample outcome counts. A controller status of ok means the pipeline completed; inspect those counts for model-level outcomes such as missing generated patches or Judge results.

🔒 Network and history isolation

Recorded and Potential task containers run with Docker --network none; model traffic reaches one fixed API destination through the bundled Unix-socket bridge. Their policy also denies general network and Git history operations. Judge remains outside this boundary and retains host-network behavior.

See Stage1/2 network isolation for the trust boundary and operational details.

📊 Metrics

Metric	Meaning
LR	Recall of recorded edits matched by predicted edits
LP	Precision of predicted edits matched to recorded edits
Resolved	Whether a predicted patch passes the reference tests
Count	Number of generated tests
TV	Whether each generated test exhibits F2P behavior
Revealed	Whether each revealed bug has Judge-validated F2P evidence

💻 Resource requirements

Task images include language-specific build tools and can be large. Pull them before evaluation and provide enough Docker storage, CPU, and memory for the selected repositories. A full run can take hours depending on model latency, test duration, and concurrency.

📝 Citation

If you use Active-SWE, please cite:

@article{li2026activeswe,
  title={Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports},
  author={Li, Haobin and Deng, Ping and Qian, Weizhong and Jiang, Liang and Huang, Zhenyu and Yang, Mouxing and Peng, Xi},
  journal={arXiv preprint arXiv:2608.04682},
  year={2026}
}

🙏 Acknowledgements

Active-SWE draws inspiration from the data organization and evaluation design of SWE-bench dataset and SWE-bench source code.

📄 License

Active-SWE is distributed under the Apache-2.0 License.