Releases: iroiro147/dreambench-swe
Release list
DreamBench-SWE v2.1.0 — external-systems successor audit
DreamBench-SWE v2.1.0 is the additive successor release for Paper A. It publishes the current manuscript and the separately preregistered external-systems audit while preserving the original v2.0.5 tag and every v2.0.5 asset unchanged.
The successor result is deliberately conservative: external discriminant validity was not statistically demonstrated. This release does not claim superiority, equivalence, or broad product-level generalization. Proposed external conditions that failed the frozen pre-evaluation conformance gate were not benchmark-evaluated.
Release assets:
- Sanitized self-contained evidence artifact:
e8b8d27c7d532aee477419fd90904d93a734378beec4d46d7a06a5fe29463c2b - Paper PDF:
f0201fdc6a380fcf045fd0054340e4b6fbdc86afede1d96a54df7ace9db0622f - Flat arXiv source:
f07216c7c3cf4ad582d1f504c757173c20adcec96561b51496b97860eee525c8 - Manifest:
34c8fd8ecc5d554029811e5ca7b4e5cbf3fc7460c0d969cb932b92b8e2b5ed07 - Detached release ledger:
9e8654e8875c4466febf02addfb6afd7e0adecf9117434aa144e5c193f951090
The manifest binds the frozen evaluation source, launcher source, and release source. The evidence artifact ships a public verifier and selected regression tests; raw hosted-model logs, credentials, private analyzer inputs, hidden oracles, and local operational paths are excluded.
DreamBench-SWE v2.0.5 — audited Paper A release
DreamBench-SWE v2.0.5 is the audited public release for Paper A.
This release preserves the frozen confirmatory analysis while closing submission-package, release-binding, runbook, and public-history hygiene gaps. The public Git graph was reconstructed with neutral GitHub noreply metadata. Historical v2.0.0-v2.0.2 artifact tarballs were withdrawn because they unintentionally contained raw operational fixtures; their release records retain explicit disclosure.
Release hashes:
- Artifact:
c7b5803d6c30f0b146ce91884b5609d6969c055b4f0160e6de1d0e7adca6eb23 - arXiv source:
f37b4de3af546804d3782845ed29b10478391494b22fc6eb3f67d759259f78db - Paper PDF:
4ed203e8e928ed80361f8729dd4e970a4aa52cb00ad901164944e1ae594c375d - Manifest:
b774a688d08961c174ac04096e9f1da538b3246468e273e9d18b48e99c623e6e - Package checksums:
ebdcdb2ba824d0c85f12b7cbdaec49f23fcd33c699dfafd91ab03dc7b42eb5a4 - Detached release ledger:
b28af0bccd99c7f510767f52dcd983d31bfaef6abffcd596522dfb2673efa5b8
The paper reports a conservative non-rejection result for the preregistered primary comparison; it does not claim superiority or equivalence. Live hosted-model reruns are new experiments, not exact replays of the frozen fold.
DreamBench-SWE v2.0.4
Presentation-only refresh for arXiv submission. This release keeps the public artifact tarball byte-identical to v2.0.3 and updates the manuscript PDF/arXiv source after the final adversarial layout audit: LaTeX float placement is constrained, glossary appendix headings are separated cleanly, and public-release references now point to v2.0.4. Canonical analyzer outputs and reported paper numbers are unchanged.
Public history reconstruction notice (2026-07-10)
This release artifact already contained the sanitized replacement fixtures and was retained. The tag now points into the audited reconstructed graph. Use v2.0.5 for the current release.
The public Git graph and tag metadata were reconstructed with a neutral GitHub noreply identity. External clones or caches made before reconstruction cannot be revoked. The frozen Paper A analysis and reported results were not changed by this operation.
DreamBench-SWE v2.0.3
DreamBench-SWE v2.0.3 public artifact for Paper A.\n\nArtifact SHA-256: 2dd35ecc6aac441c030ccf640a3abda85709abb51f0f264262b8b0a7b42c16fd\narXiv source SHA-256: 954075ae506e3e9bf47092a44e009ef66095d86d2e347bd1b77240cd235db482\nPDF SHA-256: 97c53ce28e362477c94b9638189ce81971014f997fc2a0b13e2148e943d74595\n\nThe private lab repository remains private; this release is the clean public artifact/source package.
Public history reconstruction notice (2026-07-10)
This release artifact already contained the sanitized replacement fixtures and was retained. The tag now points into the audited reconstructed graph. Use v2.0.5 for the current release.
The public Git graph and tag metadata were reconstructed with a neutral GitHub noreply identity. External clones or caches made before reconstruction cannot be revoked. The frozen Paper A analysis and reported results were not changed by this operation.
DreamBench-SWE v2.0.2
Public DreamBench-SWE v2 artifact and arXiv source package.
Artifact SHA-256: cc489a35a840b0fafd929be64cdc6f6205de620a1fe95f49afbebe6e39dc532f
ArXiv source SHA-256: e895fe8c4036ccc317f66babac76cb99d85414bd158bff2dff479229f577110d
This patch refreshes the arXiv source/PDF after fixing public-release references and the printed v2_ladder.pdf checksum in the appendix.
Public history reconstruction notice (2026-07-10)
The artifact tarball with SHA-256 cc489a35a840b0fafd929be64cdc6f6205de620a1fe95f49afbebe6e39dc532f was withdrawn because it unintentionally included raw operational fixtures. The other audited assets remain available. Use v2.0.5 for the current public artifact.
The public Git graph and tag metadata were reconstructed with a neutral GitHub noreply identity. External clones or caches made before reconstruction cannot be revoked. The frozen Paper A analysis and reported results were not changed by this operation.
DreamBench-SWE v2.0.1
Public DreamBench-SWE v2 artifact and arXiv source package. Artifact SHA-256: 5b62a63d3869c6108d7fb0c453e21910b8a2d17f44ce81d9a359e9e9771c907d. ArXiv source SHA-256: 19426ccba4ba422aa58a7a0b1fe15a2164ae6d848454e8df401c5e3f6b2a27af.
Public history reconstruction notice (2026-07-10)
The artifact tarball with SHA-256 5b62a63d3869c6108d7fb0c453e21910b8a2d17f44ce81d9a359e9e9771c907d was withdrawn because it unintentionally included raw operational fixtures. The other audited assets remain available. Use v2.0.5 for the current public artifact.
The public Git graph and tag metadata were reconstructed with a neutral GitHub noreply identity. External clones or caches made before reconstruction cannot be revoked. The frozen Paper A analysis and reported results were not changed by this operation.
DreamBench-SWE v2 — Paper A Artifact
DreamBench-SWE v2 — Paper A Submission Artifact
Artifact SHA-256: 539241e85982a40dc61bd692eece14b0fcdbff174fbaef2be5b7a16a7f412516
arXiv source package SHA-256: c919edcae62181b5d8c77d34624b2490c9f3436b87688be4e6fb99dcd379ea44
Verify the artifact download with:
shasum -a 256 dreambench-swe-artifact.tar.gzIncluded: public benchmark fixtures, evaluation scripts, frozen folded analysis artifacts, package manifest/checksums, tests, and publication figure PDFs.
Excluded by design: hidden scoring oracle labels, reference-solution trajectories, and raw hosted-model result directories. These exclusions preserve benchmark integrity and avoid exposing answer keys. Live hosted-model reruns require external agent credentials and are new experiments, not exact replay of the paper fold.
Public history reconstruction notice (2026-07-10)
The artifact tarball with SHA-256 539241e85982a40dc61bd692eece14b0fcdbff174fbaef2be5b7a16a7f412516 was withdrawn because it unintentionally included raw operational fixtures. The other audited assets remain available. Use v2.0.5 for the current public artifact.
The public Git graph and tag metadata were reconstructed with a neutral GitHub noreply identity. External clones or caches made before reconstruction cannot be revoked. The frozen Paper A analysis and reported results were not changed by this operation.