v1.0.0: EMNLP 2026 camera-ready artifact

The evaluation harness described in the EMNLP 2026 System Demonstrations paper,
together with the analyses it supports.

What a result depends on

Every run is identified by the things that can change its outcome. The agent
CLIs, the Eclipse JDT.LS snapshot and the tree-sitter grammars are pinned to
fixed versions in docker/backend.Dockerfile. Before that, a rebuild picked up
whatever was newest and could not reproduce an earlier run. The chunker version
is derived from the versions actually installed and forms part of the retrieval
index identity, so an index built by an older parser is never silently reused.

CITATION.cff records how to cite the artifact. docs/release.md records what a
release is and what has to hold before one is cut.

Analyses in this release

• scripts/chunking_ablation.py sweeps the retrieval window from 20 to 240 lines
  over nine repositories, measuring how often a window holds a whole definition
  and how much unrelated code it carries. It needs no model and costs nothing to
  rerun.
• scripts/repeat_campaign.py and scripts/repeat_stats.py run a configuration k
  times, resumably, and report pass@k with per-task flip rates.
• scripts/passk_report.py and scripts/run_budget.py handle reporting and
  costing from measured tokens.
• scripts/figures.py redraws every documentation figure from the per-task
  exports in docs/exports/ and refuses to render when a number disagrees with
  the total the study recorded.

Findings

AST chunking beats fixed line windows by 25 to 30 percentage points across prompt
modes, and retrieval over line windows scores below no retrieval at all. Repeating
a run matters for retrieval, where pass@3 is roughly double pass@1, and not at all
for the baseline or for sub-agent delegation, which returned identical verdicts
every time.

Sub-agent delegation reaches 66 of 100 against 86 of 100 for AST retrieval on the
same tasks. In 19 of those tasks the agent never delegated, so over the 81 that
did exercise the regime the rate is 70 percent. Both numbers are reported.

Fixes

A model server still fetching its weights was reported as healthy, so retrieval
runs were accepted and then failed. That defect and the chunker version defect
above are covered by regression tests in server/tests/test_retrieval_s2.py.

Security and performance

Next.js moves to 15.5.27, and the image optimizer at /_next/image is disabled.
The dashboard ships no next/image, but Next serves that route regardless, and
it has carried unauthenticated remote-execution advisories. The route now returns
404, so the surface is gone rather than merely patched. Four Python packages with
published advisories move to their fixed versions: anyio, cryptography, urllib3
and pyjwt.

Only the primary keys were indexed, and neither PostgreSQL nor SQLite indexes a
foreign key for you, so the run views, the exporter and the retention pass all
scanned run_task to find a run's tasks. Four indexes cover the columns those
queries filter on, including the composite the scheduler uses to pick the oldest
queued run.