Skip to content

v0.35.0

Choose a tag to compare

@Mathews-Tom Mathews-Tom released this 15 Oct 00:06
· 5 commits to main since this release
de1c555

[0.35.0] - 2026-10-15

This release makes client-config installation atomic, reports unexpected status snapshot failures in the diagnostics log, and gives temporary index stores explicit scratch-directory ownership. Client setup is split into per-client modules without changing its existing public imports or generated content. The R3x SWE A/B harness gains deterministic sampling, pre-registered analysis, provider-selected campaigns, and an MCP arm; these are harness capabilities, not evidence of token savings or improved solve rates.

Re-run archex install-client <host> with the hook and status-line options you use to regenerate client files through the atomic writer. Consumers that passed delete_dir_on_close to IndexStore must use IndexStore.scratch() for an owned temporary store; caller-supplied directories are never deleted by a plain IndexStore(path).

Added

  • R3x population sampler. scripts/swe_ab_sample.py draws the Stage 1 (24 tasks) and Stage 2 SWE-bench Pro V2 samples deterministically: a seed fixed in code, a sha256 rank order within each repository, Stage 1 at two per repository plus one for the two largest, Stage 2 by largest-remainder allocation and disjoint from Stage 1. Stage 2 takes --stage2-tasks N (at least 100, recorded in the manifest); the pre-registered size is max(100, ⌈max(370, n(m)) / k⌉) for the k configurations that clear the Stage 1 solve-rate floor and the m primary arms kept after the adoption gates (209 for k = 3, m = 3). --without-hc and --without-m drop an arm from Stage 2. Instances that fail Stage 0 validity are replaced by the next valid instance of the same repository; the tool emits the suite's plan JSON, a manifest of every walked instance, and the Stage 0 instance list.
  • R3x pre-registered analysis. scripts/swe_ab_analysis.py computes exactly the pre-registration's primary metrics on a validated result directory: per-configuration geometric-mean billed-token ratios for H, HC, and M against A0, read superior, worse, equivalent, or inconclusive (Holm over the arms, one family per direction); the pooled solve-rate guardrail and a two-sided pooled completion test per arm (better, worse, no_difference_detected), with the configuration as stratum; repository-clustered bootstrap intervals (10,000 resamples, seed 20260909), McNemar tests, a sensitivity check without blocked pairs, and every Stage 1 gate (solve-rate floor, lever share, HC and MCP adoption, hook activity, A0-vs-A0 noise with the implied power at the Stage 2 size). It refuses mixed-emulation stages and unscored cells.
  • R3x MCP arm, M. The archex MCP server registered in the agent's omp profile as archex install-client omp installs it (mcp.json with one server, archex mcp; archex not on PATH, no hook); --tools adds mcp__archex_context and mcp__archex_query_repo, the server's default first surface. Cells record mcp_config_sha256, archex_mcp_calls, and archex_mcp_tools, and the channel table gains archex-MCP (artifact version 3). The shared omp overlay sets mcp.startupTimeoutMs: 0, without which omp rejected the allow-list before the server registered its tools. Stage 0 adds mcp_tools_in_m_arm and mcp_call_executes_in_m_arm; a profile directory carrying its own mcp.json is refused.
  • R3x OpenRouter campaign. campaigns/openrouter.yml runs the three configurations through OpenRouter with each model pinned to one upstream host (openRouterRouting.only: DeepInfra for Qwen 3.8 27B, NextBit for Gemma 4 26B A4B, both bf16) and those hosts' prices (benchmarks/swe_ab/openrouter-open-models.yml).
  • Solve-rate floor. Stage 1 gate solve_floor: a configuration whose A0 solve rate (both repetitions) is under 15% is uninformative for the guardrail. With --pilot-analysis, Stage 2's pooled guardrail excludes it (reported per configuration, labelled as uninformative) and reads not_estimable if none pass; efficiency stays per configuration.
  • Cells record the inputs the Stage 1 gates need. out_of_patch_read_tokens_compounded (reads of files outside the gold and test patches) and the ledger's not_fresh_after_first_edit; Stage 0 adds annotate_latency_in_container, the steady-state annotate latency inside each Pro image against the hook's 0.5 s budget.
  • Stage 0 checks for the campaign's provider route. provider_route (omp's catalog lists every configuration's selector under the campaign's provider), provider_key_accepted (a zero-token chat request naming a nonexistent model is rejected for its model, e.g. Muna 404 model_not_found or OpenRouter 400, while 401/403 fail; the key is never printed), effort_request_shape (each configuration's first request carries its reasoning_effort, which catches Qwen sending none), and provider_reachable_from_container; one_real_cell_per_model becomes one_real_cell_per_configuration.
  • Credit exhaustion stops the suite. A cell that ends in HTTP 402 or an insufficient-credits error is recorded as credit_exhausted, filed under quota-blocked/, never retried or scored; running cells finish and the suite exits with status 5, resumable after a top-up. Muna's response shape for it is unverified.
  • Frozen provider config and omp overlay. benchmarks/swe_ab/muna-models.yml declares the Muna provider and its models (Qwen with compat.thinkingFormat: openai, so --thinking reaches the request as reasoning_effort) and holds Muna's prices (USD per million tokens, input / cached input / output: Qwen 3.8 27B 0.25 / 0.02 / 1.75, Gemma 4 26B 0.065 / 0.02 / 0.30), which feed omp's modelled cost and so the cost ceiling; a docker run refuses to start while any input, output, or cached-input price is 0 unless the plan sets a token ceiling. benchmarks/swe_ab/omp-campaign.yml (--config) raises omp's retry budget to 20 so a capacity 429 is retried for about 13 minutes (measured 785–789 s against the stub, 398–400 s by default) and keeps retry.waitForUsageReset off. Cells record both files' SHA-256 and the resolved provider base URL.
  • scripts/swe_ab_stub_provider.py http_error steps take a body, returned verbatim without a retry-after header, to reproduce Muna's 429 JSON errors.
  • SWE A/B campaigns: the provider is a data file. A campaign file (benchmarks/swe_ab/campaigns/<name>.yml: a provider config and the configurations, each a label, model id, and thinking effort) selects the provider; every plan names one in campaign, and swe_ab_stage0.py and swe_ab_sample.py take --campaign. The provider, base URL, key variable, and prices come from the campaign's omp models.yml (exactly one provider), so a new provider or model needs no code change. campaigns/muna.yml reproduces the three Muna configurations with the unchanged muna-models.yml; campaigns/openrouter-space-bunny.yml runs OpenRouter's free preview stealth/space-bunny-alpha with OPENROUTER_API_KEY (benchmarks/swe_ab/openrouter-models.yml).
  • Token ceiling for free models. Plans take token_ceiling (billed tokens) and the suite --token-ceiling (can only lower it). It is enforced before each cell like the dollar ceiling (exit status 3) and by validate; a docker run, the sampler, and validate refuse a campaign with an unpriced model unless it is set, since omp's cost model reads $0 there.

Changed

  • IndexStore owns the scratch directory it creates. IndexStore.scratch() makes a temporary directory and close() deletes that directory only; the delete_dir_on_close constructor flag, which removed any caller-supplied parent directory, is gone, so a plain IndexStore(path) never deletes anything.
  • Client setup is a per-client package. Shared plans and atomic write/merge helpers, MCP registration and discovery, per-host hook installers and templates, status-line adapters, and agent-file guidance have separate modules. The existing public archex.client_setup imports and generated client content are unchanged.
  • Campaign cells run the annotation hook with a 5-second budget. Every agent container gets ARCHEX_HOOK_TIMEOUT_SECONDS=5 and cells record hook_timeout_seconds (one value per stage). Under Rosetta emulation all 290 annotate calls timed in Stage 0 exceeded the shipped 0.5 s (medians 1.06–2.15 s), so the hook arms would otherwise have been inert. Stage 0's latency check uses the same budget.
  • omp starts on Alpine (musl) task images, through benchmarks/swe_ab/omp-entry.sh and a glibc loader shipped in the omp bundle; the protonmail and some teleport images are musl. Task images are pulled with retries, Stage 0 keeps going when one instance's container cannot start, and its in-container omp check uses the bundle's entry point (--container-omp-command).
  • The SWE A/B campaign moves from subscriptions to open models through a campaign file. The four subscription models are replaced by three configurations, qwen-3.8-27b@low, qwen-3.8-27b@high, and gemma-4-26b-a4b-it@high (Gemma ignored effort on Muna, so it runs at high only), in campaigns/openrouter.yml (draft) and campaigns/muna.yml. Cells are artifact version 3 and the validator refuses any cell whose provider base URL is not the campaign's (which is how rehearsal-stub cells are refused), whose provider is not the campaign's, whose credential variables are not exactly the campaign's key variable, or whose provider-config SHA-256 or thinking differs from the campaign's. Plans, cells, and analysis strata use the configuration label.
  • The suite schedules for provider capacity. Cells run by (model family, task, configuration, arm, repetition) so one model stays loaded, --prune-images frees a task's image per family, and a rate-limit or capacity block is re-run after --block-cooldown-seconds (default 300) up to --quota-retries per invocation.
  • Stage 2 is sized from the pilot instead of fixed at 100 tasks, and the pre-registration records an A0 feasibility check (18 cells, never data) before Stage 1.

Removed

  • The omp auth broker integration. The broker, its usage guard (QuotaGuard, quota readings, --broker-* flags, exit status 4), the agent.db login reader, and Stage 0's model_routes_and_logins, broker_healthy, broker_usage_readable, and broker_reachable_from_container checks are deleted.

Fixed

  • A verifier that runs past its 3,000-second cap now scores the cell unresolved instead of recording a harness error with zero tokens. Stage 0 records an instance it cannot score as an error instead of aborting the whole run.
  • A re-run of a cell no longer reads the previous attempt's session. On a suite re-run after a block, docker cp nested the new session inside the old attempt's directory and the runner parsed the old one, so a re-run that succeeded would have been recorded at the blocked attempt's usage and cost. The runner now moves an earlier attempt's files to <work dir>/prior-attempts/<n>/ before each attempt.
  • The hook-activity gate counts only searches the hook could annotate. Its denominator excludes searches with no hits, and "after the first edit" (stale-index declines that do not count) is decided by call order, so a grep issued in the same request as the first edit and run after it is no longer charged to the hook.
  • install-client writes client configs atomically. Every file it writes (the MCP config, the agent file, and the hook, post-edit, session-primer and status-line files) went through Path.write_text, which truncates before it writes, so a kill mid-install could leave ~/.claude.json or ~/.codex/config.toml empty. The new content now goes to a temporary file in the target's directory, is synced, and is renamed over the target. A symlinked target is followed, so a dotfile-managed link survives; an existing file keeps its mode, and a new file gets 0666 minus the umask; a failed write raises and removes its temporary file. The target's directory must now be writable, not only the file. archex still takes no lock, since the clients do not honour one: installing while a client is running can lose a concurrent client update, which docs/INSTALLATION_TRUST_CONTRACT.md now states.
  • Status snapshot failures reach the diagnostics log. Snapshot publication during indexing and watch-observation stamping remain non-fatal, but unexpected failures now record status_snapshot_publish_error or status_snapshot_watch_stamp_error with the exception and cache/repository path instead of disappearing at debug level.

Validation

  • Tagged commit: de1c555e5717e013dc8f1d05b48491c08491bcf1 (PR #697 merge). The tag is annotated and the distributions were built from its detached worktree.
  • Main-branch CI, Docker builds, and status-card checks passed on that merge commit. The first Python 3.11 CI attempt timed out in the Cursor diagnostic test; the unchanged-tree retry passed. No test or timeout was weakened.
  • Wheel and source distribution built successfully and passed uvx twine check. Wheel version, core SciPy/MCP dependencies, optional-extra gating, and the packaged client-setup modules were verified.
  • Core-only clean wheel install on Python 3.11 passed version reporting, real-source indexing, BM25/graph query, context fetch handles, and annotation in an initialized fixture.
  • Packaged MCP configuration and supported search-hook, post-edit-hook, and status-line installation passed for Claude Code, Codex, Cursor, omp, Pi, and OpenCode in isolated directories.
  • PyPI publication exposes both distributions with SHA-256 hashes matching the tagged build. A no-cache, isolated Python 3.11 install from the public PyPI index reports archex, version 0.35.0; it passed after the index propagation delay.