Repository navigation
v0.35.0
[0.35.0] - 2026-10-15
This release makes client-config installation atomic, reports unexpected status snapshot failures in the diagnostics log, and gives temporary index stores explicit scratch-directory ownership. Client setup is split into per-client modules without changing its existing public imports or generated content. The R3x SWE A/B harness gains deterministic sampling, pre-registered analysis, provider-selected campaigns, and an MCP arm; these are harness capabilities, not evidence of token savings or improved solve rates.
Re-run archex install-client <host> with the hook and status-line options you use to regenerate client files through the atomic writer. Consumers that passed delete_dir_on_close to IndexStore must use IndexStore.scratch() for an owned temporary store; caller-supplied directories are never deleted by a plain IndexStore(path).
Added
- R3x population sampler.
scripts/swe_ab_sample.pydraws the Stage 1 (24 tasks) and Stage 2 SWE-bench Pro V2 samples deterministically: a seed fixed in code, a sha256 rank order within each repository, Stage 1 at two per repository plus one for the two largest, Stage 2 by largest-remainder allocation and disjoint from Stage 1. Stage 2 takes--stage2-tasks N(at least 100, recorded in the manifest); the pre-registered size is max(100, ⌈max(370, n(m)) / k⌉) for the k configurations that clear the Stage 1 solve-rate floor and the m primary arms kept after the adoption gates (209 for k = 3, m = 3).--without-hcand--without-mdrop an arm from Stage 2. Instances that fail Stage 0 validity are replaced by the next valid instance of the same repository; the tool emits the suite's plan JSON, a manifest of every walked instance, and the Stage 0 instance list. - R3x pre-registered analysis.
scripts/swe_ab_analysis.pycomputes exactly the pre-registration's primary metrics on a validated result directory: per-configuration geometric-mean billed-token ratios for H, HC, and M against A0, readsuperior,worse,equivalent, orinconclusive(Holm over the arms, one family per direction); the pooled solve-rate guardrail and a two-sided pooled completion test per arm (better,worse,no_difference_detected), with the configuration as stratum; repository-clustered bootstrap intervals (10,000 resamples, seed 20260909), McNemar tests, a sensitivity check without blocked pairs, and every Stage 1 gate (solve-rate floor, lever share, HC and MCP adoption, hook activity, A0-vs-A0 noise with the implied power at the Stage 2 size). It refuses mixed-emulation stages and unscored cells. - R3x MCP arm, M. The archex MCP server registered in the agent's omp profile as
archex install-client ompinstalls it (mcp.jsonwith one server,archex mcp; archex not onPATH, no hook);--toolsaddsmcp__archex_contextandmcp__archex_query_repo, the server's default first surface. Cells recordmcp_config_sha256,archex_mcp_calls, andarchex_mcp_tools, and the channel table gainsarchex-MCP(artifact version 3). The shared omp overlay setsmcp.startupTimeoutMs: 0, without which omp rejected the allow-list before the server registered its tools. Stage 0 addsmcp_tools_in_m_armandmcp_call_executes_in_m_arm; a profile directory carrying its ownmcp.jsonis refused. - R3x OpenRouter campaign.
campaigns/openrouter.ymlruns the three configurations through OpenRouter with each model pinned to one upstream host (openRouterRouting.only: DeepInfra for Qwen 3.8 27B, NextBit for Gemma 4 26B A4B, both bf16) and those hosts' prices (benchmarks/swe_ab/openrouter-open-models.yml). - Solve-rate floor. Stage 1 gate
solve_floor: a configuration whose A0 solve rate (both repetitions) is under 15% is uninformative for the guardrail. With--pilot-analysis, Stage 2's pooled guardrail excludes it (reported per configuration, labelled as uninformative) and readsnot_estimableif none pass; efficiency stays per configuration. - Cells record the inputs the Stage 1 gates need.
out_of_patch_read_tokens_compounded(reads of files outside the gold and test patches) and the ledger'snot_fresh_after_first_edit; Stage 0 addsannotate_latency_in_container, the steady-state annotate latency inside each Pro image against the hook's 0.5 s budget. - Stage 0 checks for the campaign's provider route.
provider_route(omp's catalog lists every configuration's selector under the campaign's provider),provider_key_accepted(a zero-token chat request naming a nonexistent model is rejected for its model, e.g. Muna 404model_not_foundor OpenRouter 400, while 401/403 fail; the key is never printed),effort_request_shape(each configuration's first request carries itsreasoning_effort, which catches Qwen sending none), andprovider_reachable_from_container;one_real_cell_per_modelbecomesone_real_cell_per_configuration. - Credit exhaustion stops the suite. A cell that ends in HTTP 402 or an insufficient-credits error is recorded as
credit_exhausted, filed underquota-blocked/, never retried or scored; running cells finish and the suite exits with status 5, resumable after a top-up. Muna's response shape for it is unverified. - Frozen provider config and omp overlay.
benchmarks/swe_ab/muna-models.ymldeclares the Muna provider and its models (Qwen withcompat.thinkingFormat: openai, so--thinkingreaches the request asreasoning_effort) and holds Muna's prices (USD per million tokens, input / cached input / output: Qwen 3.8 27B 0.25 / 0.02 / 1.75, Gemma 4 26B 0.065 / 0.02 / 0.30), which feed omp's modelled cost and so the cost ceiling; a docker run refuses to start while any input, output, or cached-input price is 0 unless the plan sets a token ceiling.benchmarks/swe_ab/omp-campaign.yml(--config) raises omp's retry budget to 20 so a capacity 429 is retried for about 13 minutes (measured 785–789 s against the stub, 398–400 s by default) and keepsretry.waitForUsageResetoff. Cells record both files' SHA-256 and the resolved provider base URL. scripts/swe_ab_stub_provider.pyhttp_errorsteps take abody, returned verbatim without aretry-afterheader, to reproduce Muna's 429 JSON errors.- SWE A/B campaigns: the provider is a data file. A campaign file (
benchmarks/swe_ab/campaigns/<name>.yml: a provider config and the configurations, each a label, model id, and thinking effort) selects the provider; every plan names one incampaign, andswe_ab_stage0.pyandswe_ab_sample.pytake--campaign. The provider, base URL, key variable, and prices come from the campaign's ompmodels.yml(exactly one provider), so a new provider or model needs no code change.campaigns/muna.ymlreproduces the three Muna configurations with the unchangedmuna-models.yml;campaigns/openrouter-space-bunny.ymlruns OpenRouter's free previewstealth/space-bunny-alphawithOPENROUTER_API_KEY(benchmarks/swe_ab/openrouter-models.yml). - Token ceiling for free models. Plans take
token_ceiling(billed tokens) and the suite--token-ceiling(can only lower it). It is enforced before each cell like the dollar ceiling (exit status 3) and byvalidate; a docker run, the sampler, andvalidaterefuse a campaign with an unpriced model unless it is set, since omp's cost model reads $0 there.
Changed
IndexStoreowns the scratch directory it creates.IndexStore.scratch()makes a temporary directory andclose()deletes that directory only; thedelete_dir_on_closeconstructor flag, which removed any caller-supplied parent directory, is gone, so a plainIndexStore(path)never deletes anything.- Client setup is a per-client package. Shared plans and atomic write/merge helpers, MCP registration and discovery, per-host hook installers and templates, status-line adapters, and agent-file guidance have separate modules. The existing public
archex.client_setupimports and generated client content are unchanged. - Campaign cells run the annotation hook with a 5-second budget. Every agent container gets
ARCHEX_HOOK_TIMEOUT_SECONDS=5and cells recordhook_timeout_seconds(one value per stage). Under Rosetta emulation all 290 annotate calls timed in Stage 0 exceeded the shipped 0.5 s (medians 1.06–2.15 s), so the hook arms would otherwise have been inert. Stage 0's latency check uses the same budget. - omp starts on Alpine (musl) task images, through
benchmarks/swe_ab/omp-entry.shand a glibc loader shipped in the omp bundle; the protonmail and some teleport images are musl. Task images are pulled with retries, Stage 0 keeps going when one instance's container cannot start, and its in-container omp check uses the bundle's entry point (--container-omp-command). - The SWE A/B campaign moves from subscriptions to open models through a campaign file. The four subscription models are replaced by three configurations,
qwen-3.8-27b@low,qwen-3.8-27b@high, andgemma-4-26b-a4b-it@high(Gemma ignored effort on Muna, so it runs athighonly), incampaigns/openrouter.yml(draft) andcampaigns/muna.yml. Cells are artifact version 3 and the validator refuses any cell whose provider base URL is not the campaign's (which is how rehearsal-stub cells are refused), whose provider is not the campaign's, whose credential variables are not exactly the campaign's key variable, or whose provider-config SHA-256 or thinking differs from the campaign's. Plans, cells, and analysis strata use the configuration label. - The suite schedules for provider capacity. Cells run by (model family, task, configuration, arm, repetition) so one model stays loaded,
--prune-imagesfrees a task's image per family, and a rate-limit or capacity block is re-run after--block-cooldown-seconds(default 300) up to--quota-retriesper invocation. - Stage 2 is sized from the pilot instead of fixed at 100 tasks, and the pre-registration records an A0 feasibility check (18 cells, never data) before Stage 1.
Removed
- The omp auth broker integration. The broker, its usage guard (
QuotaGuard, quota readings,--broker-*flags, exit status 4), theagent.dblogin reader, and Stage 0'smodel_routes_and_logins,broker_healthy,broker_usage_readable, andbroker_reachable_from_containerchecks are deleted.
Fixed
- A verifier that runs past its 3,000-second cap now scores the cell unresolved instead of recording a harness error with zero tokens. Stage 0 records an instance it cannot score as an error instead of aborting the whole run.
- A re-run of a cell no longer reads the previous attempt's session. On a suite re-run after a block,
docker cpnested the new session inside the old attempt's directory and the runner parsed the old one, so a re-run that succeeded would have been recorded at the blocked attempt's usage and cost. The runner now moves an earlier attempt's files to<work dir>/prior-attempts/<n>/before each attempt. - The hook-activity gate counts only searches the hook could annotate. Its denominator excludes searches with no hits, and "after the first edit" (stale-index declines that do not count) is decided by call order, so a grep issued in the same request as the first edit and run after it is no longer charged to the hook.
install-clientwrites client configs atomically. Every file it writes (the MCP config, the agent file, and the hook, post-edit, session-primer and status-line files) went throughPath.write_text, which truncates before it writes, so a kill mid-install could leave~/.claude.jsonor~/.codex/config.tomlempty. The new content now goes to a temporary file in the target's directory, is synced, and is renamed over the target. A symlinked target is followed, so a dotfile-managed link survives; an existing file keeps its mode, and a new file gets0666minus the umask; a failed write raises and removes its temporary file. The target's directory must now be writable, not only the file. archex still takes no lock, since the clients do not honour one: installing while a client is running can lose a concurrent client update, whichdocs/INSTALLATION_TRUST_CONTRACT.mdnow states.- Status snapshot failures reach the diagnostics log. Snapshot publication during indexing and watch-observation stamping remain non-fatal, but unexpected failures now record
status_snapshot_publish_errororstatus_snapshot_watch_stamp_errorwith the exception and cache/repository path instead of disappearing at debug level.
Validation
- Tagged commit:
de1c555e5717e013dc8f1d05b48491c08491bcf1(PR #697 merge). The tag is annotated and the distributions were built from its detached worktree. - Main-branch CI, Docker builds, and status-card checks passed on that merge commit. The first Python 3.11 CI attempt timed out in the Cursor diagnostic test; the unchanged-tree retry passed. No test or timeout was weakened.
- Wheel and source distribution built successfully and passed
uvx twine check. Wheel version, core SciPy/MCP dependencies, optional-extra gating, and the packaged client-setup modules were verified. - Core-only clean wheel install on Python 3.11 passed version reporting, real-source indexing, BM25/graph query, context fetch handles, and annotation in an initialized fixture.
- Packaged MCP configuration and supported search-hook, post-edit-hook, and status-line installation passed for Claude Code, Codex, Cursor, omp, Pi, and OpenCode in isolated directories.
- PyPI publication exposes both distributions with SHA-256 hashes matching the tagged build. A no-cache, isolated Python 3.11 install from the public PyPI index reports
archex, version 0.35.0; it passed after the index propagation delay.