v0.8.0: install from PyPI, arci demo, arci.yaml, arci init

v0.8.0 — install from PyPI, arci demo, arci.yaml, arci init

The onboarding release. Nothing about the decision rule, its calibration, the record schema
(arci/0.5) or any recorded result changed; every archived decision still re-derives byte for byte.

Install

uvx --from agent-reliability-ci arci demo            # the recorded retry story, no clone, no model call
uv tool install agent-reliability-ci                  # the arci command

PyPI status: the release workflow built the wheel and the sdist from the tag, smoke-tested the wheel
on Ubuntu and macOS (Python 3.11 and 3.14) from an empty directory, and published both files to PyPI
with PEP 740 attestations through trusted publishing (run 38365660814; the publish job succeeded on
its re-run at 08:25Z, after the trusted publisher was registered).

What is new

• PyPI package and release pipeline. A version tag builds the wheel and sdist once, smoke-tests
  that wheel on Ubuntu and macOS (Python 3.11 and 3.14) from an empty directory, and publishes the
  same files through PyPI trusted publishing with attestations (.github/workflows/release.yml,
  docs/RELEASING.md, design record 0005).
• arci demo. The six-step retry story from an installed wheel: the frozen experiment BLOCKs the
  candidate that dropped a retry (A 192/200 vs B 132/200 at 200 trials per arm), the failing trace is
  diffed, the three injected faults shrink to the one that matters, the reduced bundle replays
  offline, and the repaired agent passes. The demo computes its counts fresh and says whether they
  matched the recorded run; the recorded run itself is unchanged evidence
  (docs/results/retry-demo-n200.md, docs/reports/ci-gate-2026-10-06/logs/hero-demo-2026-10-05.txt).
• arci.yaml. A thin descriptor whose keys are the sealed manifest's fields with the same
  defaults, compiled and sealed exactly as the Python builders do; arci run takes it directly and
  --out defaults to runs. arci schema prints the JSON Schema generated from the model
  (docs/schema/arci.schema.json, docs/DESCRIPTOR.md, design record 0007).
• arci init. One command, four references (baseline, candidate, toolset, oracle), a validated,
  runnable smoke descriptor (8 trials per arm: the smallest design in which a total collapse can
  still BLOCK; expect INCONCLUSIVE). References are checked by importing them in a child process,
  the way a trial would (design record 0008).
• Honest smoke runs. arci run prints a stderr notice for designs under 100 trials per arm and
  points at arci plan; stdout, reports and records are unchanged.
• The current directory is on the child import path under the arci console script, as it
  already was under python -m arci.cli (design record 0006).
• CI. Wheel smoke on Ubuntu and macOS, an install-and---help-only Windows job (execution stays
  unsupported on Windows), and an Action fixture that must BLOCK.
• Measurement protocol. docs/measurement/: time-to-first-verdict and cold-read protocol with a
  receipt template; failed attempts are kept; no telemetry.
• Recipes. examples/recipes/python_agent (a blank project: four references, arci init, the
  smoke run, arci plan, the confirmatory run, then diff, minimize, replay; docs/recipes/python-agent.md)
  and examples/recipes/mcp_agent (a stdlib JSON-RPC client against the MCP toolset bridge with a
  hand-written command-arm descriptor; docs/recipes/mcp-inspector.md). The Inspector preflight in
  that recipe is recorded as blocked on 2026-10-13: Inspector 2.10.1 (TypeScript SDK 1.29.0) rejects
  the bridge's protocol revision 2026-07-28; the ARCI steps run without it.
• README. The first screen is the question, one sentence, two commands, what PASS means, what a
  small run means, and the boundary (design record 0009).
• Also in this version, from main since v0.7.0: arci plan, opt-in arci report --pass-k, the
  platform-portability fixes, the technical report package and the guardrail results.

Recorded runs this release cites (none new)

• Seeded retry demo, 2026-09-23: 192/200 vs 132/200, bounds [-0.405, -0.183], BLOCK.
• Jev day one, 2026-10-03: 200/200 vs 0/200, BLOCK; minimised failure replays REPRODUCED.
• Guardrail, 2026-10-05: 200/200 vs 130/200, BLOCK; guardrail-live-b-ask-provider_down-50 (50/50 vs 38/50,
  bounds [-0.4016, -0.0348]) INCONCLUSIVE, kept as is.
• Local model, N=400: 376/400 vs 308/400, INCONCLUSIVE by 0.008, not re-run.

Measured in this release's run (orchestrator's machine, 15-core Apple Silicon)

• examples/retry_agent/hero_demo.py --n 200 from the checkout (the same six steps): 38 s. Through uvx --from <the 0.8.0 wheel> arci demo in an empty
  directory, including environment creation: 80 s with a cold uv cache (9 s resolve/install + 71 s
  run), 72 s and 73 s warm. Own-agent path on the sample tree (arci init + arci run at
  N=8): 3 s to a validated INCONCLUSIVE. Receipts in docs/measurement/receipts/, labelled as
  maintainer-machine runs; no cold-reader measurement yet.

Limits that have not moved

PASS is relative non-inferiority within the declared margin, never absolute reliability. Not a
sandbox. Only calls that cross the tool boundary are seen. Replay is of the boundary, not the
model. INCONCLUSIVE is common at small N and is the honest answer. POSIX only. See "What it does
not do" in the README and docs/TRUST_MODEL.md.

Verify

shasum -a 256 -c SHA256SUMS                                   # the attached files
uvx --from agent-reliability-ci==0.8.0 arci demo

Built by release workflow run 38365660814 from tag v0.8.0 (commit 48e60ac):

10ddc6823b446edd745d7904c5ab24a2fa098a0b9313ccc2df1b3277c4212899  agent_reliability_ci-0.8.0-py3-none-any.whl
99476b7b8923724eeb6392f7fa37ba4189c6fc5bb40d2ce9f10d7b3d578a9a01  agent_reliability_ci-0.8.0.tar.gz

The same bytes come out of uv build --no-sources on a clean checkout of that commit (verified on the
maintainer's machine: identical checksums). Wheel built by hatchling 1.32.4, Metadata-Version 2.5.

v0.7.0

v0.7.0 (2026-10-05)

• Calibration audit of the decision model (2026-10-03): bench/jev_calibration/ is a
  pre-registered audit of jev-1.13.0 on CLINC150 (4,500 in-scope + 1,000 out-of-scope) and
  Banking77 (3,080), three passes each plus a bundling check, 26,140 requests, about USD 4.6
  at list price. PROTOCOL.md (hash posted on PR #2 before the first request) fixes the
  questions, passes, measures and seven expectations; run.py seals every answer in a
  hash-chained store; metrics.py (exact binomial intervals, ECE/MCE/Brier/log loss,
  tie-aware AUROC and average precision, bootstrap, agreement) is unit tested; analyze.py
  verifies the chains and writes metrics.json, report.md, reliability diagrams and a
  self-verifying compact store from which every number re-derives. All seven expectations
  met: zero-shot accuracy 0.921 [0.912, 0.928] on CLINC150 (a supervised TF-IDF+LR trained on
  15,000 examples: 0.925) with ECE 0.024; 0.801 on Banking77 with ECE 0.084 and mid-range
  over-confidence; 0.970 accuracy at confidence >= 0.9 on 83% coverage; out-of-scope AUROC
  0.977 (yes/no question) against 0.911 from 1 - p_max; 99.4% repeat agreement; p50 322 ms.
  Write-up: docs/results/jev-calibration/README.md.
• Added: examples/jev_triage_agent/experiment.py builds live manifests
  (live_manifest, decisions.upstream = "http", pinned model, paced trial starts, prior
  runs) and writes them from the command line
  (python -m examples.jev_triage_agent.experiment --out manifest.json), so the day-one
  sequence in docs/design/0004-day-one.md runs from files. examples/jev_triage_agent/day_one.sh
  drives it end to end: preflight, the clean pair, the N=50 trio, the Node agent through the
  official SDK, the pre-registered N=200 run, and a minimised, replayed failure, with a gate
  report per run and a refusal to finish if the key appears in any output. Results:
  docs/results/jev/.
• Run against Jev (2026-10-03): the pre-registered day one ran end to end against
  https://api.typesafe.ai with jev-1.13.0 pinned. A vs B under decision_low_confidence at
  N=200: 200/200 vs 0/200, bounds [-1.000, -0.957], BLOCK; A vs A and A vs C PASS at N=50; the
  Node agent through the unchanged official SDK BLOCK; the minimised failure replays REPRODUCED;
  an exploratory clean run at N=50 routed all 100 decisions correctly at confidence 1.0. 902
  recorded decisions, about USD 0.018 at list price. Sealed stores, the preflight receipt, the
  invalid first attempt and one write-up per run: docs/results/jev/; every decision.json
  re-derives from its committed trials.jsonl. Findings: docs/DECISIONS.md, "What day one
  found".
• Added: arci preflight --allow-unlisted-model. The account's GET /v1/models lists only
  the aliases jev-latest and jev-preview, so the pre-registered preflight exits 2 on a pinned
  versioned id. With the flag, preflight prints the listed names, sends the smoke request, and
  exits 0 only if the validated response reports exactly the pinned id (3 otherwise). Without the
  flag nothing changes except a hint after the exit-2 message. Tests:
  tests/unit/cli/test_preflight_unlisted_model.py.
• Added: ARCI_HTTPS_PROXY=http://host:port makes the decision upstream sender tunnel through
  an HTTP CONNECT proxy, for environments whose only egress is such a proxy; unset, the direct
  connection is unchanged, and a malformed value is a transport error. Tests:
  tests/unit/decisions/test_upstream_proxy.py.
• Fixed: the three preflight smoke questions carry instructions; the vendor rejects a
  question with neither instructions nor criteria (400), so the pre-registered smoke request
  could not be accepted as written.
• Fixed: examples/jev_triage_agent/day_one.sh puts the repository root on PYTHONPATH
  (the manifests name examples.jev_triage_agent.* modules, invisible to the arci console
  script), insists on running from the repository root, and stops with exit 4 unless both arms of
  the N=1 clean pair PASS, so a harness fault that breaks both arms equally cannot be mistaken
  for a measurement.
• Fixed: two reader threads, the grader's response reader and the MCP boundary's frame
  reader, could end a read on a "not ready" poll verdict computed before the child wrote and
  before the parent asked them to stop. On a loaded machine the thread was descheduled across
  the child's exit, the stale verdict dropped the whole response, and the trial ended ERROR
  (grader returned invalid output, or boundary exited before returning for command agents)
  at about 1 trial in 400 with 8 workers on 4 cores, which is enough to turn a 200-per-arm
  experiment into an ERROR verdict on a busy CI runner. The worker reader already followed
  the correct rule; both readers now snapshot the stop flag before polling, and only a poll
  made after the stop was observed may end the read. Regression tests:
  tests/unit/runner/test_reader_stop_race.py.

evidence-2026-10-06: ARCI CI-gate evidence package

ARCI evidence package, 2026-10-06

Technical report, operating-characteristics metrics, archived examples, checksums and reproduction tooling: docs/reports/ci-gate-2026-10-06/.
Source commit: 78af270 (merge of #9; PR-head CI: https://github.com/ajaysurya1221/agent-reliability-ci/actions/runs/37351932149; post-merge CI on main: https://github.com/ajaysurya1221/agent-reliability-ci/actions/runs/37353490939; first attempt: three jobs passed and check / Python 3.11 failed on the known ERROR-trial flake in the frozen hero-decisions test (fourth occurrence, same tree had passed on the PR head); the failed job was re-run: completed/success).
SHA256SUMS SHA-256: 35520d8027f9ceaeaa10ac4c340d97340f39f11a0e6d1f86cd4569217e98da57

1. ARCI binds declared decision rules into trial identity and rejects inconsistent rules and incomplete stores.
2. The planner enumerates the existing Clopper-Pearson gate for independent arms, K=1, one look and N <= 400; 0.95 vs 0.75 at N=200 gives P(BLOCK) = 0.3647316956588898.
3. All 18 released Jev and guardrail decisions, including five invalid-attempt decisions, re-derive byte for byte on the tested macOS arm64 and Linux x86-64 environments.
4. Released examples connect regression verdicts to boundary divergence and offline failure replay; live reductions remain labelled "reduced".
5. frontier-scout a28f27d's published matrix rejects out-of-scope and unsupported-approval controls in enforcing mode and accepts benign scope controls; unsigned approval remains UNVERIFIED.

Verify from a checkout of the source commit with its environment installed (uv sync --frozen):

(cd docs/reports/ci-gate-2026-10-06 && shasum -a 256 -c SHA256SUMS)
.venv/bin/python docs/reports/ci-gate-2026-10-06/reproduce.py --check --full --strict

For container replay, follow docs/reports/ci-gate-2026-10-06/reproduction/README.md.
Claim limits and the must-not-claim list: REPORT.md section 7. Verifier matrix and remaining limitations: REPORT.md Appendix A.

v0.6.0

v0.6.0 (2026-09-22)

Day-one Jev readiness. The upstream behavior below is derived from vendor and official SDK
documentation and tested against shaped fixtures; it was not run against Jev because no key was
available.

• A1: HTTP decision upstreams accept a path-prefixed base URL, including the Vercel AI Gateway
  TypeSafe route and its typesafe-ai/jev model id.
• A2: Response validation accepts the tested SDK response shapes: unknown metadata is tolerated and rounded
  probability sums are accepted within 1e-3, while required keys and pinned-model checks remain.
• A3: Harness-owned request headers, allowlisted response headers, deterministic transport
  diagnostics, and one recorded bounded wait on an upstream 429/529 make gateway attempts auditable.
• B4: arci preflight MANIFEST checks credentials, model discovery and one tiny smoke decision,
  then emits a credential-free receipt with exit 0/2/3 semantics.
• B5: HTTP experiments pace trial starts through a shared parent token bucket; users divide the
  provider rate by the maximum decisions per trial.
• B6: Markdown reports count completed decision recordings and their available token usage,
  and show estimated input cost at the list price read on 2026-09-22 (fixtures and replays are
  free; this is an estimate, not billed spend). Incomplete admissions are excluded; before-injected
  results contribute zero tokens.
• B7: The Jev guide now covers the three-command day-one workflow, provider limits, privacy,
  sampled answers and live-run minimality; docs/results/TEMPLATE-jev.md standardises write-ups.
• C8: The triage example gains a Node 20+ ES-module agent using @typesafe-ai/sdk 0.6.0 through
  environment-only new TypeSafeClient(), plus CI coverage on Python 3.14 / Node 22.
• C9: Decision records retain an independent raw upstream snapshot before perturbation; live
  HTTP upstreams, removal trials ending in ERROR, and replay INVALID results report reduced, never
  1-minimal.
• D10: The offline triage fixture can inject seeded synthetic miscalibration for testing
  confidence gates; it is explicitly not evidence about Jev's calibration.