v1.0.0 — experiment lab

Release v1.0.0 — experiment lab

Date: 2026-08-14
Hosted: https://pax-agentic-rag.vercel.app · Vercel project pax-agentic-rag only
Baseline season doc: docs/SEASON.md

Summary

P2 graduates from a bounded research demo to a free-path experiment lab: experiment
records, n≥40 control goldens with difficulty predicates, three tools with per-tool budgets,
payload experiment packs, honest load numbers, and a CASESTUDY that states what is still
PLANNED.

Added

• ExperimentRecord (id, seed, budget, note ids, stop reason, pack hash, tool calls)
• Golden control set 48 cases (was 18), including tool_budget and more unanswerable /
  thin / budget slices; critic-notes-exist-not-success still refuses
• Third tool: fixture lexicon lookup; per-tool max_calls in state; stop reason
  tool_budget_spent (ADR-0006)
• Experiment pack: policy + two run payloads + compare + experiments + pack_hash
  (POST /v1/experiments/pack, /pack UI, directory/zip round-trip tests)
• Load artifact: docs/assets/load.json (n=50 free-path researches, p50/p95, cold start)
• Optional max_tool_calls on research request (OpenAPI)

Still PLANNED (honest)

• Durable multi-instance run storage (DB / object store)
• Live web tools or default paid LLM planner/critic
• Hosted HTTP-P1 without captain-configured PRODUCTION_RAG_URL
• Tier-2 quality judges in CI
• Production capacity claims from single-process load.json
• Agent plugin marketplace / unrestricted tool registry
• Second Vercel project

Hosted contract

• Project: pax-agentic-rag · URL: https://pax-agentic-rag.vercel.app
• Never production-rag.vercel.app (Ipsura)
• Fixture retriever default; HTTP P1 opt-in; CI offline
• Smoke: pwsh scripts/hosted_smoke.ps1

Verify

pytest -q
python -m agentic_rag.evals.run
python scripts/load_fake_research.py --n 50
pwsh scripts/hosted_smoke.ps1

v0.3.0 — compare payloads, hosted smoke, critic-can-lose

agentic-rag-research v0.3.0 — compare, hosted contract, critic can lose

Date: 2026-08-14
Hosted: https://pax-agentic-rag.vercel.app (Vercel project pax-agentic-rag only)

What this release proves

A hiring manager can put two finished runs next to each other after a serverless recycle
forgets in-memory ids. Compare is on payloads, not server ids
(ADR-0005). The free-path critic can still lose
when notes exist but are off-topic. Hosted contract smoke is a script + transcript, not a
flaky network CI job.

Surface

Endpoint / page	Role
POST /v1/research	Bounded research run (unchanged)
GET /v1/research/stream	SSE plan → retrieve → note → critique
GET /v1/runs/{id}	Stored artifact (in-process, ephemeral)
GET /v1/runs/{id}/run.json	Full run download (compare input)
GET /v1/runs/{id}/trace.json	Trace-only download
POST /v1/runs/compare	Typed diff of two payloads
/compare	Load two files; side-by-side + API diff
scripts/hosted_smoke.ps1	health · done · refuse · stream · compare

Evidence

python -m pytest -q
python -m agentic_rag.evals.run
# expect pass_rate 1.0, billed_usd 0.0, 18/18
pwsh scripts/hosted_smoke.ps1 -BaseUrl https://pax-agentic-rag.vercel.app

• Identical fixtures → empty, byte-stable compare response.
• Refused vs done → typed field diffs (status, stop_reason, …).
• Golden critic-notes-exist-not-success: notes present, status refused.
• Multi-hop sequence: critique after retrieve-1 precedes and justifies retrieve-2.

Honesty

• Fixture retriever only on the public host. No live-web research claim.
• HTTP to production-rag stays opt-in; CI never points at a production-rag Vercel host.
• Control metrics only — steps, stop reasons, citation presence, refused_unanswerable.
• No OpenAI required for tests or the free path.

Upgrade notes

Clients that only downloaded trace.json should also keep run.json if they want to
compare after recycle. Compare request body is { "left": <RunArtifact>, "right": <RunArtifact> }.

v0.2.0 — auditable free path

agentic-rag-research v0.2.0 — auditable free path

Three-week depth campaign after v0.1.0: notes as a store, runs as artifacts, streaming
step UI, and control-only evals. Hosted fixture demo:
https://pax-agentic-rag.vercel.app.

What this release is

The free path is still credential-free and still makes no retrieval or answer quality
claim. What changed is auditability:

1. Notes store — typed claims with provenance; critic scores grounded, on-topic notes.
2. Run artifacts — GET /v1/runs/{id} and downloadable traces by correlation id.
3. Streaming — SSE plan → retrieve → critique with stable offsets.
4. Control scorecard — steps, stop reasons, citation presence, refused unanswerable.

Included beyond v0.1.0

• ADR-0004 — why notes are structured; why the free
  critic is not an LLM.
• Bounded in-process run store (default 32); process-local, not durable across isolates.
• GET /v1/research/stream and UI that watches it, then fetches the stored run.
• Scorecard fields for stop-reason distribution and unanswerable refusal rate.
• CASESTUDY 10-minute DEMO: refused run + downloaded stored trace.

Evidence boundary (unchanged honesty)

Measured	Not measured
Budgets, stop reasons, citation presence	Answer quality
Trace completeness and byte-stable free path	Live-web research
Off-corpus refusal	Uplift vs single-pass or vs GPT

Optional HTTP-to-P1 tests require RUN_P1_INTEGRATION=1 and a local
PRODUCTION_RAG_URL. CI leaves them skipped. Never point the suite at a public
production-rag host.

Free verification

python -m pytest -q
python -m agentic_rag.evals.run
# expect pass_rate 1.0, billed_usd 0.0, 17/17

Hosted DEMO steps: CASESTUDY.md.

agentic-rag-research v0.1.0 — bounded research, visible decisions

agentic-rag-research v0.1.0 — bounded research, visible decisions

These notes describe M1–M6 on the exact release commit. The tag is created only after the
SHIP.md gate and exact-SHA CI pass.

What this release is

agentic-rag-research is a bounded research loop that plans sub-questions, retrieves
evidence, critiques coverage, and either produces a cited report or stops with an explicit
reason. Its default path is deterministic, local, and credential-free.

This is portfolio series project #2. It consumes the evidence contract from
production-rag through an optional HTTP
boundary instead of copying that service's hybrid retrieval stack.

Included

• Library, FastAPI, and CLI surfaces over the same research loop.
• done, refused, and budget_exhausted terminal outcomes.
• Step budgets, no-repeat progress, named evidence gaps, resolvable citations, and a full
  plan/tool/critique/synthesis/stop trace.
• A default fake retriever over 20 packaged passages: no key, network, signup, or billed
  provider call.
• An opt-in production-rag HTTP retriever with mock-transport and contract tests. A live
  end-to-end service result is not claimed by this release draft.
• A 17-case deterministic evaluation set spanning five behavior slices. The JSON scorecard
  reports expectation pass rate, mean steps, citation presence, and terminal-status counts.
• A dark accessible UI at / with the report, resolved citations, status, retrieval steps,
  request id, expandable trace timeline, and typed HTML failures.
• A deterministic search_notes tool the critic may request to inspect notes already
  gathered. It cannot retrieve, generate, write, or contact a provider.

Evidence boundary

The fake evaluation measures control-flow conformance against a committed fixture. It does
not measure retrieval quality, answer quality, faithfulness, latency, production readiness,
or agent uplift over a one-pass answer baseline. The current runner has no such baseline.

Free verification

python -m pip install -e ".[dev]"
python -m pytest -q
python -m agentic_rag.evals.run --pretty
python -m agentic_rag.research \
  --question "Why use reciprocal rank fusion?" --retriever fake
python -m uvicorn agentic_rag.main:app --port 8010

Open http://127.0.0.1:8010/ for the UI. Expected evaluation evidence: 17 cases evaluated
with provider fake, billed cost $0, and every declared expectation passing.

Known limits

• No hosted demo, hosted-model quality result, authentication, rate limiting, streaming, or
  multi-tenancy.
• No arbitrary web, shell, filesystem, write, or sub-agent tools.
• HTTP retrieval requires a separately running, compatible production-rag service.
• Multi-agent coordination belongs to series project #3, not this release.