v1.0.0-rc.2

Second release candidate, themed "Evidence": check a trace's integrity, write behavior contracts over what the agent did, get statistical verdicts and judge budgets, and judge with OpenAI, Anthropic, Gemini or a local ACP agent.

Added

• Native Anthropic (Messages API) and Gemini (generateContent) judges: ATTEST_JUDGE_PROVIDER=anthropic or gemini, with keys from ATTEST_JUDGE_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_API_KEY then GEMINI_API_KEY, ATTEST_JUDGE_MODEL required, and ATTEST_JUDGE_BASE_URL for gateways. Each provider has its own rate limit (ATTEST_RATE_LIMIT_ANTHROPIC[_BURST], ATTEST_RATE_LIMIT_GEMINI[_BURST]); an outage is not_evaluated: provider_unavailable. doctor reports the new providers and key sources. See docs/guides/judges.md.
• Judge results record the provider in judge_metadata.provider next to model; reports print judge: <provider>/<model>.
• attest calibrate --persist stores labels keyed by judge provider and model (--judge-provider, --judge-model), so calibration is never pooled across judges.
• ATTEST_JUDGE_PROVIDER=acp runs judges locally through an Agent Client Protocol agent started by ATTEST_JUDGE_ACP_COMMAND (for example a Claude Code or Codex ACP adapter) on the developer's own subscription. The engine refuses it when CI or a CI vendor variable such as GITHUB_ACTIONS is defined. The agent runs in an empty temporary directory and its own process group, gets no file-system or terminal capability, and has every permission request rejected; it is not sandboxed, and the engine warns about this at startup. Every call, including simulated-user turns from generate_user_message, is bounded by ATTEST_JUDGE_TIMEOUT_S. The agent's environment loses provider keys and endpoint switches (OPENAI_API_KEY, OPENAI_BASE_URL, CODEX_API_KEY, ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_BASE_URL, CLAUDE_CODE_USE_BEDROCK, CLAUDE_CODE_USE_VERTEX, GEMINI_API_KEY, GOOGLE_API_KEY, ATTEST_JUDGE_*), and logs name only the agent executable. Rate limit: ATTEST_RATE_LIMIT_ACP[_BURST]. See docs/guides/judges.md.
• attest trace check <otlp.json> reports the integrity of an OpenTelemetry trace export (orphan spans, parent cycles, duplicate span ids, unpaired tool calls and results, each with its span id) and its evidence coverage (model spans without tokens, cost, message content or model; tool spans without call id or content), as text or --json, exiting 1 on integrity errors. attest trace convert <otlp.json> turns the export into an Attest trace. Conversion from OTLP now runs in the engine on the current OpenTelemetry GenAI semantic conventions; malformed input is rejected with an error naming the span. See docs/guides/trace-check.md.
• Model-call budgets enforced by reservation: ATTEST_MAX_JUDGE_CALLS caps judge requests and ATTEST_BUDGET_USD caps the spend of judge and embedding calls per engine session; evaluate_batch accepts a per-batch budget (max_judge_calls, max_cost_usd). Every judge call, repeat run, bias probe, simulated-user message, retry and paid embedding call reserves one call and its worst-case cost (price × prompt bytes + price × output-token limit) before it is sent and is reconciled to the reported token usage, so concurrent assertions cannot overspend (a failed call keeps its reservation; a request without an output limit is capped at 1024 tokens); a call that cannot reserve is not sent and its assertion is not_evaluated: budget_exhausted. Prices come from a built-in table (gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, text-embedding-3-small, text-embedding-3-large) or ATTEST_JUDGE_PRICE_IN / ATTEST_JUDGE_PRICE_OUT (USD per 1M tokens); a USD budget with an unpriced judge or embedding model, or with the acp judge, is a startup error. See docs/guides/configuration.md and docs/guides/judges.md.
• evaluate_batch results carry usage: model calls and the judge calls among them, input, output and cached tokens, reconciled cost_usd, retry_cost_usd, unpriced_calls and budget_refused. Report-v2 accepts it as summary.usage; the engine's markdown report prints model calls · tokens (cached) · $ · retry cost. Judge results cost the same per token on every provider (Anthropic and Gemini judges previously reported $0).
• Judges can abstain: built-in rubrics (now version v2) ask for {"abstain": true, "reason": "..."} when the output lacks the evidence the rubric needs, and the assertion is not_evaluated: judge_abstained with the reason. Only the structured field counts; no phrase in an explanation is matched. Abstentions are not cached, and abstaining repeat runs and bias probes are left out.
• attest policy evaluate --baseline <tag> with a regression: {confidence, tolerance} rule (defaults 0.95 and 0.05) compares the pass rate of each assertion with at least 2 runs in both the report and the baseline (fixed-N trials) with the baseline using a Wilson interval; single runs keep the block_on_regression status comparison, and trial assertions it cannot compare are inconclusive and returns verdict pass, regressed (exit 1) or inconclusive (exit 2) with per-assertion intervals. Report-v2 rows accept a 1-based trial and test rows a trials count. See docs/guides/verdicts.md.
• attest baseline pin stores each assertion's passes, evaluated runs and not-evaluated runs, folding the rows of fixed-N trials into one entry; baseline show prints them. The baselines table in $ATTEST_CACHE_DIR/attest.db gains the columns on first use, and rows pinned earlier count as one run.
• Fixed-N trials: pytest --attest-trials=N runs every test that uses the attest fixture N times, and the Vitest trials() helper (N from n or ATTEST_TRIALS) runs a test body N times. During a trial evaluate() records without enforcing; the test fails when the pass rate of its evaluated trials is below the minimum pass rate (default 1.0, so any failing trial fails it; --attest-min-pass-rate / @pytest.mark.attest_min_pass_rate(...), Vitest minPassRate on attestGlobalSetup or trials(), or ATTEST_MIN_PASS_RATE), not-evaluated trials are left out of the rate, and a trial that raises counts as a failed trial whose rows are replaced by one hard_fail row per assertion. Report rows carry trial and test rows trials; attest policy evaluate --baseline <tag> with a regression rule turns them into pass, regressed or inconclusive. Assertion ids inside a test are derived from the test id and the assertion's position, so they match across trials and runs. See docs/guides/statistical-verdicts.md.
• Per-test budgets: @pytest.mark.attest_budget(max_judge_calls=..., max_cost_usd=...) and the Vitest testBudget: { maxJudgeCalls, maxCostUsd } option of evaluate/inspect (budget on trials()) send the test's remaining budget with each evaluation; agent spend from metadata.cost_usd is checked between trials only, and further trials stop once the test budget is spent.
• The pytest and Vitest run summaries print attest usage: <calls> model calls · <tokens> tokens (<cached> cached) · $<cost> · retry cost $<cost>, and reports record it in summary.usage. The SDKs know the budget_exhausted and judge_abstained reasons.
• Trace schema version 2 (proto/attest/v1/trace-v2.schema.json, fixtures in protocol-tests/trace-v2/): optional conversation turns, agent handoffs, conversation_id, and per-step retry_of, error, recorded request/response exchange, span_id, parent_span_id and tool_call_id. The engine accepts schema versions 0–2; version 1 traces are unchanged.
• Behavior-contract trace checks in the engine (Layer 3, type: "trace"): tool_cardinality (calls per tool, optionally per argument value, reported as issue_refund called 2× for order_id=O-1042 (max 1)), trajectory (strict, unordered, subset, superset, partial_order), tool_order, cycle (period 1–4), failure_streak and arg_predicate. They match tool calls only and count retried calls. Specs are listed in proto/attest/v1/protocol-spec.md.
• Behavior-contract DSL in Python and TypeScript: tool_cardinality / toolCardinality, trajectory, trajectory_partial_order / trajectoryPartialOrder, tool_called_before / toolCalledBefore, no_tool_cycles / noToolCycles, max_tool_failure_streak / maxToolFailureStreak and tool_arg_satisfies / toolArgSatisfies. Both SDKs emit identical specs, checked by protocol-tests/contracts/dsl.json. See docs/guides/behavior-contracts.md.

Changed

• The loop_detection trace check shares the tool_cardinality counter; its failure explanation now reads fetch called 3× (max 2). Its result status is unchanged.
• BREAKING: attest calibrate --persist requires --judge-provider and --judge-model. The calibration table in $ATTEST_CACHE_DIR/attest.db gains judge_provider and judge_model columns on first use; labels stored before have neither and are not counted for any provider.
• BREAKING: The Python OTelAdapter converts spans through the engine (attest-engine trace convert) instead of its own mapping, so it needs an engine and returns trace v2. Tool steps take args and result from gen_ai.tool.call.arguments / gen_ai.tool.call.result (previously call_id, parameters and output from non-standard attributes; the call id is now tool_call_id), a top-level invoke_agent span becomes a conversation turn instead of an agent_call step, and malformed spans raise attest.exceptions.OTLPConversionError. An engine binary is required (the SDK downloads its engine version, or set ATTEST_ENGINE_PATH to one built with make engine). OTelAdapter.from_otlp() converts an OTLP/JSON export given as a pathlib.Path, JSON text, bytes or a parsed mapping. Span events, span kind and resource attributes are not read. to_otel_spans() writes the current semantic convention attributes under an invoke_agent root span. Trace.to_dict() omits parent_trace_id when it is unset.
• BREAKING: The TypeScript OTelAdapter.fromSpans() converts spans through the engine and returns a Promise<Trace> (trace v2) instead of a Trace. Spans need spanContext().spanId; parentSpanId (sdk-trace-base 1.x) or parentSpanContext (2.x) and status are read. OTelAdapter.fromOtlp() converts an OTLP/JSON export, and malformed input rejects with OTLPConversionError. Invoke-agent, retrieval and execute_tool spans, previously dropped or misread, now convert as in Python. new OTelAdapter(agentId) is removed (use the static methods), span attributes must be OpenTelemetry attribute values, and fromSpans([]) rejects (an export needs at least one span). The Step and Trace types gain the trace v2 fields.
• Built-in judge rubrics are version v2 (abstain instruction added), so calibration labels stored against v1 no longer match them.
• repeat: N judges take the mean of the two middle scores for an even number of scored runs (previously the upper middle score).
• Gemini judge usage counts thinking tokens (thoughtsTokenCount) as output tokens, as they are billed.
• Judge and embedding calls stop when the batch's context is cancelled.
• attest policy evaluate --baseline folds the rows of fixed-N trials into one row per assertion (worst status, mean score, summed cost and duration) before computing transitions.
• Judge cache entries are keyed by the whole judge prompt (rubric version, criteria and agent output) instead of the agent output alone. Two passes_judge assertions with different criteria on one output previously shared one cached score. Entries cached by earlier versions are misses and are judged again.
• Cached judge scores are scoped by judge provider as well as endpoint, rubric and model, so two providers behind the same gateway with the same model name never share a score. Entries cached by earlier versions are misses and are judged again.
• Judge provider error messages have any echoed API key replaced by [redacted] and control and Unicode format characters (bidi overrides, zero-width) removed before they reach results and reports.
• A constraint on a trace field the trace does not record, such as cost_under without metadata.cost_usd, explains its not_evaluated: missing_evidence result as metadata.cost_usd absent.
• The documentation website moved to https://attest-framework.github.io/ (repository attest-framework/attest-framework.github.io, formerly attest-website). The release and SDK CI website dispatches target the renamed repository, and https://attest-framework.github.io/attest/ redirects to the new root. The old /attest-website/ path no longer resolves.

Removed

• The MkDocs site configuration (mkdocs.yml) and the pages it built outside docs/guides/. https://attest-framework.github.io/attest/ already redirects to the documentation website, which syncs docs/guides/*.md at each release.

Fixed

• Install instructions in the README, docs/guides/merge-gate.md, docs/guides/framework-adapters.md and the Python SDK README (the PyPI page) use uv add "attest-ai>=1.0.0rc1" instead of pip install, and npm install -D "@attest-ai/core@>=1.0.0-rc.1" "@attest-ai/vitest@>=1.0.0-rc.1". A bare pip install attest-ai, uv add attest-ai or npm install @attest-ai/core installs 0.9.0, the newest stable release, until 1.0.0 is published.

Published components

Component	Status
Engine binaries and checksums (this release)	published
PyPI attest-ai==1.0.0rc2	published
npm @attest-ai/core@1.0.0-rc.2	published
npm @attest-ai/vitest@1.0.0-rc.2	published
Website sync	dispatched (release-sync to attest-framework/attest-framework.github.io)