Skip to content

v1.0.0-rc.3

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 03 Nov 16:43
· 481 commits to main since this release
bf735df

Third release candidate, themed "Flight recorder": capture a real run or incident as a verified fixture, replay the agent's real code against it in CI with zero model calls, check what the agent did to a simulated world with effect contracts, and get a pull-request summary, inline annotations, JUnit and diff-based test selection.

Added

  • attest capture --name <name> -- <command> turns a run into a regression fixture. The engine starts a loopback listener hosting a recording proxy for OpenAI Chat Completions, OpenAI Responses and Anthropic Messages (streaming and not) and an OTLP/HTTP trace receiver (http/protobuf and http/json, optionally gzip), runs the command with OPENAI_BASE_URL, ANTHROPIC_BASE_URL and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT pointing at it, forwards model calls to the upstreams set by those variables in the capture's own environment (default https://api.openai.com/v1 and https://api.anthropic.com), and writes tests/attest/fixtures/<name>.attest.json (--out-dir to change). Request headers are never recorded; values of environment variables whose names end in KEY, TOKEN, SECRET or PASSWORD and key-shaped strings (sk-…, AKIA…, ghp_…, bearer credentials, private keys) are replaced by [redacted] in keys and values. Trace and span ids, provider response and tool-call ids, timestamps, durations and emails are templated to stable placeholders across the trace, the exchanges and their replay keys, so capturing the same scripted run twice writes the same bytes (timestamps, numeric or ISO-8601, are numbered by position, never by value: the fixture is walked with each object's start-like keys first and end-like keys last, so every start precedes its end and a parent brackets its children, and runs that differ only in timing write the same bytes). A model call the agent cancels or does not read to the end, and an OTLP export request still in flight at the end or rejected, fail the capture, since the fixture could not replay the run. SIGINT and SIGTERM sent to attest capture are forwarded to the command. --from <otlp.json> builds a trace-only fixture (not replay-capable) from an export; --serve hosts the listeners for in-process helpers. attest capture --check-redaction <fixture> exits 1, naming the JSON path, when a fixture holds a secret.
  • Capture drafts deterministic contracts from the run as ceilings on what it observed (the tools called, the order of their first calls, per-id cardinality for each tool called with an id / *_id argument, the structured output's shape, and a cost ceiling at 1.5× the observed cost) and marks the fixture verified only when they all pass and the captured trace passes trace check; otherwise it exits 1 naming the reason. A normal capture drafts each tool's cardinality at the observed per-id maximum, so a healthy run that reads an id twice verifies. --expect-fail saves an incident reproduction: cardinality is drafted at one call per id, the fixture is verified when drafted contracts fail, and those are recorded as expected failures (INC-1042: cardinality-issue_refund-per-order_id).
  • Python capture helpers: with attest.record("<name>", expect_fail=...) captures the code in the block by starting attest-engine capture --serve and setting OPENAI_BASE_URL, ANTHROPIC_BASE_URL and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for the block (construct model clients and the OpenTelemetry exporter inside it; the global tracer provider is flushed on exit), and raises attest.CaptureError when the fixture is not verified. Captures cannot overlap (record() sets process-wide variables). attest.load_fixture(path) reads a fixture. For a verified fixture, attest capture and record() draft tests/attest/test_<name>.py (name lowercased, other characters as _) from its contracts with the expect() DSL, unless that file exists: a verified fixture's test enforces them with attest.evaluate, an incident reproduction's test checks with attest.inspect that the expected failures still fail.
  • TypeScript capture helpers: record(name, fn, { expectFail, flush }) from @attest-ai/core captures what fn does through attest-engine capture --serve (pass the tracer provider's forceFlush as flush) and throws CaptureError when the fixture is not verified; loadFixture(path) reads a fixture. For a verified fixture, npx attest capture and record() draft tests/attest/<name>.test.ts with attestExpect and inspect / evaluate from @attest-ai/vitest, unless that file exists. Captures cannot overlap. Every drafted contract has a TypeScript DSL call. See docs/guides/capture.md.
  • Fixture schema v1 (proto/attest/v1/fixture-v1.schema.json, fixtures in protocol-tests/fixture-v1/): the trace-v2 trace, the exchanges with a replay key (model, parameters, prompt hash, tool-schema hash), the drafted contracts, their verification and a world section for the simulated world's final state (null until --world fills it). attest trace check accepts a fixture.
  • attest-engine replay --name <name> -- <command> serves a fixture's recorded model responses to the real agent: the engine's proxy answers every OpenAI Chat Completions, OpenAI Responses and Anthropic Messages request from the fixture (streamed exchanges replay as events) and makes no provider call. Requests are matched on model, parameters, prompt and tool schemas after the capture redactor is applied to the live request and every volatile value (ids, timestamps, emails, and the {{id_N}} placeholders a replayed response hands back to the agent) is renumbered by first occurrence within that request, so a live request needs no placeholder numbering of its own; identical retries consume successive recordings in order. --mode strict (default) answers a request with no recording with an error naming the first component that changed, for example replay miss: system prompt changed (hash a1b2c3d→e4f5a6b); re-record with \attest capture --refresh inc-1042`; --mode autoforwards it live only when the request carries exactly theOPENAI_API_KEYorANTHROPIC_API_KEYof the engine's environment for that provider (the proxy adds no credentials), and otherwise reports the miss;--no-liverefuses every live forward. A model name in a miss message is quoted.replay --serveis the long-lived form for SDK helpers (environment line,stop`, summary line with replayed, live and missed requests and the cost of live calls the built-in price table covers).
  • attest capture --refresh <name> re-runs the fixture's recorded command with its original flags (--expect-fail included) and overwrites the fixture. The command comes from the fixture file, so it is printed shell-quoted first and runs only after --yes or a y at a terminal (otherwise exit 2, nothing runs); it refuses when the command's arguments were redacted or templated, or the fixture came from --from or an in-process capture.
  • Report-v2 gains optional replay provenance: tests[].replay (fixtures, mode, replayed, live, misses, miss) and summary.replay (replayed, live, misses, live_cost_usd, unpriced_live_calls). summary.usage keeps counting only attest-initiated model calls. Valid and invalid examples are in protocol-tests/report-v2/.
  • replay_miss joins the not_evaluated reasons (ReasonReplayMiss in Go, proto/attest/v1/report-v2.schema.json, protocol-tests/verdicts/verdict.schema.json): a fixture-bound test in auto replay mode without a provider key reports it. The SDKs set it; the engine's own batches never return it.
  • Python replay: the attest_replay("<name>") pytest fixture (and attest.start_replay()) runs the agent's real code against a fixture's recorded model responses. --attest-replay=strict|auto|off (also ATTEST_REPLAY; default strict when CI is set, auto otherwise) chooses what a request with no recording does: strict fails the test with the engine's miss message, naming the changed component and the attest capture --refresh command, whatever the agent raised and whatever marker the test carries (xfail(strict=True) and Vitest it.fails cannot turn it into an expected failure); auto goes live when a provider key is set and otherwise skips the test as not_evaluated: replay_miss (a failure under --attest-strict). The terminal summary prints attest replay: N replayed · M live model calls · $X.XX, and --attest-report writes the per-test and summary replay provenance. See docs/guides/replay.md.
  • TypeScript replay: @attest-ai/core adds startReplay() and @attest-ai/vitest adds useReplay(name, { mode, egress, strict }), with the same modes (ATTEST_REPLAY=strict|auto|off) and outcomes. AttestReporter writes the replay provenance and prints the same summary line.
  • Provider-egress guard: pytest --attest-egress=block, ATTEST_EGRESS=block or useReplay(name, { egress: "block" }) fails in-process connections to api.openai.com, api.anthropic.com, generativelanguage.googleapis.com and *.openai.azure.com; loopback is always allowed. It is off by default. With the guard on, replay runs the engine with --no-live, so no miss goes live. Control and format characters in miss and engine-error text are escaped before they reach a test report.
  • Attest's own L6 judge integration tests (sdks/python/tests/integration/test_judge_replay.py) run passes_judge through ATTEST_JUDGE_BASE_URL pointed at the replay proxy, from fixtures recorded against a scripted OpenAI-compatible judge. make record-judge-fixtures re-records them against a real provider; it needs ATTEST_JUDGE_API_KEY (or OPENAI_API_KEY) and ATTEST_JUDGE_MODEL and refuses to run without them.
  • attest-engine world --file world.yaml -- <command> serves the external system an agent acts on as a local stateful HTTP fake. world.yaml (schema proto/attest/v1/world-v1.schema.json, fixtures in protocol-tests/world-v1/) declares JSON resources with create, list and get routes, an idempotency-key header for creates, seed records and a fault script, and names the environment variable that carries the world's base URL (for example PAYMENTS_BASE_URL). A repeated idempotency key replays the first response and commits nothing. Faults: timeout_after_commit commits the create, then holds the response until the client disconnects (30 s cap), so the outcome never depends on a wall-clock race; 429 and 500 fire on listed request ordinals or by probability from a seeded generator, and the same seed and request sequence give the same outcomes. A malformed file exits 2 naming the JSON path (/resources/refunds/routes/1). --serve prints {"env":...}, serves until stop, then prints {"world":<final snapshot>}; --state-out writes the snapshot. GET <base URL>/_attest/state returns the current state as a world-v1 snapshot.
  • attest capture --world world.yaml and attest-engine replay --world world.yaml host the world in the same session as the recording or replaying proxy. Capture stores the world's final state (world_version, definition hash, state, faults fired and their count) in the fixture's world section, redacted like the rest of the fixture, and email addresses in record keys and values are templated to {{email_N}} with the same numbering as the trace (one address, one token across the fixture; ids and timestamps in records are kept as the world held them). The section is evidence only: replay --world starts a fresh world from world.yaml, not from the fixture. A capture without --world keeps null. attest capture --refresh refuses a fixture that carries a world section, since a refresh would drop it.
  • An over-long request line (over 10 MiB) no longer kills the engine: it is answered with a -32600 REQUEST_TOO_LARGE error carrying the request's id (null when the id is not in the first bytes) and the engine keeps serving.
  • effect assertions (Layer 3, deterministic, no new layer) check what the agent did to the world: count of matching records (equals, at_most, at_least), unique by fields, and idempotent keys. The spec embeds a world-v1 snapshot as its evidence, so the evaluator interface is unchanged. A violation reads, for example, effect violated: 2 refunds for O-1042 (expected 1). A null snapshot is not_evaluated with reason missing_evidence. See proto/attest/v1/protocol-spec.md.
  • Python world fixtures: attest.start_world(path) (and the attest_world(path) pytest fixture) hosts a world.yaml through attest-engine world --serve and sets its base URL variable (PAYMENTS_BASE_URL) until the test ends; the session has env, base_url, state() (the current world-v1 snapshot, from GET <base URL>/_attest/state) and stop() (the final snapshot). It composes with attest_replay in one test (two engine processes, merged environment). attest.record(..., world=path) captures with --world. expect(trace).world(snapshot) adds count(resource, equals=, at_most=, at_least=, **where), unique(resource, by=, **where) and idempotent(resource, key=, **where) effect checks; world(None) is not_evaluated (missing_evidence).
  • TypeScript world fixtures: startWorld(path) from @attest-ai/core and useWorld(path) from @attest-ai/vitest (composes with useReplay) with the same session, record(name, fn, { world }), and attestExpect(trace).world(snapshot) with count(resource, { equals, atMost, atLeast, where }), unique(resource, { by, where }) and idempotent(resource, { key, where }). Both SDKs emit the same specs; protocol-tests/contracts/effect-dsl.json pins them and the engine's result for each.
  • docs/guides/world-fixtures.md: the world model, the world.yaml reference with the INC-1042 payments example, faults and determinism, idempotency semantics, running a world, the effect DSL, composing with replay, and limits.
  • Effect checks guard the request size: expect(...).world(snapshot) measures the snapshot once and raises ValueError / Error before sending when the checks of one chain would embed it more than 9 MiB in total (MAX_WORLD_BYTES), since the engine limits a request line to 10 MiB. The Python codec sends UTF-8 instead of \uXXXX escapes, so non-ASCII text no longer inflates up to 6x. In Python resource is positional-only and filters can be given as a where dict (use it for fields named key, by or equals); a filter given both ways raises TypeError. TypeScript layerForType("effect") returns 3, like the engine.
  • attest report summary --report <path>... [--policy-result <file>] prints the pull-request summary as Markdown: ### attest: N passed · N failed · N not evaluated (plus · N skipped for plain skips and expected failures), N replayed · N live model calls · $X, the usage line with retry cost, not-evaluated counts by reason, a capped table of failures and replay misses with file:line, and the policy verdict. Several reports merge; unreadable input exits 2.
  • attest report junit --report <path>... --out <file> writes JUnit XML from report-v2 (one suite per test file, file and line on each case, <failure> for failed tests, <skipped> for skipped tests, with the reasons when their assertions were not evaluated). attest report rows --report <path>... --json prints the failures and replay misses as {file, line, test_id, assertion_id, type, status, reason, explanation} rows. A strict-mode replay miss, which fails its test without an assertion row, appears as a replay_miss row.
  • attest select --base <ref> [--json] [--repo <dir>] lists the tests a diff needs: touched test files, and the tests that reference a touched fixture by capture name. Lockfiles, dependency manifests, attest config, world files and any file it does not recognize select all; markdown selects nothing. Exit 2 when git or the ref fails.
  • Guides docs/guides/baselines.md, policies.md and selection.md.
  • pr-annotate v2 (.github/actions/pr-annotate, ships in rc.3): report accepts several paths (newline or space separated) and the policy runs on each, the worst exit winning (an evaluator error, then 1, 2, 0). It posts one summary comment per pull request, rendered by attest report summary plus a link to the run, found by the hidden marker <!-- attest:pr-annotate --> and updated in place; only marker comments written by the token's own login are updated (the Actions bot, or the user for a personal token), so a planted marker is ignored. The summary is always written to the job summary too. The comment is posted only on same-repository pull_request events; fork pull requests and tokens that cannot write get the job summary and a notice or warning, never a failed step. Failures and replay misses from attest report rows become ::error (hard failures, replay misses) and ::warning (soft failures, not-evaluated) annotations at the test's file and line, relative to the repository root (a row's file is prefixed with its report's directory), escaped per the workflow-command rules, with control characters stripped, messages capped, and at most 10 errors and 9 warnings per step (one annotation counts the rest). The engine never receives the token, the policy result is written under RUNNER_TEMP and not echoed, and ##[ is broken up in every logged string. The engine is found as engine-binary, then ATTEST_ENGINE_PATH, then PATH, then ~/.attest/bin/<engine-version>/attest-engine (engine-version defaults to the action's own VERSION). The optional comment-author input names the login whose comment is updated (<app-slug>[bot] for a GitHub App token, which cannot look itself up). Inputs and the exit-code output are unchanged. See docs/guides/annotations.md.
  • attest init --ci github writes an attest report junit step and uploads attest-junit.xml with the report, and sets permissions to contents: read for the workflow with pull-requests: write on the job only (both SDKs).

Changed

  • Behavior change: pr-annotate no longer posts pull request review comments (they could only target lines in the diff). Failures are workflow annotations and the summary is one updated comment; the action now runs attest-engine report summary and report rows, so the engine is required even without a policy. A caller pinned to @v1.0.0-rc.1 or @v1.0.0-rc.2 is unaffected until it moves to the rc.3 tag.
  • attest report summary renders every report-derived string (test ids, file paths, explanations, not-evaluated reasons, policy rules and details) as an inline code span, with control, bidi and zero-width characters removed and the fence longer than any backtick run in the text, so agent output in a pull request comment cannot ping users, link issues, inject HTML or hide the rest of the comment.
  • The engine builds with Go 1.27 (engine/go.mod go 1.27.0; CI and the release cross-compile use Go 1.27). Go 1.25 is outside Go's two-release support window and no longer gets security fixes. Prebuilt engine binaries are unaffected; building from source, including through the setup-attest action (which reads the version from engine/go.mod), needs Go 1.27 or later.
  • The Python and TypeScript SDK workflows build the engine (make engine, Go 1.27) before their tests and run when engine/** changes, so the engine-backed integration tests run in CI instead of skipping.

Security

  • The capture listener binds 127.0.0.1 on an ephemeral port and serves only under a random 128-bit path token handed to the command in its environment (other paths are 404 and nothing is recorded), refuses requests whose Host is not a loopback name or that carry an Origin header, admits 64 concurrent requests, bounds request and recorded response bodies at 32 MiB and a session's recordings at 512 MiB (the capture fails past it), checks OTLP protobuf span, attribute, value and nesting limits on the wire before decoding it, and closes when the command exits. Provider credentials pass through to the upstream in memory only. Redaction covers secrets streamed across several response deltas (Chat Completions content, refusal, reasoning content, audio transcript and function or tool-call arguments; every Responses *.delta event; Anthropic text, partial_json, thinking and citations), which become [redacted] in the first delta. A response is a stream when its media type is text/event-stream (any parameters) or the request asked for one; a stream that does not parse fails the capture rather than being written. It also covers environment names ending in _PASS, _PWD, PASSWD, CREDENTIALS, CREDENTIAL, DSN or AUTH (not PWD, OLDPWD or *_ASKPASS), the password of any URL with userinfo, Stripe, Groq, Hugging Face and xAI keys, percent-decoded exchange paths with key, api_key, apikey, token, access_token, sig and signature query values, the values of secret-named command flags and of NAME=value arguments whose name marks a secret, and secrets an in-process block loads into its environment. Names from the trace are escaped in terminal output.
  • The replay listener is the capture listener in another mode: 127.0.0.1 on an ephemeral port under a random 128-bit path token, loopback Host only, no Origin, 64 concurrent requests. A fixture is read with a 256 MiB size limit and refused when it has more than 10,000 exchanges, a truncated or empty exchange, an unknown provider or a status outside 100–599. Live requests are redacted before matching, so a secret in a request never reaches a miss message or a summary, and replay answers a miss with HTTP 422, which the official SDKs do not retry.
  • The world listener is the capture listener: 127.0.0.1 on an ephemeral port under a random 128-bit path token, loopback Host only, no Origin, 64 concurrent requests. world.yaml is untrusted input: at most 256 KiB, one YAML document, string keys only, no unquoted timestamps, validated against world-v1 before it is used. The world refuses request bodies over 1 MiB, holds at most 10,000 records and 1 MiB of state, faults listed included (507 beyond), lists at most 1,000 fired faults in a snapshot (faults_fired_count stays exact), rejects integers beyond 2^53-1 and non-finite numbers in request bodies and in world.yaml, accepts only the query filters a resource declares, and releases held responses when the session closes.

Fixed

  • @attest-ai/vitest declares vitest >=3.0.0 <5. Vitest 5 removed vitest/suite, which the plugin imports, so the old >=3.0.0 range accepted a Vitest it cannot load. The install commands in the README, docs/guides/behavior-contracts.md and docs/guides/merge-gate.md now pin vitest@^4, since a bare vitest installs Vitest 5. Vitest 5 support is planned.
  • TypeScript OTelAdapter.fromOtlp / fromSpans no longer hang for 60 s and throw trace convert timed out under parallel load. They ran the engine with spawnSync and the export on stdin, which on macOS occasionally never returned: about 1 call in 100 with 8 concurrent callers. The engine now runs as an asynchronous child process, which also stops the conversion from blocking the event loop.

Published components

Component Status
Engine binaries and checksums (this release) published
PyPI attest-ai==1.0.0rc3 published
npm @attest-ai/core@1.0.0-rc.3 published
npm @attest-ai/vitest@1.0.0-rc.3 published
Website sync dispatched (release-sync to attest-framework/attest-framework.github.io)