Repository navigation
v1.0.0-rc.3
Pre-release
Pre-release
·
481 commits
to main
since this release
Third release candidate, themed "Flight recorder": capture a real run or incident as a verified fixture, replay the agent's real code against it in CI with zero model calls, check what the agent did to a simulated world with effect contracts, and get a pull-request summary, inline annotations, JUnit and diff-based test selection.
Added
attest capture --name <name> -- <command>turns a run into a regression fixture. The engine starts a loopback listener hosting a recording proxy for OpenAI Chat Completions, OpenAI Responses and Anthropic Messages (streaming and not) and an OTLP/HTTP trace receiver (http/protobufandhttp/json, optionally gzip), runs the command withOPENAI_BASE_URL,ANTHROPIC_BASE_URLandOTEL_EXPORTER_OTLP_TRACES_ENDPOINTpointing at it, forwards model calls to the upstreams set by those variables in the capture's own environment (defaulthttps://api.openai.com/v1andhttps://api.anthropic.com), and writestests/attest/fixtures/<name>.attest.json(--out-dirto change). Request headers are never recorded; values of environment variables whose names end inKEY,TOKEN,SECRETorPASSWORDand key-shaped strings (sk-…,AKIA…,ghp_…, bearer credentials, private keys) are replaced by[redacted]in keys and values. Trace and span ids, provider response and tool-call ids, timestamps, durations and emails are templated to stable placeholders across the trace, the exchanges and their replay keys, so capturing the same scripted run twice writes the same bytes (timestamps, numeric or ISO-8601, are numbered by position, never by value: the fixture is walked with each object's start-like keys first and end-like keys last, so every start precedes its end and a parent brackets its children, and runs that differ only in timing write the same bytes). A model call the agent cancels or does not read to the end, and an OTLP export request still in flight at the end or rejected, fail the capture, since the fixture could not replay the run. SIGINT and SIGTERM sent toattest captureare forwarded to the command.--from <otlp.json>builds a trace-only fixture (not replay-capable) from an export;--servehosts the listeners for in-process helpers.attest capture --check-redaction <fixture>exits 1, naming the JSON path, when a fixture holds a secret.- Capture drafts deterministic contracts from the run as ceilings on what it observed (the tools called, the order of their first calls, per-id cardinality for each tool called with an
id/*_idargument, the structured output's shape, and a cost ceiling at 1.5× the observed cost) and marks the fixture verified only when they all pass and the captured trace passes trace check; otherwise it exits 1 naming the reason. A normal capture drafts each tool's cardinality at the observed per-id maximum, so a healthy run that reads an id twice verifies.--expect-failsaves an incident reproduction: cardinality is drafted at one call per id, the fixture is verified when drafted contracts fail, and those are recorded as expected failures (INC-1042:cardinality-issue_refund-per-order_id). - Python capture helpers:
with attest.record("<name>", expect_fail=...)captures the code in the block by startingattest-engine capture --serveand settingOPENAI_BASE_URL,ANTHROPIC_BASE_URLandOTEL_EXPORTER_OTLP_TRACES_ENDPOINTfor the block (construct model clients and the OpenTelemetry exporter inside it; the global tracer provider is flushed on exit), and raisesattest.CaptureErrorwhen the fixture is not verified. Captures cannot overlap (record()sets process-wide variables).attest.load_fixture(path)reads a fixture. For a verified fixture,attest captureandrecord()drafttests/attest/test_<name>.py(name lowercased, other characters as_) from its contracts with theexpect()DSL, unless that file exists: a verified fixture's test enforces them withattest.evaluate, an incident reproduction's test checks withattest.inspectthat the expected failures still fail. - TypeScript capture helpers:
record(name, fn, { expectFail, flush })from@attest-ai/corecaptures whatfndoes throughattest-engine capture --serve(pass the tracer provider'sforceFlushasflush) and throwsCaptureErrorwhen the fixture is not verified;loadFixture(path)reads a fixture. For a verified fixture,npx attest captureandrecord()drafttests/attest/<name>.test.tswithattestExpectandinspect/evaluatefrom@attest-ai/vitest, unless that file exists. Captures cannot overlap. Every drafted contract has a TypeScript DSL call. Seedocs/guides/capture.md. - Fixture schema v1 (
proto/attest/v1/fixture-v1.schema.json, fixtures inprotocol-tests/fixture-v1/): the trace-v2 trace, the exchanges with a replay key (model, parameters, prompt hash, tool-schema hash), the drafted contracts, their verification and aworldsection for the simulated world's final state (nulluntil--worldfills it).attest trace checkaccepts a fixture. attest-engine replay --name <name> -- <command>serves a fixture's recorded model responses to the real agent: the engine's proxy answers every OpenAI Chat Completions, OpenAI Responses and Anthropic Messages request from the fixture (streamed exchanges replay as events) and makes no provider call. Requests are matched on model, parameters, prompt and tool schemas after the capture redactor is applied to the live request and every volatile value (ids, timestamps, emails, and the{{id_N}}placeholders a replayed response hands back to the agent) is renumbered by first occurrence within that request, so a live request needs no placeholder numbering of its own; identical retries consume successive recordings in order.--mode strict(default) answers a request with no recording with an error naming the first component that changed, for examplereplay miss: system prompt changed (hash a1b2c3d→e4f5a6b); re-record with \attest capture --refresh inc-1042`;--mode autoforwards it live only when the request carries exactly theOPENAI_API_KEYorANTHROPIC_API_KEYof the engine's environment for that provider (the proxy adds no credentials), and otherwise reports the miss;--no-liverefuses every live forward. A model name in a miss message is quoted.replay --serveis the long-lived form for SDK helpers (environment line,stop`, summary line with replayed, live and missed requests and the cost of live calls the built-in price table covers).attest capture --refresh <name>re-runs the fixture's recorded command with its original flags (--expect-failincluded) and overwrites the fixture. The command comes from the fixture file, so it is printed shell-quoted first and runs only after--yesor ayat a terminal (otherwise exit 2, nothing runs); it refuses when the command's arguments were redacted or templated, or the fixture came from--fromor an in-process capture.- Report-v2 gains optional replay provenance:
tests[].replay(fixtures,mode,replayed,live,misses,miss) andsummary.replay(replayed,live,misses,live_cost_usd,unpriced_live_calls).summary.usagekeeps counting only attest-initiated model calls. Valid and invalid examples are inprotocol-tests/report-v2/. replay_missjoins thenot_evaluatedreasons (ReasonReplayMissin Go,proto/attest/v1/report-v2.schema.json,protocol-tests/verdicts/verdict.schema.json): a fixture-bound test inautoreplay mode without a provider key reports it. The SDKs set it; the engine's own batches never return it.- Python replay: the
attest_replay("<name>")pytest fixture (andattest.start_replay()) runs the agent's real code against a fixture's recorded model responses.--attest-replay=strict|auto|off(alsoATTEST_REPLAY; defaultstrictwhenCIis set,autootherwise) chooses what a request with no recording does:strictfails the test with the engine's miss message, naming the changed component and theattest capture --refreshcommand, whatever the agent raised and whatever marker the test carries (xfail(strict=True)and Vitestit.failscannot turn it into an expected failure);autogoes live when a provider key is set and otherwise skips the test asnot_evaluated: replay_miss(a failure under--attest-strict). The terminal summary printsattest replay: N replayed · M live model calls · $X.XX, and--attest-reportwrites the per-test and summary replay provenance. Seedocs/guides/replay.md. - TypeScript replay:
@attest-ai/coreaddsstartReplay()and@attest-ai/vitestaddsuseReplay(name, { mode, egress, strict }), with the same modes (ATTEST_REPLAY=strict|auto|off) and outcomes.AttestReporterwrites the replay provenance and prints the same summary line. - Provider-egress guard:
pytest --attest-egress=block,ATTEST_EGRESS=blockoruseReplay(name, { egress: "block" })fails in-process connections toapi.openai.com,api.anthropic.com,generativelanguage.googleapis.comand*.openai.azure.com; loopback is always allowed. It is off by default. With the guard on, replay runs the engine with--no-live, so no miss goes live. Control and format characters in miss and engine-error text are escaped before they reach a test report. - Attest's own L6 judge integration tests (
sdks/python/tests/integration/test_judge_replay.py) runpasses_judgethroughATTEST_JUDGE_BASE_URLpointed at the replay proxy, from fixtures recorded against a scripted OpenAI-compatible judge.make record-judge-fixturesre-records them against a real provider; it needsATTEST_JUDGE_API_KEY(orOPENAI_API_KEY) andATTEST_JUDGE_MODELand refuses to run without them. attest-engine world --file world.yaml -- <command>serves the external system an agent acts on as a local stateful HTTP fake.world.yaml(schemaproto/attest/v1/world-v1.schema.json, fixtures inprotocol-tests/world-v1/) declares JSON resources with create, list and get routes, an idempotency-key header for creates, seed records and a fault script, and names the environment variable that carries the world's base URL (for examplePAYMENTS_BASE_URL). A repeated idempotency key replays the first response and commits nothing. Faults:timeout_after_commitcommits the create, then holds the response until the client disconnects (30 s cap), so the outcome never depends on a wall-clock race;429and500fire on listed request ordinals or by probability from a seeded generator, and the same seed and request sequence give the same outcomes. A malformed file exits 2 naming the JSON path (/resources/refunds/routes/1).--serveprints{"env":...}, serves untilstop, then prints{"world":<final snapshot>};--state-outwrites the snapshot.GET <base URL>/_attest/statereturns the current state as a world-v1 snapshot.attest capture --world world.yamlandattest-engine replay --world world.yamlhost the world in the same session as the recording or replaying proxy. Capture stores the world's final state (world_version, definition hash, state, faults fired and their count) in the fixture'sworldsection, redacted like the rest of the fixture, and email addresses in record keys and values are templated to{{email_N}}with the same numbering as the trace (one address, one token across the fixture; ids and timestamps in records are kept as the world held them). The section is evidence only:replay --worldstarts a fresh world fromworld.yaml, not from the fixture. A capture without--worldkeepsnull.attest capture --refreshrefuses a fixture that carries a world section, since a refresh would drop it.- An over-long request line (over 10 MiB) no longer kills the engine: it is answered with a
-32600REQUEST_TOO_LARGEerror carrying the request's id (null when the id is not in the first bytes) and the engine keeps serving. effectassertions (Layer 3, deterministic, no new layer) check what the agent did to the world:countof matching records (equals,at_most,at_least),uniqueby fields, andidempotentkeys. The spec embeds a world-v1 snapshot as its evidence, so the evaluator interface is unchanged. A violation reads, for example,effect violated: 2 refunds for O-1042 (expected 1). Anullsnapshot isnot_evaluatedwith reasonmissing_evidence. Seeproto/attest/v1/protocol-spec.md.- Python world fixtures:
attest.start_world(path)(and theattest_world(path)pytest fixture) hosts aworld.yamlthroughattest-engine world --serveand sets its base URL variable (PAYMENTS_BASE_URL) until the test ends; the session hasenv,base_url,state()(the current world-v1 snapshot, fromGET <base URL>/_attest/state) andstop()(the final snapshot). It composes withattest_replayin one test (two engine processes, merged environment).attest.record(..., world=path)captures with--world.expect(trace).world(snapshot)addscount(resource, equals=, at_most=, at_least=, **where),unique(resource, by=, **where)andidempotent(resource, key=, **where)effect checks;world(None)isnot_evaluated(missing_evidence). - TypeScript world fixtures:
startWorld(path)from@attest-ai/coreanduseWorld(path)from@attest-ai/vitest(composes withuseReplay) with the same session,record(name, fn, { world }), andattestExpect(trace).world(snapshot)withcount(resource, { equals, atMost, atLeast, where }),unique(resource, { by, where })andidempotent(resource, { key, where }). Both SDKs emit the same specs;protocol-tests/contracts/effect-dsl.jsonpins them and the engine's result for each. docs/guides/world-fixtures.md: the world model, theworld.yamlreference with the INC-1042 payments example, faults and determinism, idempotency semantics, running a world, the effect DSL, composing with replay, and limits.- Effect checks guard the request size:
expect(...).world(snapshot)measures the snapshot once and raisesValueError/Errorbefore sending when the checks of one chain would embed it more than 9 MiB in total (MAX_WORLD_BYTES), since the engine limits a request line to 10 MiB. The Python codec sends UTF-8 instead of\uXXXXescapes, so non-ASCII text no longer inflates up to 6x. In Pythonresourceis positional-only and filters can be given as awheredict (use it for fields namedkey,byorequals); a filter given both ways raisesTypeError. TypeScriptlayerForType("effect")returns 3, like the engine. attest report summary --report <path>... [--policy-result <file>]prints the pull-request summary as Markdown:### attest: N passed · N failed · N not evaluated(plus· N skippedfor plain skips and expected failures),N replayed · N live model calls · $X, the usage line with retry cost, not-evaluated counts by reason, a capped table of failures and replay misses withfile:line, and the policy verdict. Several reports merge; unreadable input exits 2.attest report junit --report <path>... --out <file>writes JUnit XML from report-v2 (one suite per test file,fileandlineon each case,<failure>for failed tests,<skipped>for skipped tests, with the reasons when their assertions were not evaluated).attest report rows --report <path>... --jsonprints the failures and replay misses as{file, line, test_id, assertion_id, type, status, reason, explanation}rows. A strict-mode replay miss, which fails its test without an assertion row, appears as areplay_missrow.attest select --base <ref> [--json] [--repo <dir>]lists the tests a diff needs: touched test files, and the tests that reference a touched fixture by capture name. Lockfiles, dependency manifests, attest config, world files and any file it does not recognize selectall; markdown selects nothing. Exit 2 when git or the ref fails.- Guides
docs/guides/baselines.md,policies.mdandselection.md. pr-annotatev2 (.github/actions/pr-annotate, ships in rc.3):reportaccepts several paths (newline or space separated) and the policy runs on each, the worst exit winning (an evaluator error, then 1, 2, 0). It posts one summary comment per pull request, rendered byattest report summaryplus a link to the run, found by the hidden marker<!-- attest:pr-annotate -->and updated in place; only marker comments written by the token's own login are updated (the Actions bot, or the user for a personal token), so a planted marker is ignored. The summary is always written to the job summary too. The comment is posted only on same-repositorypull_requestevents; fork pull requests and tokens that cannot write get the job summary and a notice or warning, never a failed step. Failures and replay misses fromattest report rowsbecome::error(hard failures, replay misses) and::warning(soft failures, not-evaluated) annotations at the test's file and line, relative to the repository root (a row'sfileis prefixed with its report's directory), escaped per the workflow-command rules, with control characters stripped, messages capped, and at most 10 errors and 9 warnings per step (one annotation counts the rest). The engine never receives the token, the policy result is written underRUNNER_TEMPand not echoed, and##[is broken up in every logged string. The engine is found asengine-binary, thenATTEST_ENGINE_PATH, thenPATH, then~/.attest/bin/<engine-version>/attest-engine(engine-versiondefaults to the action's ownVERSION). The optionalcomment-authorinput names the login whose comment is updated (<app-slug>[bot]for a GitHub App token, which cannot look itself up). Inputs and theexit-codeoutput are unchanged. Seedocs/guides/annotations.md.attest init --ci githubwrites anattest report junitstep and uploadsattest-junit.xmlwith the report, and setspermissionstocontents: readfor the workflow withpull-requests: writeon the job only (both SDKs).
Changed
- Behavior change:
pr-annotateno longer posts pull request review comments (they could only target lines in the diff). Failures are workflow annotations and the summary is one updated comment; the action now runsattest-engine report summaryandreport rows, so the engine is required even without apolicy. A caller pinned to@v1.0.0-rc.1or@v1.0.0-rc.2is unaffected until it moves to the rc.3 tag. attest report summaryrenders every report-derived string (test ids, file paths, explanations, not-evaluated reasons, policy rules and details) as an inline code span, with control, bidi and zero-width characters removed and the fence longer than any backtick run in the text, so agent output in a pull request comment cannot ping users, link issues, inject HTML or hide the rest of the comment.- The engine builds with Go 1.27 (
engine/go.modgo 1.27.0; CI and the release cross-compile use Go 1.27). Go 1.25 is outside Go's two-release support window and no longer gets security fixes. Prebuilt engine binaries are unaffected; building from source, including through thesetup-attestaction (which reads the version fromengine/go.mod), needs Go 1.27 or later. - The Python and TypeScript SDK workflows build the engine (
make engine, Go 1.27) before their tests and run whenengine/**changes, so the engine-backed integration tests run in CI instead of skipping.
Security
- The capture listener binds
127.0.0.1on an ephemeral port and serves only under a random 128-bit path token handed to the command in its environment (other paths are 404 and nothing is recorded), refuses requests whoseHostis not a loopback name or that carry anOriginheader, admits 64 concurrent requests, bounds request and recorded response bodies at 32 MiB and a session's recordings at 512 MiB (the capture fails past it), checks OTLP protobuf span, attribute, value and nesting limits on the wire before decoding it, and closes when the command exits. Provider credentials pass through to the upstream in memory only. Redaction covers secrets streamed across several response deltas (Chat Completions content, refusal, reasoning content, audio transcript and function or tool-call arguments; every Responses*.deltaevent; Anthropic text,partial_json, thinking and citations), which become[redacted]in the first delta. A response is a stream when its media type istext/event-stream(any parameters) or the request asked for one; a stream that does not parse fails the capture rather than being written. It also covers environment names ending in_PASS,_PWD,PASSWD,CREDENTIALS,CREDENTIAL,DSNorAUTH(notPWD,OLDPWDor*_ASKPASS), the password of any URL with userinfo, Stripe, Groq, Hugging Face and xAI keys, percent-decoded exchange paths withkey,api_key,apikey,token,access_token,sigandsignaturequery values, the values of secret-named command flags and ofNAME=valuearguments whose name marks a secret, and secrets an in-process block loads into its environment. Names from the trace are escaped in terminal output. - The replay listener is the capture listener in another mode:
127.0.0.1on an ephemeral port under a random 128-bit path token, loopbackHostonly, noOrigin, 64 concurrent requests. A fixture is read with a 256 MiB size limit and refused when it has more than 10,000 exchanges, a truncated or empty exchange, an unknown provider or a status outside 100–599. Live requests are redacted before matching, so a secret in a request never reaches a miss message or a summary, and replay answers a miss with HTTP 422, which the official SDKs do not retry. - The world listener is the capture listener:
127.0.0.1on an ephemeral port under a random 128-bit path token, loopbackHostonly, noOrigin, 64 concurrent requests.world.yamlis untrusted input: at most 256 KiB, one YAML document, string keys only, no unquoted timestamps, validated against world-v1 before it is used. The world refuses request bodies over 1 MiB, holds at most 10,000 records and 1 MiB of state, faults listed included (507beyond), lists at most 1,000 fired faults in a snapshot (faults_fired_countstays exact), rejects integers beyond 2^53-1 and non-finite numbers in request bodies and inworld.yaml, accepts only the query filters a resource declares, and releases held responses when the session closes.
Fixed
@attest-ai/vitestdeclaresvitest >=3.0.0 <5. Vitest 5 removedvitest/suite, which the plugin imports, so the old>=3.0.0range accepted a Vitest it cannot load. The install commands in the README,docs/guides/behavior-contracts.mdanddocs/guides/merge-gate.mdnow pinvitest@^4, since a barevitestinstalls Vitest 5. Vitest 5 support is planned.- TypeScript
OTelAdapter.fromOtlp/fromSpansno longer hang for 60 s and throwtrace convert timed outunder parallel load. They ran the engine withspawnSyncand the export on stdin, which on macOS occasionally never returned: about 1 call in 100 with 8 concurrent callers. The engine now runs as an asynchronous child process, which also stops the conversion from blocking the event loop.
Published components
| Component | Status |
|---|---|
| Engine binaries and checksums (this release) | published |
PyPI attest-ai==1.0.0rc3 |
published |
npm @attest-ai/core@1.0.0-rc.3 |
published |
npm @attest-ai/vitest@1.0.0-rc.3 |
published |
| Website sync | dispatched (release-sync to attest-framework/attest-framework.github.io) |