Repository navigation
v1.0.1
Hotfix themed "Red stays red". On 1.0.0 a failing assertion could be reported as green: an engine drift line, a request timeout or concurrent evaluations could hand one test another test's verdict, a pytest-xdist run wrote an empty report that the policy gate passed, and a regression policy without its baseline passed silently. 1.0.1 routes every engine response by request id in both SDKs, writes complete reports under parallel runners, and makes attest policy evaluate fail closed when an input is missing. Several fixes turn a silent pass into a failure; they are marked Behavior change below.
Fixed
- Behavior change: Python and TypeScript SDKs route every engine response by request id. A test that previously passed on another request's response (after an engine
drift_alertline, after anATTEST_ENGINE_TIMEOUT, or underit.concurrent) now reports its own, possibly failing, verdict. Lines without an id are dropped and logged. - Behavior change:
evaluate_batchraisesProtocolDesyncError(Python) or rejects with it (TypeScript) when the response's assertion ids differ from the ids sent. - Behavior change: a request timeout restarts the engine; the timed-out request raises (rejects with)
EngineTimeoutErrorand every other pending request fails, so a late reply never reaches another test. The new engine gets the remainingATTEST_MAX_JUDGE_CALLS/ATTEST_BUDGET_USDsession budget: the configured limit minus the usage already returned, minus a hold for each lost request that can call a model (its per-batchbudgetif it sent one, otherwise everything left). Spend the SDK cannot observe (simulated-user messages, engine error replies, replies dropped after cancellation) leaves the restarted engine no budget, so a restart can never refill the session budget. - Behavior change:
pytest -n N --attest-report=<path>writes one report containing every worker's tests and assertions; before, the report was empty andattest policy evaluatepassed. If a worker ends without sending its report, the run exits with an internal-error status and no report is written (an older file at the path is removed). - Behavior change:
attest policy evaluateexits 3 instead of passing when the policy setsblock_on_regressionorregressionwithout--baseline, ormin_mutation_scorewithout--mutation; the message names the missing input. - Behavior change:
attest policy evaluateexits 3 when a report has no tests while itstests_collectedis greater than 0 (a parallel run that lost its results). - The engine reports drift inside
evaluate_batchresults (results[].drift) instead of writing an id-lessdrift_alertline to stdout. - Python: responses over 64 KiB (for example 400+ assertions) no longer crash the session; the SDK's stream limit matches the engine's 10 MiB request line limit.
- TypeScript:
EngineManager.start()rejects when the engine exits or closes before it is ready, with the engine's stderr tail (also when the engine dies before the first write, which used to surface as a barewrite EPIPE), andinitializetimes out instead of hanging. Write errors after startup carry the stderr tail. - TypeScript: request timers are cleared, so a standalone script exits when its work is done instead of 30 s later.
- Python:
EngineManager.send_requeststays public and id-correct (it goes through an attachedAttestClient, otherwise the id-matched handshake), andEngineManager.stop()shuts down cleanly when anAttestClientreader is attached. A secondAttestClienton one engine raisesRuntimeErrorinstead of stealing the first one's responses. - Engine history and caches share one SQLite handle with WAL and
busy_timeout, so parallel engine processes no longer lose history rows toSQLITE_BUSY;attest baseline pinandattest calibrateopen the database the same way and wait instead of failing. - History pruning runs in the background in short batched deletes (age through a new
created_atindex, the row cap per recently recorded assertion id) instead of a whole-table delete insideevaluate_batch, removing the stall near 2,800 evaluations per assertion. - The
attest-aiPyPI quick start is a runnable pytest test that evaluates its chains and fails on a wrong assertion; it used an undefinedtraceand built chains it never evaluated. - Guide examples in framework-adapters, multi-agent and simulation evaluate their chains and use existing methods (
required_tools()replaces the nonexistenttool_called()).
Added
- report-v2 files written by the pytest plugin and the Vitest reporter carry a top-level
tests_collected(tests collected after deselection);report-v2.schema.jsondeclares it.
Changed
- The protocol spec states that the engine writes only responses to stdout and that SDKs route by request id;
docs/guides/configuration.mddocumentsATTEST_ENGINE_TIMEOUT(seconds in Python, milliseconds in TypeScript) and what a timeout does to the session budget, and notes that budgets apply per pytest-xdist worker. - Install instructions use
attest-ai>=1.0.1and@attest-ai/core@>=1.0.1/@attest-ai/vitest@>=1.0.1;@attest-ai/vitestdeclares the peer@attest-ai/core >=1.0.1. The annotations guide pinspr-annotate@v1.0.1. - The PyPI classifier is
Development Status :: 4 - Beta(was5 - Production/Stable) until 1.1.0 completes its re-evaluation. - Engine Go dependencies:
go.opentelemetry.io/proto/otlp1.11.1,golang.org/x/text0.43.0,modernc.org/sqlite1.61.1,google.golang.org/protobuf1.36.12.
Removed
- The
drift_alertstdout notification (types.DriftAlertNotification).
Published components
| Component | Status |
|---|---|
| Engine binaries and checksums (this release) | published |
PyPI attest-ai==1.0.1 |
published |
npm @attest-ai/core@1.0.1 |
published |
npm @attest-ai/vitest@1.0.1 |
published |
| Website sync | dispatched (release-sync to attest-framework/attest-framework.github.io) |