Conformance

Conformance

demo/ holds a twelve-scenario conformance suite — one fixture set, one runner — covering the governed-execution lifecycle end to end. This suite is the "Validated integration" gate: no package should carry that label without a passing run, and today none does (see below).

Running it

ALGENTA_BASE_URL=http://localhost:8000 \
ALGENTA_API_KEY=your-key \
python -m demo.conformance.runner --json evidence.json

Run from a checkout of the repository, against your own live, self-hosted engine. There is deliberately no default base URL — a default pointing anywhere other than your own engine is how "self-hosted" quietly stops being true. The runner exits 2 without one.

Exit codes: 0 all exercisable scenarios passed · 1 at least one failed · 2 misconfigured.

The 12 scenarios, and which actually run

Measured against a current Algenta engine build over HTTP with a real Postgres — 9 exercisable, 3 not.

#	Scenario	Status
1	Successful read-only recommendation	exercisable
2	Execution denied by policy (409 named gate)	exercisable
3	Execution paused for approval	needs capability
4	Approval granted, execution resumed	exercisable
5	Approval rejected	exercisable
6	Duplicate execution prevented	exercisable
7	Expired / reused approval rejected	exercisable
8	Modified plan_hash rejected	exercisable
9	Upstream timeout as typed retryable error	needs infra
10	Execution receipt (versioned envelope)	exercisable
11	Audit export whose X-Content-SHA256 verifies	exercisable
12	Replay produces the same deterministic result	needs infra

Why three do not run

• 3 — Execution paused for approval. The engine refuses rather than pauses: executing an unapproved plan returns 409 plan_not_approved and the plan stays proposed. There is no suspended execution to resume — approval is a separate prior call (propose → approve → execute). That is a sound design, but it is not pause/resume, and asserting pause/resume would be asserting a design intention. The scenario becomes exercisable once the engine's Responses-protocol approval-required event ships.
• 9 — Upstream timeout. Needs a dependency that genuinely times out: a fault-injection proxy or the engine's compute path under real load. Patching a client to raise would test the mock, not the engine.
• 12 — Replay determinism. Determinism is a property of the engine's compute kernels, and this configuration runs with the accelerated compute runtime disabled (ALGENTA_SKIP_MOJO_RUNTIME=1). Comparing two runs of the execute path there would compare an artifact list to itself.

Scenarios 9 and 12 belong in a scheduled live tier against a fully booted engine stack.

A blocked scenario is not a pass. The runner reports 9 passed, 0 failed, 3 blocked and prints each reason. pytest.skip was avoided deliberately — "10 passed, 2 skipped" reads like success.

Proof the gate works

A suite that cannot fail is decoration. The same runner was pointed at two engines:

Engine	Result	Exit
main (with the audit export, receipt envelope, and named error codes)	9 passed, 0 failed	0
main before those three landed	2 passed, 7 failed	1

Both runs are committed under demo/conformance/evidence/. The passing run alone would prove nothing without the run that fails.

Fixtures

demo/fixtures/expected.json carries the semantic evidence every framework example must reproduce, including the named error code each gate returns. It was generated from a live run, not authored by hand, and records the engine build it was observed against — regenerate it rather than editing it.

What "validated integration" means

The suite currently exercises the engine directly over HTTP — the shared baseline. Per-framework adapters that drive the same fixture set through each package (LangChain, Pydantic AI, LiteLLM, Haystack, LlamaIndex, MAF, AI SDK) are the next milestone. Until those adapters pass, no package in this repository is labelled "Validated integration" — each package's own passing test suite runs against a real local stub server, which is necessary but not sufficient evidence.

Also still missing: cassette replay (ALGENTA_DEMO_REPLAY=1) for offline CI — cassettes must be recorded from live runs, never hand-written — and the nightly live tier that unblocks scenarios 9 and 12.

Full detail: demo/README.md.