e2e-runbooks: an OpenSpec schema for behaviour-only end-to-end tests an AI agent can run #1256
Lukk17
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I built an OpenSpec custom schema, e2e-runbooks, for a problem ordinary tests handle badly: checking
features whose output legitimately varies (AI endpoints especially).
Ask an AI endpoint the same question twice and you can get two different answers that are both correct,
so an exact-match assertion is useless. e2e-runbooks tests assert on observable behaviour instead: the
HTTP status, the response content, the state left behind in Postgres / Qdrant / MinIO. Those checks pass
or fail on what the system did, not on the exact words it chose. So an agent (or a person) can run the
test against a live stack and get a real pass/fail, and every run is logged with start/end UTC, duration,
and a best-estimate token count.
Each capability gets an immutable spec, an immutable tasks-template, and one timestamped run record per
execution (proposal -> test-spec -> tasks-template -> run).
Repo: https://github.com/Lukk17/openspec-schemas
Schema: https://github.com/Lukk17/openspec-schemas/tree/master/openspec/schemas/e2e-runbooks
Feedback welcome.
Beta Was this translation helpful? Give feedback.
All reactions