ci: run the scenario corpus in ladder-tests, and let corpus changes reach it by Yaraslaut · Pull Request #473 · LASTRADA-Software/morph

Yaraslaut · 2026-09-06T21:10:06Z

Fixes #462.

Nothing in CI ran scripts/scenario/run_scenarios.py. grep -rn run_scenarios .github/workflows/ returned exactly one hit and it was a comment saying so.
The corpus could be refused wholesale by a real server with every workflow
green, and was — 16 of 16 ledger scenarios refused for five days and through a
merge (#460), while drift-guard.yml's scenario-coverage job reported
ledger actions 18/18 dispatched, workflows 16/16 throughout. That job parses
the corpus; only running it can see a refusal.

What this does

1. ladder-tests runs the corpus, after its ctest step.

That job is where it belongs, and drift-guard.yml's own deferral comment says
why: running the corpus needs the ladder_<rung>_server binaries, a build that
workflow deliberately does not have. ladder-tests already builds every one of
them unconditionally (cmake/morph_add_rung.cmake:373-377), so the cost the
comment cited as the blocker is already paid there. The run is not moved into
drift-guard.yml, which stays fast and dependency-free.

--build-dir build/gcc-debug names the build so no stale binary from another
tree can be selected. --rung is passed once per corpus directory found on
disk rather than left to the driver's default, so the set that runs is
demonstrably the set that exists.

A missing binary is not tolerated. run_scenarios.py exits 2 when a server
is absent and that is left exactly as it is.

2. scripts/scenario/ joins ci-path-regex, with the pin the issue asked
for.

Before this change a pull request touching only the corpus matched nothing and
skipped ladder-tests entirely — so the gate would have been green on exactly
the pull requests most able to break the corpus. check_rung_filters.sh now
probes two corpus paths against the generated regex (behavioural, like its
existing rung checks, so a pattern rewritten into one that matches nothing
fails rather than passing), and test_check_rung_filters.sh reintroduces the
omission and requires the gate to catch it.

3. A pre-flight check refuses a corpus this job could not have run.

Two ways a corpus can drop out of the run without anything saying so: a
scenarios/<name>/ directory with no RungSpec (the driver's default set is
the intersection of the two, so it is simply not in it), and a rung this job
builds no server for (loud — exit 2 — but only after a full Qt + ladder
build). Both are this issue's own failure one level down, so both are checked
on a bare checkout, in seconds, before anything is installed or compiled.

4. drift-guard.yml's comment saying this run "is its own change" now
points at the change rather than deferring it. No behavioural change there.

The bank interaction (#470), decided

#470 adds a 22-scenario bank corpus and a ladder_bank_server. Bank is
deliberately not in examples/rungs.txt — unnumbered prior art, its server
built by a local add_executable(), not morph_add_rung(). ladder-tests
builds servers from rungs.txt, so it will not build bank's; the driver's
default rung set is every rung with a scenarios/<rung>/ directory, so it
will include bank. A whole-corpus run in ladder-tests would hit a missing
ladder_bank_server and exit 2 — after a twenty-minute build.

Chosen: run every corpus that exists on disk, and fail loudly up front on
any corpus this job does not build a server for. Not "run only what this job
builds" — that is the silent skip the issue exists to end, and it would have
quietly run five of six corpora the day #470 lands. Not "an explicit list in
the workflow" either, since a list is a second copy that can fall behind; the
list is derived from disk and the pre-flight is the check that it matches what
the job can actually run.

#470 touches both scripts/scenario/** and examples/rungs.txt, so once this
lands it will trigger ladder-tests on its own pull request, not only
after merging. The job goes red in its first thirty seconds — before any
install or compile — with:

::error::scripts/scenario/scenarios/bank/ is a corpus for 'bank', which is not
listed in examples/rungs.txt -- this job configures no ladder_bank_server for
it. Either list it there, or build its server in this job and extend this check
to expect it.

What will need doing when #470 lands (a follow-up on top of it, not part of
this PR — bank does not exist on origin/master):

configure bank's server in ladder-tests — -DMORPH_BUILD_BANK_EXAMPLE=ON
on that job's Configure step. Its apt step already installs
libsqlite3-dev, libyaml-cpp-dev, libzip-dev, unixodbc-dev and
libsqliteodbc for the ladder's own Lightweight fetch, and its comments
already cite examples/bank/CMakeLists.txt as the reason. bank: give it a server, and a scenario corpus over all 41 actions #470 gates
add_executable(ladder_bank_server ...) on MORPH_BUILD_QT, which this job
already sets, so the option is the only missing piece;
teach the pre-flight's second branch that bank's server comes from
examples/bank/CMakeLists.txt rather than from examples/rungs.txt — one
extra arm, not a relaxation: the requirement stays "name the thing that
builds this corpus's server, or fail".

That is deliberately a red build rather than a silent skip. Either the corpus
runs or CI says which one does not.

Evidence

Both runs are of ladder-tests → Run the scenario corpus against the built
servers, on the same tree, differing only in one scenario file.

Green — this branch, corpus as it stands

Application ladder, run 34060226868 — success

corpus rungs: --rung bookmarks --rung kanban --rung ledger --rung pastebin --rung polls
bookmarks: 14 file(s) against ws://127.0.0.1:37077
  ok   an-edit-replaces-every-field.scenario
  ...
kanban: 21 file(s) against ws://127.0.0.1:37235
  ...
ledger: 16 file(s) against ws://127.0.0.1:45071
  ok   a-rule-and-the-category-it-sets.scenario
  ok   a-second-editor-on-the-same-book.scenario
  ok   a-signed-in-session-is-what-authorises-work.scenario
  ok   accounts-of-every-kind.scenario
  ok   an-unbalanced-entry-changes-nothing.scenario
  ok   bootstrap-a-book-over-the-wire.scenario
  ok   budget-limit-and-spend-report.scenario
  ok   categorise-an-account.scenario
  ok   exactly-once-store-transaction.scenario
  ok   import-a-statement-chunk.scenario
  ok   open-account-transact-report-close.scenario
  ok   store-list-and-undo-an-entry.scenario
  ok   submit-a-report-and-poll-it.scenario
  ok   two-books-are-isolated.scenario
  ok   update-rule-version-conflict.scenario
  ok   zero-sum-holds-per-currency.scenario
pastebin: 12 file(s) against ws://127.0.0.1:43349
  ok   broken-on-purpose.scenario (failed, as it is meant to)
  ...
polls: 10 file(s) against ws://127.0.0.1:39347
  ...

every scenario passed in: bookmarks, kanban, ledger, pastebin, polls

73 files, 73 expected verdicts, against five real servers started by the step.

The pre-flight, from the same job, four steps earlier:

ok: bookmarks -- corpus, RungSpec and ladder_bookmarks_server all accounted for
ok: kanban -- corpus, RungSpec and ladder_kanban_server all accounted for
ok: ledger -- corpus, RungSpec and ladder_ledger_server all accounted for
ok: pastebin -- corpus, RungSpec and ladder_pastebin_server all accounted for
ok: polls -- corpus, RungSpec and ladder_polls_server all accounted for

Red — #474, one scenario file reverted to the integer enum spelling

Application ladder, run 34060255730 — failure, and the only failing step in the job

#474 is based on this branch and its entire diff is
scripts/scenario/scenarios/ledger/categorise-an-account.scenario, restored to
its pre-8a67ffe7 form — the spelling #460 measured as refused.

ledger: 16 file(s) against ws://127.0.0.1:40087
  ok   a-rule-and-the-category-it-sets.scenario
  ...
  ok   step 4 (line 27) [books] session principal=$who token=$token
FAIL step 5 (line 30): do OpenAccount ledgerId=1 name="Categorised cash" kind=0 currency=1
  expected: @kind == ok
  actual:   @kind == err
  client:   books (modelId=7986178768489167878)
  reply:    err message="1:48: expected_quote\x0a   {"ledgerId":1,"name":"Categorised cash","kind":0,"currency":1}\x0a                                                  ^"
.../scenarios/ledger/categorise-an-account.scenario: 29 steps, 5 assertions, 1 failure
  FAIL categorise-an-account.scenario
...
FAILED: ledger
##[error]Process completed with exit code 1.

expected_quote from a real ladder_ledger_server — the exact refusal that
sat unnoticed for five days and one merge. Every other step of that job
succeeded, including Build and ctest, so the red is the corpus and nothing
else. Restoring the file is what makes it green: that restored file is what the
green run above ran.

The path filter, demonstrated

#474's diff is one file, under scripts/scenario/, and nothing else
(gh pr view 474 --json files → scripts/scenario/scenarios/ledger/categorise-an-account.scenario (+6/-6)).
On master today that matches no alternative in ci-path-regex, so
ladder-tests would not have run at all — the job that just caught this would
have been skipped on the change that broke it. With this PR's entry it runs;
from that job's Determine whether the ladder needs to run step:

ladder path filter: ^(examples/pastebin/|...|scripts/ladder_rungs\.sh$|scripts/scenario/|examples/LADDER\.md|...)

and the job proceeded through every subsequent step. That the red run happened
at all is the path-filter proof: a corpus-only change reached the gate.

Locally, the same three paths against the old and new pattern:

scripts/scenario/scenarios/ledger/two-books-are-isolated.scenario
  BEFORE: run=false  <- ladder-tests skipped
  AFTER : run=true   <- ladder-tests runs
scripts/scenario/run_scenarios.py
  BEFORE: run=false  <- ladder-tests skipped
  AFTER : run=true   <- ladder-tests runs
scripts/scenario/README.md
  BEFORE: run=false  <- ladder-tests skipped
  AFTER : run=true   <- ladder-tests runs

Cost

Measured from the green run's own step timings (morph-docker-3, warm caches):

step	duration
Check every scenario corpus has a server this job builds	< 1 s (21:49:21 → 21:49:21)
Run the scenario corpus against the built servers	23 s (22:01:53 → 22:02:16)
`ladder-tests` total	13 m 24 s (21:48:58 → 22:02:22)

~24 s added to a 13 m 24 s job — about 3%. On the GitHub-hosted path,
where the same job last took 40 m 02 s with a cold cache
(run 33904711229),
the same 24 s is about 1%. It is a fixed cost: five servers started, 73
scenario files driven, no compilation.

The red run's failing step took 27 s, so the cost is the same whichever way it
goes — the gate does not get slow only when it has something to say.

One second-order cost, stated plainly. ladder-sanitizers shares this
filter by design (scripts/ladder_rungs.sh says both steps must stay
identical, and check_rung_filters.sh enforces that both derive it), so a
corpus-only pull request now also triggers the ASan ladder build — the largest
single job in the workflow. Giving the two jobs different patterns to avoid
that would reintroduce exactly the divergence that file exists to prevent, so
this PR does not do it.

Known follow-up, deliberately not done here

scripts/scenario/README.md's "In CI" section still says "What CI does not
do is run the corpus", which this PR makes false. #470 rewrites that exact
paragraph (the five → the six ladder_<name>_server binaries), so editing it
here would be a guaranteed conflict on a file that PR owns. It is left to
whichever of the two lands second.

Verified locally

$ bash scripts/check_rung_filters.sh
...
All 39 rung-filter checks passed.            (was 37 before; +2 corpus probes)

$ bash scripts/test_check_rung_filters.sh
ok: the unmodified tree passes
ok: caught: a rung missing from wasm-ladder.yml's on.push.paths
ok: caught: a rung missing from wasm-ladder.yml's on.pull_request.paths
ok: caught: a rung missing from codecov.yml's components
ok: caught: ci.yml's filter hand-written again instead of generated
ok: caught: the generated ci.yml regex silently omitting a rung
ok: caught: the generated ci.yml regex silently omitting scripts/scenario/
ok: caught: a declared rung missing from examples/rungs.txt
ok: caught: wasm-ladder.yml's path lists becoming unreadable to the parser
ok: caught: examples/rungs.txt listing no rungs at all

scripts/check_rung_filters.sh detects every drift it claims to.

$ python3 scripts/scenario/scenario_coverage.py     -> exit 0
$ python3 scripts/scenario/test_morph_scenario.py   -> Ran 132 tests ... OK
$ python3 -c "import yaml; ..."                     -> ci.yml, drift-guard.yml, wasm-ladder.yml all parse

bash scripts/ladder_rungs.sh ci-path-regex, before and after — one
alternative added, nothing else moved:

1c1
< ...|\.github/workflows/ci\.yml$|scripts/ladder_rungs\.sh$|examples/LADDER\.md|...
---
> ...|\.github/workflows/ci\.yml$|scripts/ladder_rungs\.sh$|scripts/scenario/|examples/LADDER\.md|...

The pre-flight check's four failure modes, exercised against scratch copies of
the tree (the exact run: script extracted from ci.yml):

### case 1: a corpus directory with no RungSpec in run_scenarios.py
::error::scripts/scenario/scenarios/bank/ has no RungSpec in scripts/scenario/run_scenarios.py, ...
exit=1

### case 2: a bank corpus WITH a RungSpec, bank absent from examples/rungs.txt (what #470 will land)
::error::scripts/scenario/scenarios/bank/ is a corpus for 'bank', which is not listed in examples/rungs.txt ...
exit=1

### case 3: a corpus for a listed rung that declares no src/server (lims)
::error::examples/lims/src/server/ does not exist, so cmake/morph_add_rung.cmake declares no ladder_lims_server ...
exit=1

### case 4: no corpus directories at all (a vacuous gate)
::error::scripts/scenario/scenarios/ holds no corpus directories -- the scenario step at the end of this job would report success having run nothing
exit=1

And the run step, on a tree with no build (exit 2, not a skip):

corpus rungs: --rung bookmarks --rung kanban --rung ledger --rung pastebin --rung polls
run_scenarios: build/gcc-debug/examples/bookmarks/ladder_bookmarks_server does not exist -- is bookmarks's server built?
exit=2

🤖 Generated with Claude Code

https://claude.ai/code/session_01C6uB9zxdSG8qp3VNAAyFvF

…each it Nothing in CI executed scripts/scenario/run_scenarios.py. The corpus could therefore be refused wholesale by a real server with every workflow green, and was: sixteen of sixteen ledger scenarios refused for five days and through a merge (morph#460), while drift-guard.yml's scenario-coverage job reported "ledger actions 18/18 dispatched, workflows 16/16" the whole time. That job parses the corpus; only running it can see a refusal. Three changes. `ladder-tests` now runs the corpus after its ctest step. It is the right job: drift-guard.yml deferred this because running the corpus needs the ladder_<rung>_server binaries and that workflow deliberately has no build, but `ladder-tests` already builds every one of them unconditionally (cmake/morph_add_rung.cmake). --build-dir names the build so no stale binary from another tree can be picked up, and --rung is passed once per corpus directory found on disk rather than left to the driver's default, so the set that runs is demonstrably the set that exists. A missing binary is not tolerated: run_scenarios.py exits 2, which is correct and is not softened. `scripts/scenario/` joins ci-path-regex. Without it a pull request touching only the corpus matched nothing and skipped `ladder-tests` altogether -- so the one gate that can catch a broken scenario would have been absent from exactly the changes most able to break one, which is morph#179's defect one directory over. check_rung_filters.sh now probes two corpus paths against the generated regex, behaviourally, and its self-test reintroduces the omission and requires the gate to catch it. A pre-flight check refuses a corpus this job could not have run. A scenarios/<name>/ directory with no RungSpec drops out of the driver's default set in silence, and a corpus for a rung this job configures no server for is only reported after a full Qt + ladder build. Both are this issue's own failure one level down, so both are checked on a bare checkout in seconds. This is what resolves the bank interaction (morph#470): examples/bank is deliberately not in examples/rungs.txt and its server comes from a local add_executable(), so when its corpus lands the check fails by name rather than letting five of six directories run and calling that success. drift-guard.yml's comment saying this run "is its own change" now points at the change instead of deferring it. Fixes #462 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C6uB9zxdSG8qp3VNAAyFvF

codecov · 2026-09-06T21:49:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…expect it (#475) Master went red on the merge of #470 (bank's server and 22-scenario corpus) into #473's pre-flight, exactly as #473's own comment said it would: ::error::scripts/scenario/scenarios/bank/ is a corpus for 'bank', which is not listed in examples/rungs.txt -- this job configures no ladder_bank_server for it. Either list it there, or build its server in this job and extend this check to expect it. That is the gate working. A corpus with no server in this job would otherwise have run five of six directories and reported green, which is morph#462's defect one level down. So this takes the second branch the error names rather than weakening the check or adding bank to examples/rungs.txt -- the latter would enrol bank in wasm-ladder.yml's build loop, coverage.sh and codecov.yml's per-rung components, which is a scope decision that rungs.txt and examples/LADDER.md have already settled the other way. ladder-tests now configures -DMORPH_BUILD_BANK_EXAMPLE=ON. That option alone guards add_subdirectory(examples/bank) in the root CMakeLists.txt; examples/bank/CMakeLists.txt then declares ladder_bank_server inside its own if(MORPH_BUILD_QT), which this job already sets. Nothing new to install: the apt step already names Lightweight's libsqlite3-dev, libyaml-cpp-dev, libzip-dev, unixodbc-dev and libsqliteodbc (MORPH_BUILD_LADDER=ON fetches the same pinned ORM through examples/common) and libgl1-mesa-dev, and the server links only Qt6::Core and Qt6::WebSockets, both already installed. The Build step names the target after the all-target build -- a no-op when the flag is present, and "no rule to make target" when someone drops it, instead of a silent absence surfacing as the scenario step's exit 2 a Qt build later. The pre-flight gains a third arm, not a relaxation: a corpus that is neither a listed rung nor served by an add_executable(ladder_<name>_server) in examples/<name>/CMakeLists.txt still fails. The arm is keyed on that declaration rather than on the literal name "bank", so deleting the target makes the check fire again -- verified by commenting the add_executable() out and watching bank go red -- and a scratch scenarios/nonesuch/ still fails. scripts/scenario/README.md's "In CI" section said CI does not run the corpus because no workflow builds the servers. That has been false since #473; rewritten to describe the pre-flight, the whole-corpus run, and the scripts/scenario/ entry in the job's path filter. Verified locally with the job's own flags: ladder_bank_server builds, all six servers exist, `run_scenarios.py` over bank+bookmarks+kanban+ledger+pastebin+ polls reports "every scenario passed", ctest -L ladder -LE stress is 1026/1026, scripts/check_rung_filters.sh is 39/39, and `ladder_rungs.sh ci-path-regex` is byte-identical (neither of its inputs is touched). Claude-Session: https://claude.ai/code/session_01C6uB9zxdSG8qp3VNAAyFvF Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Yaraslaut mentioned this pull request Sep 6, 2026

DEMO ONLY, DO NOT MERGE: negative control for #473 (scenario gate must go red) #474

Closed

Yaraslaut merged commit 5978835 into master Sep 7, 2026
42 checks passed

Yaraslaut mentioned this pull request Sep 7, 2026

master is red: build ladder_bank_server in ladder-tests, and let the pre-flight expect it #475

Merged

Navigation Menu

Uh oh!

ci: run the scenario corpus in ladder-tests, and let corpus changes reach it - #473

ci: run the scenario corpus in ladder-tests, and let corpus changes reach it#473
Yaraslaut merged 1 commit into
masterfrom
feat/462-scenario-ci-gate

Yaraslaut commented Sep 6, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Yaraslaut commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

The bank interaction (#470), decided

Evidence

Green — this branch, corpus as it stands

Red — #474, one scenario file reverted to the integer enum spelling

The path filter, demonstrated

Cost

Known follow-up, deliberately not done here

Verified locally

Uh oh!

codecov Bot commented Sep 6, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Yaraslaut commented Sep 6, 2026 •

edited

Loading