Running the Suite

Running this suite means a person sitting in front of two windows: a running install of the application on one side, and the case library on the other. The library does not drive the application, does not call it, and does not know it exists. Everything here is about arranging those two windows well enough that a session produces evidence instead of impressions.

What you point it at

Any running install. The cases assume a system, not a specific deployment, so a stack you built from source, a seeded evaluation box, or a clean install you intend to keep are all valid targets — the hub's Try Forge First compares those options and Installation covers standing one up.

Which target you choose determines which part of the library is even runnable:

• A clean, empty install is the only thing Phase 0 can be run against. P0 starts at "nobody has ever signed in" and there is no way to get that state back except a fresh database. If you want the bootstrap cases, run them first or not at all.
• A seeded install skips you past P0–P3 into a system that already has users, customers, parts and jobs. Good for the cross-cutting suites and for P4/P5 exception work; useless for anything asserting first-run behaviour.
• The static demo SPA is not a valid target for most of the library. It has no server, so nothing it shows you is enforced — no validation beyond the form, no workflow gates, no posting, no permissions. A case whose pass_criteria is "the system blocked it" cannot pass or fail there. The list-view and layout suites are the only ones it can honestly answer.

Two properties of the library shape the session before you start. Phases are sequential and cumulative — P2 assumes P1's records exist, P4 assumes everything before it — while the cross-cutting suites are mostly independent and are written to run against the state Phase 5 leaves behind. And several branch choices made in P0, notably accounting mode and shipping mode, propagate forward: the choice is sticky for the whole session, and testing the other side of a branch means a second session, not a mid-run change of mind.

Reading the cases with no tooling at all

The lowest-cost way to use this repo is to read it. Every case is Markdown on GitHub with the YAML rendered in a fenced block, so docs/ is browsable end to end without cloning anything. A tester with a printout and a browser can execute any case in the library; the runner adds bookkeeping, not capability.

Start from a phase manifest rather than a content file. The manifest gives you the intended order, which cases are required, and the checkpoints — the content file is just where the cases happen to live.

The runner

test-bed/ is an Angular SPA with no backend. It compiles the Markdown and YAML into JSON at build time, serves it as static assets, and keeps every scrap of tester state in the browser's IndexedDB. There is no server to stand up, no account to create, nothing to configure.

git clone https://github.com/armoryworks/forge-test.git
cd forge-test/test-bed
npm install
npm start

You need Node and npm — match the Node major version pinned in .github/workflows/ci.yml, which is what CI builds and tests with.

Command	What it does
npm start	Compiles content, then serves the runner on the dev server
npm run build:content	Compiles docs/ into test-bed/public/assets/data/*.json and nothing else
npm run build	Production build; compiles content first
npm test	Vitest units

The content compile is wired to npm's prestart and prebuild hooks, which is the whole reason those two npm scripts exist as separate entries from ng.

Trap: ng serve directly skips the content compile. The prestart hook only fires for npm start. Run npx ng serve (as the repo's own CLAUDE.md suggests) and you get the runner serving whatever JSON was last built — quietly, with no warning that your edits to docs/ are not in it. Same for ng build versus npm run build. If a case you just wrote is not showing up, this is why.

Trap: the Docker build context is the repository root, not test-bed/. The image build runs the content compile, which reads ../docs, so the correct invocation is docker build -f test-bed/Dockerfile . from the repo root. Building from inside test-bed/ fails.

The compiled JSON lives under test-bed/public/assets/data/ and is committed to the repo, so a clone can serve the runner without a compile. runner-platform.md §2.1 states the opposite — that build outputs are gitignored and produced on the deployment host — and the code wins: they are tracked. The practical effect is that a change to docs/ is only visible to someone else after you commit the recompiled JSON alongside it. Two files in that directory, fixture.json and modules.json, are not compiler outputs at all; they are hand-maintained and the compiler will not regenerate them.

For hosting the runner beside a real install rather than on a laptop, forge-deploy carries it as a service under its test compose profile, bound and ported through its own environment variables and pulling the published image on a real deployment. That is how a team works through the library together — each tester still keeps their own results locally, but everyone reads the same build of the library.

What a case run needs

Starting a run asks for three things, in this order, and each one narrows what you will be shown:

1. Roles. The runner derives the available role list by taking the union of the roles field across every case — there is no separate roles file, so a new role appears in the picker the moment a case declares it. You are shown only cases whose roles intersect your selection.
2. Flows, optionally. Narrows further to the business journeys you care about — quote-to-cash, vendor-to-asset and the rest. Leaving it empty runs everything for the selected roles. Cases with no flows tag surface only when the filter is off.
3. Optional modules. Tick the ones your install actually has. Cases tagged with a module you did not enable are dropped entirely. This is where you say whether the built-in accounting module is in play; the full-GL entry in the catalogue is honest that no application implements it yet, so leaving it off is the normal answer.

From there the loop is: read the title, goal and preconditions; confirm the preconditions hold; work the steps one at a time, comparing what you see to the expected before moving on; then record pass, fail or blocked against the pass_criteria. Blocked is a real answer and is worth using — it means the case could not be executed because of a prior failure or an environment problem, and it is not the same signal as a fail.

Skip-ahead is first-class rather than an escape hatch, because real sessions are partial and interrupted. Jumping to a checkpoint shows you the state that should already exist and asks you to confirm it — either because you ran the prior cases, because you have a pre-seeded environment, or because you are knowingly accepting that failures caused by missing state are yours to interpret. The session records that you skipped, so the export says so.

A note on writing a failure: the note is the deliverable. "Login is broken" is not a bug report. What you clicked, what you saw, how long it took, what the screen said — that is.

Where the results live, and their limits

Everything is in IndexedDB in the browser you ran it in. Multiple sessions coexist and are listed with their progress; closing the browser and coming back resumes the incomplete one.

That has consequences you should plan around rather than discover:

• Results are per-browser and per-device. There is no sync and no server. The runner can export a session as JSON, but there is no import — moving a half-finished session to another machine is not possible today, only handing the export to someone else. Finish a session where you started it.
• Clearing site data destroys a session. Export before you clear anything, and before a long gap.
• There is no aggregation across testers. Two people running the same phase produce two independent exports; combining them is a manual job. Multi-tester dashboards are explicitly out of scope.
• The concurrency suite needs two sessions at once — two tabs, two devices, or two people — and the runner has no multi-session orchestration. The cases tell each side what to do and when; the timing is manual. The suite's manifest says so up front.

The session export is a single JSON file containing the session metadata, every case result and the failure notes. It is the artefact you send to a maintainer or attach to an issue. Two things the specification describes are not in the runner: screenshot attachments — runner-platform.md §3.2 has an Attachment type storing blobs in IndexedDB and the UX flow invites the tester to attach them, but nothing in the runner implements it, so a screenshot has to travel separately from the export — and filing GitHub issues directly from the runner, which is explicitly a planned opt-in enhancement rather than a shipped feature.

How this fits with the automated suites

This repo is the only one in the project that judges Forge against a standard defined outside Forge. That is its whole reason to exist, and it means it complements rather than overlaps the automated suites the other repos own.

Suite	Owner	What it asserts
Acceptance cases (this repo)	forge-test	That the product does what a manufacturer would expect of any ERP
Playwright scenarios, Vitest units	forge-ui	That the SPA behaves as the SPA is supposed to behave
xUnit, including architecture and capability-gate tests	forge-api	That the API behaves as the API is supposed to behave, and that the code obeys its own standards

The distinction is the direction of the arrow. An automated test is written by someone who knows the implementation and asserts against it; when the product changes deliberately, the test is updated to match. A case here is written by someone forbidden from knowing the implementation and asserts against the industry; when the product cannot satisfy it, the product changes. Both are necessary and neither substitutes for the other — a green Playwright run tells you nothing about whether the workflow it exercises is the workflow a shop needs.

The practical seams: forge-ui's Playwright config assumes a stack is already running rather than starting one itself, and its scenarios need the same kind of populated install these cases do, so the two often share a target box. The hub's Developer Setup covers which automated suites need what, and Contributing covers the pre-push gates.

What CI actually gates here

Be clear-eyed about this, because the badge is easy to over-read. CI on this repo installs, runs the content compile, runs a production build, and runs Vitest. The release workflows repeat exactly that before building and pushing the image.

The content compile is the real gate. It fails the build loudly, with file paths and case IDs, on a YAML parse error, a missing required field, an empty roles or steps list, a duplicate case ID anywhere in the library, or a story scene pointing at a case that does not exist. That catches the great majority of authoring mistakes.

What it does not catch is as important. runner-platform.md §6.4 specifies a validation set that the compiler has not grown into: prerequisite_cases references are not resolved, step branch_id values are not checked against the branches declared on the case, manifest sequence and checkpoints IDs are not verified, capability IDs are not intersected against the application's catalogue, and placeholders are not checked for a value in the default fixture. Suite manifests are the exception — the compiler does intersect their planned case IDs against reality, and reports the shortfall.

The Vitest suite, meanwhile, is a single smoke spec asserting that the root component constructs. Treat npm test passing as "the app boots", not as coverage.

Back to The Case Format for how a case is written, or Home for the repo's shape.