A scenario that can't execute (connect failure, fixture missing, capability not advertised) should count as a FAILURE. The escape hatches are scenario filtering (--suite/--scenario) and the expected-failures baseline (#99) — not in-scenario skipping or silent passes.
Today every scenario hand-rolls this: most wrap the body in one big try/catch and emit their primary check ID as FAILURE on any error. That's fine for single-check scenarios but gets awkward with multiple checks at different severities (#200 discussion) — which check ID gets the connect failure is arbitrary, and the error message ends up under a label that has nothing to do with the actual problem.
Options:
- Emit all of the scenario's check IDs as FAILURE with the setup error message (consistent output shape)
- Emit a dedicated
<scenario-name>-setup check (semantically cleaner, but adds non-spec-mapped IDs)
- A
client-helper.ts utility that scenarios call on setup failure, so the convention is enforced in one place
The helper is probably the best and then we can evolve the behavior from there. Having all the check-id's emitted requires a scenario knowing all its checks, which might be useful but is a larger change. The -setup check is probably reasonable as a first cut for this.
A scenario that can't execute (connect failure, fixture missing, capability not advertised) should count as a FAILURE. The escape hatches are scenario filtering (
--suite/--scenario) and the expected-failures baseline (#99) — not in-scenario skipping or silent passes.Today every scenario hand-rolls this: most wrap the body in one big try/catch and emit their primary check ID as FAILURE on any error. That's fine for single-check scenarios but gets awkward with multiple checks at different severities (#200 discussion) — which check ID gets the connect failure is arbitrary, and the error message ends up under a label that has nothing to do with the actual problem.
Options:
<scenario-name>-setupcheck (semantically cleaner, but adds non-spec-mapped IDs)client-helper.tsutility that scenarios call on setup failure, so the convention is enforced in one placeThe helper is probably the best and then we can evolve the behavior from there. Having all the check-id's emitted requires a scenario knowing all its checks, which might be useful but is a larger change. The
-setupcheck is probably reasonable as a first cut for this.