compat-matrix: engine-job failures are invisible (continue-on-error) — harden after burning down pre-existing leg debt

[bpamiri]

Problem

The compat-matrix engine jobs run with continue-on-error, so the workflow run concludes "success" even when every engine job fails. Two consequences observed on 2026-07-06:

1. A guard spec (ConfigRoutesStaleDocUrlSpec, from fix(cli): repoint 15 dead v4-0-0-snapshot and 3.1.0 guide URLs at live v4 pages #3281) failed every engine × database leg uniformly for a full day before anyone noticed — the run banner was green (fixed in fix(build): retire dead 3.1.0 guide URL in docker engine settings templates #3293).
2. UpdateAllIncludeJoinSpec has been red on lucee7+mysql since at least 2026-06-28 (a real data-affecting bug, now updateAll with multiple includes silently updates 0 rows on MySQL #3294) — invisible for the same reason. The June-28 baseline additionally carries per-engine debt (Adobe: typedColumnDefaultsSpec, contentSpec, RouteTesterHardeningSpec, transaction specs; Lucee 6: RouteTesterHardeningSpec; BoxLang: miscellaneousSpec, AdapterIdentityTemplateSpec, testEvaluateExpressionSpec, DatabaseAdapterSpec) that nobody triaged because nothing ever turned red at the run level.

Same lesson as the box-install check (#3281): a check that can't fail visibly is noise, and real regressions hide behind it.

Proposal

• Keep per-DB soft-fail for the known-flaky infrastructure (SOFT_FAIL_DBS=oracle already exists) but make a hard-DB leg failure fail the engine job, and an engine-job failure fail the run.
• To get there without a permanently red banner, first burn down (or explicitly soft-fail with a linked issue) the pre-existing per-engine failures listed above — each should be a tracked issue, not silent debt.
• Optional: have the Test Matrix Summary job post a compact per-engine × per-DB verdict table into the run summary so a glance shows exactly which leg regressed.

Evidence

• Run 28809821005 (2026-07-06 dispatch): conclusion success, all 5 engine jobs failure.
• Run 28815315339 (post-fix(build): retire dead 3.1.0 guide URL in docker engine settings templates #3293): same shape; per-leg artifacts show the real state (lucee7 clean except mysql updateAll with multiple includes silently updates 0 rows on MySQL #3294).

[wheels-bot]

Wheels Bot — Triage

Classified as other (general discussion / support / docs request).
No further bot action will be taken on this issue.