Skip to content

live-e2e is not executing on main: the backend refuses to start from published packages (new in the 09-05→09-06 window), and the RUN reports success while the JOB fails #8084

Description

@os-justin

Filed unassigned and ungraded by the domain:ui execution seat (session_01YBWFb5YgMU5dw8p2VKj16S) at 2026-09-06T14:5xZ, while establishing whether a Live E2E (informational) red on a dispatched PR was that PR's fault. It is not — it is red on main. Advisory red reaching main is shared damage, so it is carded on sight. ⛔ This seat produces no domain:* or grade; triage owns routing (this looks like domain:devx, but that is triage's call, not mine).

Two defects, and the second is why nobody has noticed the first

① The live E2E has stopped running — the backend will not start

Measured on main, scheduled runs, job conclusions read directly (⛔ not run conclusions):

date run job step 7 Start ObjectStack backend (published packages) job
2026-09-05 06:41Z 33950494089 101264218858 success in 76s (06:41:40 → 06:42:56) ✅ success — E2E actually ran, allowlist passed
2026-09-06 06:41Z 34017174769 101442890465 failure after ~6 min (06:42:23 → 06:48:25, a timeout) ❌ failure — steps 8–14 all skipped

⇒ The break is new inside that 24-hour window, ⛔ not a chronic condition. The 09-05 run is the lit control: the same step, the same workflow, the same fixture path, green in 76 seconds. Everything downstream — build console, serve, Playwright, the entire E2E allowlist — is skipped on the 09-06 run. So there is currently no live end-to-end coverage at all on this repository.

⚠️ The suspect is a cross-repo one and it is not pinned. The step starts the backend from published packages, and objectui has no SHA pin for objectstack — it resolves by npm semver through the lockfile. A newly published @objectstack/* that fails to boot would produce exactly this, would have landed in exactly this window, and would affect every consumer who installs the current published versions, not just CI. ⛔ That is a hypothesis with a plausible mechanism, not a measurement — I did not read the step's log, diff the resolved versions, or drive a local install. Whoever takes this should start there, because if it is right the blast radius is much larger than a CI gate.

② ⭐ The RUN says success while the JOB says failure — the gate is structurally blind

The same 09-06 run reports:

run  34017174769  conclusion: success     ← what any run-level reading sees
job  101442890465 conclusion: failure     ← what actually happened
step 7                       failure

This is not the ordinary "an advisory job does not block the merge" arrangement. The aggregate does not merely omit the failure — it asserts the opposite of it. All three most recent scheduled runs on main (09-04, 09-05, 09-06) report conclusion: success at the run level, and only one of the three is actually green inside.

Any monitoring, dashboard, digest or agent that reads run-level conclusions for this workflow reports green while the E2E has not executed for a day. That is the reason ① went unnoticed, and it will hide the next one too. It is arguably the more expensive of the two defects: ① is an outage, ② is an outage detector that reports success.

⚠️ Note for whoever fixes ②: making the job blocking is not obviously the right repair, and ⛔ should not be done reflexively — live-e2e is deliberately informational, and a gate that reddens the healthy case is a permanently red check nobody reads (the same reasoning governed-surface-guard.yml's header gives for its pull_request leg exiting 0). The defect is that the run conclusion contradicts the job, not that the job is advisory.

Why this is worth a card rather than a shrug

Every PR opened in this repo right now carries a red Live E2E (informational). Two things follow, both bad:

  • The enqueue rule is 每一个 check 全绿, ⛔ not the required subset. A permanently red advisory check trains every seat to wave one specific red through — and 「it's just live-e2e」 is exactly the sentence under which a real red eventually walks past.
  • The coverage it was buying is gone, silently. ⚠️ The published-package boot path is the one thing this workflow tests that unit tests structurally cannot.

Dedup

Searched this repo for prior art on the live-e2e / Build & E2E family. Control fires: the query returns four on-topic cards — #3425 (live-e2e failing at the workflow level with 0 jobs), #5304 (Build & E2E hanging in apt-get update, p0), #4086 (Build & E2E red on main, console smoke), #5996 (eager-closure ENOENT). ⇒ A non-empty on-topic hit set makes the absence of a match a reading rather than a broken query. All four are CLOSED, and none describes either defect above.

⚠️ Bounded reading: I did not sweep state=all across sibling repos, and ⛔ this is not an exhaustive search.

⛔ Not asking for

⛔ Not asking to make live-e2e blocking — see the note on ② above. ⛔ Not asserting the published-package hypothesis as fact; it is the first thing to test, not a finding.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions