Skip to content

fix(e2e): the differential-elision block reports its failure as an off-by-one #1220

Description

@vivek7405

Problem

The differential elision (#181) block in test/e2e/e2e.test.mjs fails intermittently and takes main red with it. It failed on main at 235d0fb0 and 4822e0d1 (2026-08-02) and on an in-flight PR run, then passed on the same branch with only unrelated test-file edits in between, which is what marks it a race rather than a real breakage.

Two assertions fail, both from the same cause:

✖ the mixed page renders identically on vs off
  AssertionError: post-hydration tag structure of <main> must match on vs off
✖ the interactive counter behaves identically on vs off
  AssertionError: counter increments identically on vs off (on=4, off=3)

on=4, off=3 is the tell. The seed is 3 and E2E: e2e.test.mjs:2030 separately asserts the ON side reaches 4, so the ON page incremented correctly and the OFF page did not move off its seed at all. The click was swallowed, not mis-counted.

Suspected mechanism, NOT yet confirmed. Both tests wait for hydration with a fixed await sleep(2500) (test/e2e/e2e.test.mjs:2001 and :2020) rather than a condition. The OFF server runs with WEBJS_ELIDE: '0' (:1973), so it ships strictly MORE modules than the ON build and hydrates strictly later. On a loaded CI runner the OFF side can still be un-hydrated when the sleep expires, which explains why OFF is always the failing side.

clickCounterButton (:2971-2977) then hides the miss:

const btn = counter?.querySelector(`button[aria-label="${lbl}"]`);
btn?.click();

The optional chaining makes a not-yet-upgraded button a silent no-op. Nothing throws, the counter stays at its seed, and the failure surfaces one assertion later as a confusing value mismatch instead of "the button was not there".

The tag-structure assertion fails the same way: a half-hydrated OFF page has a different element tree than a fully-hydrated ON page.

Confirm the mechanism before fixing it. The alternative reading is that this is a genuine intermittent hydration divergence in the framework, which would be a real progressive-enhancement bug and a much bigger deal than a test race. The two are distinguishable (see acceptance criteria), and the repo has already paid for skipping this step once (#1109 cost a PR plus three issues by implementing against an unverified stated mechanism).

Design / approach

Replace every timing assumption in this block with a deterministic readiness signal, the way this repo has de-flaked e2e twice before: #202 (De-flake the prefetch e2e via a webjs:prefetch cache signal) and #813 (de-flake lazy <webjs-frame> self-load e2e body capture). Both swapped a race for an observable condition rather than lengthening a sleep. Raising 2500 to some larger number is NOT a fix: it moves the failure rate without removing the race, and it slows an already slow job.

The rest of e2e.test.mjs already does this correctly, so the pattern to copy is in-file. page.waitForFunction(...) appears at :331, :343, :361, :368, :401, :415, :446, :458; the worked example at :360-373 waits for a post-hydration DOM condition, clicks, then waits for the resulting state rather than sleeping.

Two changes:

  1. Wait for hydration on BOTH pages before snapshotting or clicking. customElements.whenDefined('my-counter') plus the element being upgraded is the natural signal here (the tag is registered at examples/blog/components/counter.ts:41). Whatever signal is chosen must hold for the OFF build too, which is the slower one.
  2. Make the swallowed click loud. clickCounterButton should fail with a clear message when the button is absent rather than optional-chaining into a no-op, so a future race reports its actual cause instead of an off-by-one counter value.

Implementation notes (for the implementing agent)

Where to edit

  • test/e2e/e2e.test.mjs describe('differential elision (#181)') at L1969-2045. The fixed sleeps are at L2001 (sleep(2500), mixed-page test), L2020 (sleep(2500), counter test), L2026 (sleep(300), after the clicks) and L2036 (sleep(1500), static-route test).
  • test/e2e/e2e.test.mjs clickCounterButton() at L2971-2977 and getCounterValue() at L2955-2964. Both are shared helpers used by tests OUTSIDE this block, so changing their failure behaviour has blast radius. Grep every caller before editing.
  • observableMain() at L1988-1996 is the snapshot helper; it already normalises the wall-clock and whitespace, so it is not the problem, but read it to understand what "identical" means here.
  • The OFF fixture server is started at L1970-1975 with startBlog(offPort, { WEBJS_ELIDE: '0' }).

Landmines / gotchas

  • The e2e suite does not run in npm test. It is gated behind WEBJS_E2E=1: WEBJS_E2E=1 node --test test/e2e/e2e.test.mjs. A green npm test proves nothing here.
  • A flake cannot be verified fixed by one green run. Run the block repeatedly (20+ iterations) and, better, under artificial load, since the failure needs a slow runner to appear. A single pass is what let this reach main in the first place.
  • Do not lengthen the sleeps. See above.
  • The ON and OFF servers are two separate processes on two ports; the OFF one is torn down in after() at L1977-1983 with a 3s SIGTERM grace. If you add setup, keep teardown symmetrical or the port leaks into later runs.
  • my-counter is a LIGHT-DOM component, so its children are on the element itself, not in a shadow root (getCounterValue notes this at L2957). A readiness probe that looks for a shadow root will never fire.
  • This test is the live guard for the DANGEROUS direction of elision (L2014-2017): if elision ever wrongly dropped an interactive component's module, the ON counter would stop incrementing. Do not weaken that assertion while de-flaking it; the fix must make the test more discriminating, not less.

Invariants to respect

  • AGENTS.md invariant 11 (no em-dashes, no hyphen or semicolon as pause punctuation, WebJs brand casing) applies to any prose or comment added.
  • packages/ is plain .js with JSDoc, but this file is already .mjs test code, so no TypeScript concerns.
  • The progressive-enhancement guarantee this test protects is AGENTS.md's core claim ("with JS off, content reads, <a> navigates, <form> server actions submit"). If the investigation shows a REAL divergence rather than a test race, stop and re-scope: that is a framework bug, not a test fix.

Tests + docs surfaces

  • e2e only. No unit, browser, or Bun layer is involved, and no packages/*/src changes, so the doc gate and the Bun-parity gate do not apply.
  • No doc surface unless the investigation turns up a real framework bug.

Acceptance criteria

  • The mechanism is CONFIRMED before any fix lands, with evidence recorded on the PR: either instrument the OFF page to show it is un-hydrated when the sleep expires, or show the click landing on an absent button. Reproduce it, do not infer it
  • If the evidence instead shows a genuine on-vs-off hydration divergence, STOP and re-scope as a framework bug rather than de-flaking a test that is correctly reporting one
  • Every fixed sleep() in the differential elision (#181) block is replaced by a deterministic readiness condition
  • clickCounterButton fails loudly with a clear message when the button is absent, instead of silently no-opping
  • Every existing caller of clickCounterButton and getCounterValue outside this block still passes
  • The block survives 20+ consecutive runs, and a run under artificial CPU load, with no failure
  • A counterfactual proves the de-flaked test still catches the real defect: with elision wrongly dropping the counter's module, the ON assertion must still fail
  • WEBJS_E2E=1 node --test test/e2e/e2e.test.mjs passes end to end

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions