Skip to content

[CI] ParallelMultiNodeSpec + LeaseMajority quarantined on hosted runners (Bun worker respawn dies after first test) #325

Description

@pathosDev

Summary

The worker-thread multi-node suites — ParallelMultiNodeSpec self-tests
(tests/unit/testkit/ParallelMultiNodeSpec.test.ts), parallel-pubsub
(tests/multi-node/parallel-pubsub.test.ts), and the in-process
LeaseMajority e2e (tests/multi-node/lease-majority.test.ts) — are
quarantined on GitHub's hosted runners via the env guard
ACTOR_TS_SKIP_FLAKY_MNS=1 (set in test.yml + multi-runtime.yml).
They still run locally and in Docker.

Root cause (diagnosed via temporary CI-only logging)

On GitHub's 2-vCPU hosted runners, Bun cannot respawn functional
worker threads after the first worker-thread test
:

  • The first worker-thread test converges fine (up=3, broker frames
    flow, ~3s).
  • Every subsequent test's workers spawn + complete the handshake
    (ready), then never run again: zero broker frames (recv=0), all
    control RPCs time out, no worker-side heartbeat after the first tick.

It is not reproducible on any other environment — passes on Bun
1.3.14 locally (Windows) and in Docker-Linux, even pinned to --cpus=2
and even under the near-full suite (199/204 files). It is not a
timeout issue (90s budget still times out), not coverage-related
(fails in multi-runtime without coverage too), not a worker leak
(awaiting terminate() + the close event didn't help), and not a
bun-version regression (1.3.14 passes locally).

LeaseMajority is collateral: the same resource starvation delays its
80ms lease-renewal timer past the TTL, the lease lapses, both partition
sides re-acquire it, and both survive (false split-brain).

What was tried (all failed on CI)

  1. awaitTimeoutMs 15s → 30s → 90s — still times out (non-convergence).
  2. await worker.terminate() — Bun's terminate is fire-and-forget void.
  3. terminateAndWait() (await the close event) — workers still don't
    respawn functionally.

Coverage

Excluding the worker-thread test harness from the coverage report
(bunfig.toml coveragePathIgnorePatterns) + lowering the floor 90 → 89
in test.yml (achievable hosted-CI line coverage is ~89.8% once the
worker suites are quarantined). LeaseMajority.ts itself stays 100%
(covered by unit tests).

CI coverage retained

The real-network integration workflow (5 real nodes over TCP, Docker)
remains the authoritative multi-node gate and is green.

To un-quarantine

Remove ACTOR_TS_SKIP_FLAKY_MNS from the two workflows + the describeMns
guards, drop the bunfig exclusion, and restore the 90 floor — once Bun
worker respawn works on hosted runners (worth re-checking on future Bun
releases, or with larger runners). Possibly file upstream with Bun.

Metadata

Metadata

Assignees

No one assigned

    Labels

    infrastructureCI / build / live-integration tests

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions