test(ci): log unhandledRejection in backend test bootstrap to debug ~22% silent CI flake#7663
Conversation
…bootstrap Backend tests on develop have a ~22% silent failure rate (mostly Windows, sometimes Linux) where mocha exits with code 1 mid-suite, producing no test failure marker, no error, and no Mocha summary. Different exit points each run. Root cause discovery is blocked by src/tests/backend/common.ts:33, which rethrows unhandled Promise rejections as uncaught exceptions but never logs the reason first. When the rethrow happens between specs, mocha exits with code 1 and the original rejection is lost - especially on Windows, where stderr is not always flushed before abrupt exit. This patch is purely diagnostic: it writes the reason (or stack) to stderr before rethrowing, and adds a matching uncaughtException handler for the same purpose. Behavior on success is unchanged. The next CI failure will surface what is actually rejecting (DirtyDB write? plugin lifecycle? socket cleanup?), so we can fix the real cause. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ⓘ You've reached your Qodo monthly free-tier limit. Reviews pause until next month — upgrade your plan to continue now, or link your paid account if you already have one. |
Review Summary by QodoLog unhandledRejection and uncaughtException in backend test bootstrap
WalkthroughsDescription• Add diagnostic logging for unhandled Promise rejections in backend tests • Log rejection reason/stack to stderr before rethrowing to prevent silent failures • Add uncaughtException handler to surface uncaught errors with stack traces • Improve CI debugging for ~22% silent test failure rate on Windows/Linux runners Diagramflowchart LR
A["Unhandled Promise Rejection"] -->|"log reason to stderr"| B["process.stderr.write"]
B -->|"then rethrow"| C["throw reason"]
D["Uncaught Exception"] -->|"log error to stderr"| E["process.stderr.write"]
E -->|"then exit"| F["process.exit 1"]
C -->|"prevents silent mocha exit"| G["CI logs show rejection reason"]
F -->|"prevents silent mocha exit"| G
File Changes1. src/tests/backend/common.ts
|
Code Review by Qodo
1. No regression test for logging
|
…p to #7663) (#7665) * test(ci): stronger diagnostics for silent backend-test exit PR #7663 added unhandledRejection / uncaughtException handlers in common.ts. The next failure after merge (run 25279692065 - Windows without plugins, Node 24) showed mocha exiting with code 1 mid-suite 261ms after the last passing test, with NEITHER handler firing. So something more drastic is killing the process - SIGKILL, OOM, fatal native error - or mocha itself called process.exit before the JS handlers in common.ts could run. Two issues with the previous attempt: 1. Handlers in common.ts only register when a spec imports common.ts. Only 27 of 47 specs do. If a non-common spec triggers the death, handlers may never have been registered. 2. process.stderr.write is asynchronous on Windows when stderr is piped (which it is under GitHub Actions). On a hard kill the buffered line never reaches the runner log. This patch: - Moves diagnostic handlers to a dedicated tests/backend/diagnostics.ts loaded via mocha --require, so they register at startup before any spec runs. - Uses fs.writeSync(2, ...) for synchronous stderr writes that the kernel completes before returning - the line lands in the log even if the process is killed milliseconds later. - Adds beforeExit / exit / signal handlers so we can discriminate the exit mechanism: clean drain vs process.exit vs SIGKILL vs signal. - Tracks last-seen test via mocha root afterEach hook so the death point is visible in the log. The next CI failure should print enough context to identify the cause, after which we can fix the real bug and drop this file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(diagnostics): exit(1) on uncaughtException so fatal errors fail fast Qodo flagged on PR #7665: the uncaughtException handler in tests/backend/diagnostics.ts only logged and returned. Once a handler is registered, Node no longer exits on its own. Specs that don't import tests/backend/common.ts (20 of 47) have only this handler — so a fatal error would have been swallowed and tests would limp along instead of failing fast. Mirror common.ts and call process.exit(1) after logging. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Backend tests on
developfail silently ~22% of the time (13 of the last 60 runs). Mocha exits with code 1 mid-suite, producing no test-failure marker, no error, and no Mocha summary. Different exit points each run, mostly on Windows runners but occasionally Linux too.Root cause discovery is blocked by
src/tests/backend/common.ts:33, which rethrows unhandled Promise rejections as uncaught exceptions but never logs the reason first:When the rethrow fires between specs (e.g. a delayed DirtyDB write rejecting after a test completed), mocha exits with code 1 and the original rejection is lost — especially on Windows, where stderr is not always flushed before abrupt exit. CI logs end with
ELIFECYCLE Test failedand a clean ✔ from the previous test, with no clue what rejected.What this PR does
Pure diagnostic patch — no behavior change on success:
unhandledRejectionhandler now writes the reason (or stack) to stderr before rethrowing.uncaughtExceptionhandler so non-Promise crashes also leave a trace.The next CI failure will surface what is actually rejecting (DirtyDB write? plugin lifecycle? socket cleanup?), at which point we can fix the real cause.
Why this isn't using the test logger
The 'test' logger goes through log4js, which uses console layouts on top of process.stdout/stderr — same flushing problem.
process.stderr.writeis the most deterministic synchronous-ish path, and matters specifically on Windows.Test plan
pnpm exec tsc --noEmit— type-checks cleanly (no new errors in `src/tree)pnpm exec mocha --import=tsx tests/backend/specs/anonymizeIp.ts— 15/15 passing locally🤖 Generated with Claude Code