Bug: WhatsApp bridge /health reports 'connected' during Baileys WebSocket flapping (428/503 loop), causing silent message loss

[kk17]

Bug: WhatsApp bridge /health reports connected during Baileys WebSocket flapping (428/503 reconnect loop), causing silent message loss

Summary

The WhatsApp bridge's /health endpoint reports connectionState: "connected" even when the Baileys WebSocket is in a rapid disconnect-reconnect loop (codes 428/503). Because the gateway relies on /health to determine platform liveness, it considers WhatsApp fully operational and never triggers the reconnection watcher. Messages sent by users during the brief disconnect windows are silently lost — neither received by the bridge nor flagged as missed.

Environment

• Hermes version: v2026.7.7.2-247-g54e186457 (0.18.2)
• Bridge: scripts/whatsapp-bridge/bridge.js, Baileys @whiskeysockets/baileys 7.0.0-rc13
• Mode: WHATSAPP_MODE=bot
• OS: Debian (Linux 6.12.30)

Problem

When WhatsApp's server starts sending disconnect codes (428 = precondition required, 503 = service unavailable), Baileys rapidly cycles through:

✅ WhatsApp connected!
⚠️ Connection closed (reason: 428). Reconnecting in 3s...
✅ WhatsApp connected!
⚠️ Connection closed (reason: 428). Reconnecting in 3s...

During the ~3-second disconnect windows, any user messages sent to the family group are not received by Baileys (the messages.upsert event never fires). These messages are permanently lost — WhatsApp delivers them to the user's phone but not to the bridge.

Root cause in bridge.js:

// Line 380
let connectionState = 'disconnected';

// Lines 425-448: connectionUpdate handler
if (connection === 'close') {
    connectionState = 'disconnected';   // Set to disconnected...
    // ... 3s later, startSocket() reconnects
} else if (connection === 'open') {
    connectionState = 'connected';      // ...then immediately back to connected
}

// Lines 1071-1077: health endpoint
app.get('/health', (req, res) => {
    res.json({
        status: connectionState,    // Reports "connected" during brief open windows
        queueLength: messageQueue.length,
        uptime: process.uptime(),
        scriptHash: SCRIPT_HASH,
    });
});

The gateway's poll loop (adapter.py line ~1233-1268) checks _check_managed_bridge_exit() but not the health status — it only checks whether the bridge process is alive, not whether the Baileys WebSocket is actually connected. And the platform reconnection watcher only runs for platforms that failed connect(), not for platforms that are flapping.

Observed impact

In a production setup (family meal-planning bot), the bridge entered a 428 loop at ~17:36 and continued until a manual gateway restart at 22:04. During this 4.5-hour window:

• 0 messages were received by the gateway (confirmed via gateway logs)
• The bridge /health reported connected intermittently
• A critical user message (meal plan for the next day, sent at 19:50) was silently lost
• The bridge produced no JSON log events for the entire period, only non-timestamped text output (Connection closed (reason: 428))
• Session key files were still being updated by Baileys during reconnection attempts, making it appear the bridge was receiving data

Expected behavior

1. The bridge should expose connection quality metrics, not just binary connected/disconnected
2. The gateway should detect sustained flapping and either restart the bridge or mark the platform as degraded

Suggested fix

Option A: Bridge-side disconnect counter (minimal)

Add a rolling disconnect counter to bridge.js that tracks recent reconnections. When the count exceeds a threshold (e.g., >5 disconnects in 60 seconds), report a new health status:

let disconnectTimestamps = [];

// In connectionUpdate close handler:
disconnectTimestamps.push(Date.now());
disconnectTimestamps = disconnectTimestamps.filter(t => t > Date.now() - 60000);

// In /health:
const isFlapping = disconnectTimestamps.length > 5;
res.json({
    status: isFlapping ? 'degraded' : connectionState,
    disconnectCount60s: disconnectTimestamps.length,
    queueLength: messageQueue.length,
    uptime: process.uptime(),
    scriptHash: SCRIPT_HASH,
});

Option B: Gateway-side detection (complementary)

In adapter.py's _poll_messages() loop, periodically check /health for status === 'degraded' or track consecutive empty polls. If the bridge reports degraded for sustained periods, trigger a bridge restart via the existing _set_fatal_error mechanism.

Additional context

• The 428/503 errors may indicate the WhatsApp Web session needs re-pairing, but this is a separate concern from the silent message loss.
• The gateway already has a reconnection watcher for failed platforms, but it doesn't cover the "connected-but-flapping" case.
• Bridge version: scriptHash: f6c1394bc6fa2984 (file hash of bridge.js)

[GottZ]

This was generated by AI during triage.

Summary

One PR addresses #63277. #63289 adds bridge-side reconnect-loop detection to report degraded health and adapter-side handling that converts that state into the existing retryable fatal/reconnect path, directly covering the reported mismatch between process liveness and Baileys WebSocket liveness.

Related pull requests

• fix(whatsapp): degrade flapping bridge health #63289 best fix — (+141/-7) — fixes, pending review follow-ups: the diff counts recent bridge disconnects, exposes sustained flapping as degraded through /health, and makes the adapter trigger retryable recovery instead of accepting transient connected states. The contributor keep_open review on fix(whatsapp): degrade flapping bridge health #63289 remains applicable because the Python test only fabricates the health response rather than testing the JavaScript threshold/window/serialization behavior, and the new environment-variable thresholds conflict with the repository's documented configuration policy.

Suggested consolidation

Merge #63289 after addressing its keep_open review by adding direct Node regression coverage for threshold crossing, window expiry, and health serialization, and by replacing the undocumented environment-variable threshold surface with fixed internal constants or repository-compliant configuration. It is the sole and best-fix PR for #63277, so there are no duplicate PRs to close.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I63277(["issue #63277 (open)"])
    P63289["PR #63289 (open)"]
    P63289 -->|best fix| I63277
    class I63277 open
    class I63277 target
    class P63289 open
    class P63289 best
    click I63277 "https://github.com/NousResearch/hermes-agent/issues/63277"
    click P63289 "https://github.com/NousResearch/hermes-agent/pull/63289"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).

Cross-PR triage: Reviewed 1 pull request and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 8 kB of PR diffs, 9 kB of issue/PR text, 1 kB of discussion (1 comments), 2 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[achhabra2]

Independent reproduction of the liveness gap, with a slightly different health-state shape from the original report.

Environment

• Hermes commit: 6b54582438e7496974d32febe381a74ae2db0f9a
• OS: Linux
• Baileys: @whiskeysockets/baileys 7.0.0-rc13
• WhatsApp bridge managed by a profile-specific Hermes gateway/systemd user service

Observed sequence

The bridge had been connected and processing messages normally. It then logged this progression:

Connection closed (reason: 503). Reconnecting in 3s...
WhatsApp connected
Connection closed (reason: 408). Reconnecting in 3s...
Connection closed (reason: 405). Reconnecting in 3s...
Connection closed (reason: 405). Reconnecting in 3s...
...repeated 405 reconnect attempts...

During the failure:

• The Node bridge process remained alive.
• The Python gateway remained alive.
• The systemd service remained active.
• The bridge HTTP port remained open.
• /health returned status: "disconnected" rather than intermittently returning connected.
• No inbound messages reached the gateway, so the outage presented as a silently non-responsive agent.

A controlled restart of the complete profile gateway recreated the bridge/socket and restored status: "connected" using the existing session credentials; no re-pair was required. The bridge remained connected through repeated health checks, then successfully completed a real inbound -> model -> outbound exchange.

This appears to confirm the core adapter liveness gap described here also applies to a bridge that remains continuously disconnected, not only one that briefly reports connected while flapping. The rolling disconnect/degraded-health path in #63289 looks like it would catch this repeated-405 state and hand it to the existing retryable recovery path.

The repeated 405 may also overlap #73700 / #61939 because current bridge code resolves the socket version with fetchLatestBaileysVersion(). I cannot conclusively attribute this incident to a stale WA Web version, however, because a full bridge restart connected successfully with the same credentials and installed Baileys version.

All chat identifiers, account identifiers, message content, and credentials have been omitted intentionally.

[GottZ]

This was generated by AI during triage.

Delta since our previous triage comment

Thanks @achhabra2 for independently reproducing and extending the reported liveness gap: the outage can persist while the bridge process, gateway, service, and HTTP endpoint remain live, with /health staying disconnected rather than briefly returning connected. This does not change the prior assessment because #63289 derives degraded from rolling disconnect events independently of the current connection state, so its approach covers both observed health-state shapes.

Suggested consolidation

The recommendation is unchanged: keep #63289 as the best fix, subject to resolving its existing contributor keep_open review before merge.

Complex graph unchanged since our previous triage comment.

Cross-PR triage: Reviewed 1 pull request and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 8 kB of PR diffs, 9 kB of issue/PR text, 4 kB of discussion (2 comments), 2 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[falbercafe]

Confirming this happens outside the original reporter's setup too — same root cause pattern, different failure mode on the crypto session layer.

Environment:

• Ubuntu 24.04, VM.Standard.A1.Flex (Oracle Cloud)
• WhatsApp mode: self-chat (personal number, not bot mode)
• Gateway managed via systemd --user service

What I observed:

/health reported {"status":"connected", ...} throughout a ~48h window where zero inbound messages were processed. Same blind spot described in this issue — the endpoint doesn't distinguish "socket technically open" from "actually receiving messages."

In my case, the extended flapping/reconnect churn (I didn't confirm 428/503 specifically, but the pattern matches — rapid reconnect cycles logged in bridge.log) appears to have corrupted the Signal session state itself, not just caused silent message loss during the disconnect windows. After the flapping period, I started seeing:

MessageCounterError: Key used already or never filled
Bad MAC
SessionError: No matching sessions found for message

A clean restart of the bridge process (gateway kept it alive via Restart=always) did not clear these errors — the same corrupted session reloaded and the same errors recurred immediately. Only a full re-pair (removing ~/.hermes/whatsapp/session/ and scanning a new QR code) resolved it.

This suggests the impact of the flapping bug described here may go beyond "lost messages during the gap" — sustained flapping can leave the Signal/Baileys session in a state that doesn't self-heal on restart, requiring a full re-pair to recover. Might be worth linking this to #63647 (blank message churn during E2EE retry) since both seem to stem from the same underlying reconnect-storm behavior.

Happy to share sanitized bridge.log excerpts around the corruption window if useful for repro.