Skip to content

A channel event announced while a server's subscription is down is lost to every browser that stayed connected #310

Description

@zopeVaibhav

This server holds two Postgres LISTEN subscriptions. One of them re-reads its state when the
subscription comes back; the other does not, and every channel deletion, pin and message announced
while it was away is lost to every browser that stayed connected
.

The one argument between them

server/src/computer/policy-listener.ts:59 subscribes with an establish callback as well as a
message handler:

await connection.listen(ACTION_POLICY_TOPIC, reread, reread);

server/src/channels/events.ts:98 subscribes with only a message handler:

await connection.listen(CHANNEL_ACTIVITY_TOPIC, (payload) => { ... });

onlisten fires on every establish, reconnects included — 22ms and 51ms on two consecutive kills of
a healthy local socket. The policy listener spends that moment re-reading its row, for the reason its
own comment gives at policy-listener.ts:44: a replica whose connection dropped "misses the
announcement and goes on enforcing the rules it read at boot". The channel listener has no such
moment, and channel-events.integration.test.ts has no equivalent of
policy-fanout.integration.test.ts:134, "catches up when its subscription comes back".

Reproduction

Postgres up (docker compose up -d postgres); no Intelligence key, no browser.

  1. Point a startChannelActivityListener at a hub with one registered connection, and a
    startPolicyListener at a policy store, on the same database.
  2. pg_notify a deleted: true channel event and change the policy. Both arrive.
  3. Take the subscription's backend pid from pg_stat_activity and pg_terminate_backend it in a
    loop until the query returns nothing, so the gap is real rather than assumed.
  4. In the gap, change the policy and pg_notify a second deleted: true event.
  5. Let both subscriptions reappear, then publish a third event.
   policy recovered the change made in the gap: true
   browser recovered the delete made in the gap: false
   channel subscription is live again:           true

The third event arrives, so the subscription is healthy. The second one is gone. Same result on
three consecutive runs.

The recovery that does not run

events.ts:11 says a client that misses events "recovers by refetching on reconnect", and the client
implements exactly that: socket.onopen at app/src/lib/channels/use-channel-events.ts:129
invalidates the roster query. But the connection that dropped is not the browser's. Its WebSocket is
untouched for the whole gap, so onopen never fires again — the comment's premise is a disconnection
the client can observe, and the loss is on a wire it cannot see.

Nothing else covers it. app/src/query-client.ts:7 sets refetchOnWindowFocus: false, and the
roster query at app/src/lib/channels/queries.ts:57 overrides neither that nor refetchInterval.
applyChannelEvent refetches only when an event names a channel the cache does not hold — a stale
row is in the cache, so later events patch it instead. The row survives as long as the tab does,
leaving exactly what channels/routes.ts:653 added its announcement to prevent.

What a fix probably has to do

onlisten is the shape but cannot be the whole of it: unlike the policy there is no row to re-read,
and a per-member delta cannot be replayed. What the callback can do is tell the browsers to do what
they already do on their own reconnect — which means the hub needs a way to reach every registered
connection rather than the members named in an event, and the client answers it with the
invalidateQueries it already calls.

One thing that is not obvious until you try it: the broadcast has to skip the first establish, or it
asks every connection to refetch on a boot where nothing was missed and the message comes to mean
"possibly a gap" rather than "there was a gap".

I have this working locally, with a test that fails against the listener as it stands, and am happy
to open it. One call I would rather leave to you: whether a re-establish should also log. The policy
listener logs loudly on a failed refresh; a line per reconnect could be noise on a flaky link.

Severity

Not urgent, but it does not heal on its own.

Nothing is corrupted and the roster query stays authoritative. What is missing is any path back to a
correct screen: nothing asks again, so barring an incidental refetch the tab renders the stale row
for as long as it stays open.

The window is not the tens of milliseconds a terminated backend gives you. It is a failover, a
rolling upgrade, a pooler bounce: every replica's subscription down at once, so everything announced
across it is lost to every connected tab together. The screen is then confidently wrong, both
surfaces agree with each other, and the first signal is somebody clicking a conversation that is not
there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions