This server holds two Postgres LISTEN subscriptions. One of them re-reads its state when the
subscription comes back; the other does not, and every channel deletion, pin and message announced
while it was away is lost to every browser that stayed connected.
The one argument between them
server/src/computer/policy-listener.ts:59 subscribes with an establish callback as well as a
message handler:
await connection.listen(ACTION_POLICY_TOPIC, reread, reread);
server/src/channels/events.ts:98 subscribes with only a message handler:
await connection.listen(CHANNEL_ACTIVITY_TOPIC, (payload) => { ... });
onlisten fires on every establish, reconnects included — 22ms and 51ms on two consecutive kills of
a healthy local socket. The policy listener spends that moment re-reading its row, for the reason its
own comment gives at policy-listener.ts:44: a replica whose connection dropped "misses the
announcement and goes on enforcing the rules it read at boot". The channel listener has no such
moment, and channel-events.integration.test.ts has no equivalent of
policy-fanout.integration.test.ts:134, "catches up when its subscription comes back".
Reproduction
Postgres up (docker compose up -d postgres); no Intelligence key, no browser.
- Point a
startChannelActivityListener at a hub with one registered connection, and a
startPolicyListener at a policy store, on the same database.
pg_notify a deleted: true channel event and change the policy. Both arrive.
- Take the subscription's backend pid from
pg_stat_activity and pg_terminate_backend it in a
loop until the query returns nothing, so the gap is real rather than assumed.
- In the gap, change the policy and
pg_notify a second deleted: true event.
- Let both subscriptions reappear, then publish a third event.
policy recovered the change made in the gap: true
browser recovered the delete made in the gap: false
channel subscription is live again: true
The third event arrives, so the subscription is healthy. The second one is gone. Same result on
three consecutive runs.
The recovery that does not run
events.ts:11 says a client that misses events "recovers by refetching on reconnect", and the client
implements exactly that: socket.onopen at app/src/lib/channels/use-channel-events.ts:129
invalidates the roster query. But the connection that dropped is not the browser's. Its WebSocket is
untouched for the whole gap, so onopen never fires again — the comment's premise is a disconnection
the client can observe, and the loss is on a wire it cannot see.
Nothing else covers it. app/src/query-client.ts:7 sets refetchOnWindowFocus: false, and the
roster query at app/src/lib/channels/queries.ts:57 overrides neither that nor refetchInterval.
applyChannelEvent refetches only when an event names a channel the cache does not hold — a stale
row is in the cache, so later events patch it instead. The row survives as long as the tab does,
leaving exactly what channels/routes.ts:653 added its announcement to prevent.
What a fix probably has to do
onlisten is the shape but cannot be the whole of it: unlike the policy there is no row to re-read,
and a per-member delta cannot be replayed. What the callback can do is tell the browsers to do what
they already do on their own reconnect — which means the hub needs a way to reach every registered
connection rather than the members named in an event, and the client answers it with the
invalidateQueries it already calls.
One thing that is not obvious until you try it: the broadcast has to skip the first establish, or it
asks every connection to refetch on a boot where nothing was missed and the message comes to mean
"possibly a gap" rather than "there was a gap".
I have this working locally, with a test that fails against the listener as it stands, and am happy
to open it. One call I would rather leave to you: whether a re-establish should also log. The policy
listener logs loudly on a failed refresh; a line per reconnect could be noise on a flaky link.
Severity
Not urgent, but it does not heal on its own.
Nothing is corrupted and the roster query stays authoritative. What is missing is any path back to a
correct screen: nothing asks again, so barring an incidental refetch the tab renders the stale row
for as long as it stays open.
The window is not the tens of milliseconds a terminated backend gives you. It is a failover, a
rolling upgrade, a pooler bounce: every replica's subscription down at once, so everything announced
across it is lost to every connected tab together. The screen is then confidently wrong, both
surfaces agree with each other, and the first signal is somebody clicking a conversation that is not
there.
This server holds two Postgres
LISTENsubscriptions. One of them re-reads its state when thesubscription comes back; the other does not, and every channel deletion, pin and message announced
while it was away is lost to every browser that stayed connected.
The one argument between them
server/src/computer/policy-listener.ts:59subscribes with an establish callback as well as amessage handler:
server/src/channels/events.ts:98subscribes with only a message handler:onlistenfires on every establish, reconnects included — 22ms and 51ms on two consecutive kills ofa healthy local socket. The policy listener spends that moment re-reading its row, for the reason its
own comment gives at
policy-listener.ts:44: a replica whose connection dropped "misses theannouncement and goes on enforcing the rules it read at boot". The channel listener has no such
moment, and
channel-events.integration.test.tshas no equivalent ofpolicy-fanout.integration.test.ts:134, "catches up when its subscription comes back".Reproduction
Postgres up (
docker compose up -d postgres); no Intelligence key, no browser.startChannelActivityListenerat a hub with one registered connection, and astartPolicyListenerat a policy store, on the same database.pg_notifyadeleted: truechannel event and change the policy. Both arrive.pg_stat_activityandpg_terminate_backendit in aloop until the query returns nothing, so the gap is real rather than assumed.
pg_notifya seconddeleted: trueevent.The third event arrives, so the subscription is healthy. The second one is gone. Same result on
three consecutive runs.
The recovery that does not run
events.ts:11says a client that misses events "recovers by refetching on reconnect", and the clientimplements exactly that:
socket.onopenatapp/src/lib/channels/use-channel-events.ts:129invalidates the roster query. But the connection that dropped is not the browser's. Its WebSocket is
untouched for the whole gap, so
onopennever fires again — the comment's premise is a disconnectionthe client can observe, and the loss is on a wire it cannot see.
Nothing else covers it.
app/src/query-client.ts:7setsrefetchOnWindowFocus: false, and theroster query at
app/src/lib/channels/queries.ts:57overrides neither that norrefetchInterval.applyChannelEventrefetches only when an event names a channel the cache does not hold — a stalerow is in the cache, so later events patch it instead. The row survives as long as the tab does,
leaving exactly what
channels/routes.ts:653added its announcement to prevent.What a fix probably has to do
onlistenis the shape but cannot be the whole of it: unlike the policy there is no row to re-read,and a per-member delta cannot be replayed. What the callback can do is tell the browsers to do what
they already do on their own reconnect — which means the hub needs a way to reach every registered
connection rather than the members named in an event, and the client answers it with the
invalidateQueriesit already calls.One thing that is not obvious until you try it: the broadcast has to skip the first establish, or it
asks every connection to refetch on a boot where nothing was missed and the message comes to mean
"possibly a gap" rather than "there was a gap".
I have this working locally, with a test that fails against the listener as it stands, and am happy
to open it. One call I would rather leave to you: whether a re-establish should also log. The policy
listener logs loudly on a failed refresh; a line per reconnect could be noise on a flaky link.
Severity
Not urgent, but it does not heal on its own.
Nothing is corrupted and the roster query stays authoritative. What is missing is any path back to a
correct screen: nothing asks again, so barring an incidental refetch the tab renders the stale row
for as long as it stays open.
The window is not the tens of milliseconds a terminated backend gives you. It is a failover, a
rolling upgrade, a pooler bounce: every replica's subscription down at once, so everything announced
across it is lost to every connected tab together. The screen is then confidently wrong, both
surfaces agree with each other, and the first signal is somebody clicking a conversation that is not
there.