Auto-reconnecting in client when WebSocket not intentionally closed

[ThatOneBro]

WebSockets are often flaky under circumstances where connectivity is spotty or inconsistent. We can make working with WebSocket Subscriptions and WebSockets in general easier for application authors using MedplumClient by auto-reconnecting behind the scenes and in the case of WebSocket Subscriptions recreating subscriptions with the same criteria automatically if the connection reconnects

[dillonstreator]

If a Websocket Subscription is disconnected due to a flaky client connection or a server process exit (expected during deployment or unexpected from an error), gracefully reconnecting in the MedplumClient makes a lot of sense.

Given the sensitive/urgent and timely nature of some of the use cases that may call for a Websocket Subscription, in addition to gracefully reconnecting, it would be valuable to enable clients to request a 'replay' of subscription-matching resources between the time of a disconnect->reconnect to ensure clients are not dropping events that could be urgent. I would anticipate this 'replay' to be a default behavior of the MedplumClient, if supported by the server.

I couldn't find this concept outlined by the FHIR spec but curious if this something that is on the Medplum teams radar and/or if there's a more appropriate workaround (than what is outlined below) to get the benefits of real-time changes in the client while also having better guarantees of not dropping events?

One workaround is a calling client could listen for a disconnect/reconnect events and query for resources matching the subscription filter between the disconnect period using the lastUpdated field, however, this unfortunately wouldn't account for the more complex filtering logic that can be encoded in the subscription extensions.

From a solutioning standpoint, the subscription could be retained after a disconnect and events buffered in redis with a "reasonable" TTL. The client would provide the same subscriptionId on reconnect which should resolve on the server if it occurs within the TTL window and drain the buffered events. I imagine the websocket observability work could help with identifying a "reasonable" TTL as well as any sort of capacity planning in the buffering #4685. Ultimately, this seems not ideal as a reconnect outside the TTL would still result in dropped events and all legitimate disconnects will always result in unnecessary buffering (not to mention the ballooning complexity).

Defer to the Medplum team but this is likely a larger conversation worth bringing to #1490