Fragmented Clusters

[scottgarner]

I'm using DOG for a project where users connect via websockets and are grouped into "namespaces". All traffic sent from one user in a namespace is sent to all other users. In general this works fine, both with a single Replica and with users spread across multiple Replicas through the use of "gossip".

Eventually, however, if a user in a namespace disconnects and reconnects or a new user connects, they seemingly end up on a totally separate cluster, isolated from other users. As other users disconnect/reconnect, they'll also end up in this new cluster. There's only one Group with a constant key. It is unfortunately not easy to reproduce, but has happened many times.

A user on the Cloudflare Developers Discord suggested that this may be because DOG doesn't persist the Group's list of Replicas. So, if something were to cause the Group to reinitialize, it would have no knowledge of the previous set of Replicas, even though users are still connected to them. This seems to align with what I'm seeing, but I'm not sure how to fully verify because I'm not sure what causes the Group to restart/reinitialize.

[qw-in]

@scottgarner I'm running into the same issue. My guess is the Group is getting cleaned up after 30s which ends up with the replicas being lost.

I guess a couple of solutions could be:

• Have some sort of heartbeat broadcast to group every few seconds to keep it up
• Add logic to persist the connection ids to storage & retrieve upon starting back up

[rennokki]

@qw-in @scottgarner I have happened to encounter this while having DOG running the Cluster Mode in soketi.

The problem was solved by passing the Cluster Durable Object ID to the Replica Durable Objects and setting this in the Replica's constructor method:

setInterval(() => {
    if (this.clusterDurableObjectId) {
        if (!this.clusterDurableObject) {
            this.clusterDurableObject = env.CLUSTERS.get(this.clusterDurableObjectId);
        }

        this.clusterDurableObject.fetch('http://heartbeat');
    }
}, 5_000);

For the store, I plan on forking this and creating a new version at soketi/dog.