[question] Websocket connect termination handling

[apetrosyan1613]

Hi! Thanks for your awesome solution! Im not sure, that it's the right place for my question, but anyway.

We use Centrifugo 5.2.2 (standalone mode) in a Kubernetes cluster (deployed via the Centrifugo helm chart) and expose it via ingress-nginx on the subpath. If I'm not wrong, we have the average number of connects/subscribes in 200-300.

So the case: when I add a new ingress rule/resource with the same domain name, which is used by Centrifugo, it causes the ingress-nginx configuration reload, which leads to the termination of all websocket connections to Centrifugo.

It causes a colossal load on the Centrifugo and service with subscribe and connect endpoints (a lot of client disconnections and connect/subscribe commands to Centrifugo).

questions:

1. I noted some performance optimisations in further Centrifugo minor versions, any chance that it will help in my case if I apply the new minor version?
2. Just any other similar cases that anyone has faced?

[FZambia]

Hi @apetrosyan1613

It's a bit abstract when you are just using "colossal" load words without concrete numbers, metrics, resources, latencies etc.

Generally the situation is known, and I can provide some advices:

• You can use ingress like Envoy which does not drop connections when configuration changes
• But your system should still be ready to a situation like this because sth before ingress can reload too, or when redeploying Centrifugo.
• See part about massive reconnect scenario in our Scaling WebSocket in Go and beyond
  blog post. TLDR: you can use connection and subscription tokens to reduce load on your session service and subscription permission service
• Reduce reconnections rate on load balancer level or using connection_rate_limit Centrifugo option. Our SDKs will reconnect eventually.
• Centrifugo PRO provides faster HTTP client to reduce the resource usage of Centrifugo raising from many HTTP requests
• Scale Centrifugo with Redis Engine and your backend to handle more requests.
• Tune min_reconnect_delay on the client side, it may be too small by default. Delay is jittered in our SDKs, so you will get less spiky load.

Probably more advices can be given here.

Regarding recent improvements in Centrifugo - of course they can help, but I think just a bit, and you better concentrate on techniques above.

[apetrosyan1613]

@FZambia Thanks for the very accurate response! I'll research all the advice.
Very appreciative, if we can keep this discussion for some time