automagical client rebalancing

[mr-git]

We have NATS deployed on Kubernetes. After deployment or rolling restart, or if one of pods gets restarted, last node (youngest) to start gets least number of client connections and it stays like that for very long time (our services tend to keep connections open for their lifetime).

Is there a way to somehow configure NATS server to automatically ask clients to rebalance?

I found #343, but there doesn't seem to be any resolution.

Asked Claude and it offered two ways, both are not "nice":

• (manually, though scriptable, but not automatic) use "lame duck" to rebalance connections by either:
  • switch node to ldm mode to force it to drain client connections
  • find client connections and issue nats --context sys req '$SYS.REQ.SERVER.<server_id>.LDM' '{"cid":1234}' to make connection with cid:1234 reconnect
• use load-balancer in front of NATS and let it actively rebalance connections at runtime

Is there a way to configure NATS so that it automatically distribute client connections?

[ripienaar]

The nats cli has nats server cluster balance for this

[ripienaar]

thats an option yes - we dont add this kind of complex orchestrations to the server we instead let it focus on messages and we do the rest elsewhere.

if its more your thing the library for balance is in nats-io/jsm.go so you can write your own tool however you like

[mr-git]

Have to add --cluster CLUSTER_ID (from: nats-io/natscli#1572) - otherwise ALL connections are close, even those across super-cluster.

From the issue #1572 and some searches on Google, I was left with impression that the suggested command would close the client connections, instead of actively lame-duck-ing them gracefully. Is that true?

The command actually prints:

The clients will trigger normal reconnect behavior. This can interrupt in-flight work.

("normal reconnect" with "interrupt in-flight" doesn't sound like friendly pair).

Does it mean that connections are forcefully closed or are they lame-duck-ingly asked to reconnect?

[ripienaar]

yes there is no API to do a per client LDM - but really LDM in most clients is a noop, they dont initiate a drain or anything

[ripienaar]

the client libraries will buffer while reconnecting, and this is exactly the same thing that happens on network hiccup etc

[mr-git]

thank you!