NATS client reconenction issue

[kripanshdevtron]

No description provided.

[kripanshdevtron]

Whenever Nats pod gets restarted, its client subscribers' stream is not able to reconnect.

To Reproduce

1. Restart Nat Server Pod. make sure it is not running in cluster mode. if that is the case then restart all the pods.
2. CI/CD events are stuck on Ui because those event listeners are not subscribed back.

Expected behavior

By any chance, Nats-server pod gets restarted then its client should be able to reconnect back.

Current behavior

whenever Nats pod gets restarted. Its Client is able to reconnect back but subscribers are not able to listen to its events

[kripanshdevtron]

Whenever nats-server pods get restarted. streams got deleted so once its clients are connected back, no streams are present due to which nats client don't find any stream to connect back. The reason for data deletion is we have not mounted any external volume for data backup, another solution for this is to run the Nats server in cluster mode with stream replica count of 3.

** References **

1. PullSubscribe does not support reconnection nats-io/nats.go#890 (comment)
2. https://docs.nats.io/using-nats/developer/connecting/reconnect/buffer
3. https://docs.nats.io/using-nats/developer/connecting/reconnect/events
4. https://docs.nats.io/using-nats/developer/connecting/cluster
5. https://docs.nats.io/nats-concepts/jetstream/streams
6. https://docs.nats.io/running-a-nats-service/nats-kubernetes/helm-charts#logging