NATS /JetStream persistent storage in docker context for Stream, Subjects, Durable Consumers, Messages

[andronachev]

Hello,

We are facing the following:

• deployed nats-jetstream in a docker pod using Deployments
• configured the Stream, Subjects, and durable consumers at runtime of the client service(s), using NATS.NET
• everything is configured using storage: "File" (files on disk on the docker pod)
• we do this configuration at runtime only when Subscribing, not when publishing, and we do it if we receive NATS.NET NATSJetStreamClientException with Id SUB-90007 (subject not existing). This is in order to not create overhead of upserting all configuration, when doing day-do-day operations after the server is configured.

However, the complete state of the server is lost at redeployment time or when pod is recreated automatically... storage "File" works on simple container restart, but not on container recreation. This means the client services are automatically reconnected, out of the box with NATS.NET, but the subscription is lost, durable consumers are lost, streams and subjects as well.

Is there a suggested approach on handling this ? unfortunately I did not find it in the documentation. We did find the "SQL Store" approach but that is in the legacy nats-streaming: https://docs.nats.io/legacy/stan/changes/configuring/persistence/sql_store . We would require something similar, I imagine.

We are also considering making the client applications 100% resilient, meaning to configure at runtime, also at publish not only at Subscribe time, and to also recreate the subscriptions and consumers on the Reconnect event handler.

Thank you

[andronachev]

related to this discussion: #3507

[bruth]

Hi @andronachev. Based on what you provided there appears to be a few moving parts we need to tease apart.

• One up-front question, are you referring to JetStream or NATS Streaming (which has been deprecated)? The link you provided reference NATS Streaming which is a separate project from NATS server (happy to expand on this if this is confusing).
• When running the Docker containers, are you ensuring the Docker volumes are setup properly so that the JetStream storage directory is durable across containers restarts?

[andronachev]

Hi @bruth thanks for replying.

• I am referring to JetStream. I was just mentioning that in NATS Streaming (deprecated) there was the possibility for storage in a SQL Database, and that in JetStream it's not implemented
• it seems we don't have the Docker volumes setup in a persistent way

In the meanwhile since starting the discussion, we implemented resilience at Publish time as well as at Reconnect time, i.e. we create the Stream and Subjects at these events, if we receive the specific NATS.NET Exceptions, for example for "No Subject Exception" ID = "SUB-90007"

I get it, then, that there is no other way other than having the Volumes set up ? since in discussion #3507 it is stated that JetStream will not support storage in an external SQL Database.

Thank you