Leaf node jetstream not receiving messages after restart #3024
Comments
|
@JH7 This is because you are using the same store directory for the 2 servers. For instance, if you were to change leaf_test.conf to this: it will work fine, even after a restart of the leaf server. Give it a try and I am sure it will work. Closing for now, will re-open if you still have issues. |
|
@kozlovic Thanks for your reply. Nevertheless, this solution does not work for me. I tried configuring Also note that in our production environment, both main and leaf node use seperate volume mounts. I'm attaching the log for the case in which i configured the |
|
You need to use a persistent server_name that survives restarts. |
kozlovic
added a commit
that referenced
this issue
Apr 13, 2022
When using subscriptions through import/exports, the server with a leafnode connection would properly send the interest over, but if the connection is recreated, this would not happen. In case of JetStream where that happens under the cover, message flow would stop after the leafnode restart because the subscriptions would be created on recovery of the JetStream assets but *before* the LeafNode connection could be established. Resolves #3024 Resolves #3027 Resolves #3009 Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
kozlovic
added a commit
that referenced
this issue
Apr 13, 2022
When using subscriptions through import/exports, the server with a leafnode connection would properly send the interest over, but if the connection is recreated, this would not happen. In case of JetStream where that happens under the cover, message flow would stop after the leafnode restart because the subscriptions would be created on recovery of the JetStream assets but *before* the LeafNode connection could be established. Resolves #3024 Resolves #3027 Resolves #3009 Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
|
@JH7 The issue should have been resolved now. Thank you for the report. |
|
@kozlovic thank you for the quick fix! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Defect
nats-server -DVoutputDescription
In our production environment, we use two instances of nats servers. One instance is the main server (accessible only from within our Kubernetes cluster) while the other one is a leaf node. The leaf node is accessible from outside of our Kubernetes cluster (via websocket).
We use accounts to isolate and manage messages and topics between the main server and the leaf node. The main server account DMZ exports a stream which the leaf node account CLIENT imports. Note that there is another account, PUBLIC, on both main and leaf server which is used as authorization account (as shown in the configs below). Furthermore, we have a jetstream on this exported stream on the leaf node's CLIENT account. This configuration does work (the jetstream shows messages when they are published on the main server) on the first start of the instances, while it breaks whenever the instances are restarted. Messages published on the main server's DMZ account are not reaching the CLIENT's jetstream on the leaf node anymore. Interestingly, by manually subscribing to the jetstream's topic using
nats --server localhost:4111 --user client --password client sub 'orders.>', the messages reach the jetstream again. As soon as the subscription is terminated, the messages are not reaching the jetstream again. Note that this issue occured in our production environment first but I was able to replicate this issue on my local machine.My guess is that some internal routing tables (in main or leaf server, or both?) are not updated when the leaf node restarts. The log does not show the re-importing of subscriptions after the leaf node has restarted.
I was not able to fix this without removing and re-creating the jetstream.
Versions of
nats-serverand affected client libraries used:nats-server: 2.7.4
nast-cli: 0.0.28
OS/Container environment (used in the reproduction):
Windows 10 (WSL2) 4.19.128-microsoft-standard; Ubuntu 20.04.3 LTS
Docker Desktop 4.3.2, Engine 20.10.11, Compose 1.29.2
Steps or code to reproduce the issue:
Main server config:
Leaf node config:
Docker compose file used:
nats --server localhost:4111 --user client --password client stream create orders '--subjects=orders.*.stream.entry' --retention=work --max-consumers=-1 --max-msgs-per-subject=-1 --max-msgs=-1 --max-bytes=-1 --max-age=-1 --max-msg-size=-1 --storage=file --discard=old --replicas=1 --dupe-window="2m0s" --no-allow-rollup --no-deny-delete --no-deny-purgenats --user service --password service pub 'orders.client.stream.entry' 'test1'nats --server localhost:4111 --user client --password client stream info ordersnats --user service --password service pub 'orders.client.stream.entry' 'test2'Expected result:
The stream contains the published message.
Actual result:
The stream does not contain the published message. Verify this by using
nats --server localhost:4111 --user client --password client stream info ordersagain. It does not show two messages.Log
The text was updated successfully, but these errors were encountered: