streaming state was recovered but cluster log path "mycluster/a" is empty

[willihelm]

I tried to setup a cluster of 3 nats-streaming nodes. I followed the documentation and set up 3 different config files in docker containers. This worked fine until I decided to use a mounted directory for the nodes. When I start the first time it works but if I restart a container I get the following error:

Willis-MacBook-Pro:a Willi$ docker run -p 8222:8222 -p 6222:6222 -p 4222:4222 -v /Users/Willi/Desktop/nats:/mnt/natsdata nats-streaming-a
[1] 2018/04/12 14:23:49.954976 [INF] STREAM: Starting nats-streaming-server[mycluster] version 0.9.2
[1] 2018/04/12 14:23:49.955142 [INF] STREAM: ServerID: mL0inzQufS9ZUPBpN4enBc
[1] 2018/04/12 14:23:49.955159 [INF] STREAM: Go version: go1.9.5
[1] 2018/04/12 14:23:49.956919 [INF] Starting nats-server version 1.0.7
[1] 2018/04/12 14:23:49.956949 [INF] Git commit [not set]
[1] 2018/04/12 14:23:49.957061 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2018/04/12 14:23:49.957118 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2018/04/12 14:23:49.957134 [INF] Server is ready
[1] 2018/04/12 14:23:49.957502 [INF] Listening for route connections on 0.0.0.0:6222
[1] 2018/04/12 14:23:49.960473 [ERR] Error trying to connect to route: dial tcp 192.168.2.104:6224: getsockopt: connection refused
[1] 2018/04/12 14:23:49.964655 [ERR] Error trying to connect to route: dial tcp 192.168.2.104:6223: getsockopt: connection refused
[1] 2018/04/12 14:23:49.985997 [INF] STREAM: Recovering the state...
[1] 2018/04/12 14:23:50.138470 [INF] STREAM: Recovered 4 channel(s)
[1] 2018/04/12 14:23:50.139032 [INF] STREAM: Cluster Node ID : a
[1] 2018/04/12 14:23:50.139456 [INF] STREAM: Cluster Log Path: mycluster/a
[1] 2018/04/12 14:23:50.147696 [INF] STREAM: Shutting down.
[1] 2018/04/12 14:23:50.150171 [INF] Server Exiting..
[1] 2018/04/12 14:23:50.150745 [FTL] STREAM: Failed to start: streaming state was recovered but cluster log path "mycluster/a" is empty

[kozlovic]

You would need to specify -cluster_log_path to point to a mounted directory too so that the raft store log is also recovered.
I will add a note to the documentation since this start to come up a lot. Thanks!

[willihelm]

Thank you for the quick answer. I thought I only have to specify this path if I use an SQL store. I will try again.

[kozlovic]

@willihelm Please check PR and see if that would have make it more clear. Thanks for the report!

[willihelm]

Still can´t get it to work. Could you please provide an example for this in a config file? My current config file looks like this:

# NATS Streaming specific configuration
streaming {
  id: mycluster
  store: file
  dir: /mnt/natsdata/
  cluster_log_path: /mnt/natsdata/
  cluster {
    node_id: "a"
    peers: ["b", "c"]
  }
  store_limits: {
    max_age: "336h"
  }
}

I also tried a paths like these:

cluster_log_path: /mnt/natslogs/a/

cluster_log_path: /mnt/natsdata/mycluster/a/

Also I don´t know if I have to mkdir first.

[kozlovic]

The -cluster_log_path is the command line parameter version of this parameter. For configuration file, please follow instructions here.
You need to put log_path in the cluster section along with node_id, peers, etc..
You do not need to specify the cluster id and node id in the path. That will be created by the server to store the node info in that location. So /mnt/nastdata should be fine.

[willihelm]

Works fine. And the PR looks good. Thanks a lot.

[tharrold]

Follow-up question: we had a node in our 3-node cluster crash (due to the / filesystem filling up). I am unable to restart NATS on that node. I removed the store, and the log_path directories and restarted, but we see the following:

[12120] 2020/04/28 11:25:21.681533 [FTL] STREAM: Failed to start: streaming state was recovered but cluster log path "/etc/nss/mtl-dev/mtl-dev" is empty
so part of the recovery seems to work.

I check, and it DID create the log_path, and there is a raft.log file present.