Cluster can't tolerate more than one service failure

[AhmedAbouelkher]

Hi,
I'm trying to create a cluster to serve as a messaging queue using pull consumers using docker composer.

docker-compose.yml

version: "3.9"

services:
  nats:
    image: "nats:2.8.4-alpine3.15"
    container_name: nats
    ports:
    - "4222:4222"
    - "8222:8222"
    volumes:
    - ~/Downloads/nats-storage/cluster/jetstream-c1:/tmp/nats/jetstream
    - ./config.c.conf:/config.conf
    command: "--config /config.conf --server_name S1"
    networks:
    - nats

  nats-c2:
    image: "nats:2.8.4-alpine3.15"
    container_name: "nats-c2"
    ports:
    - "5222:4222"
    volumes:
    - ~/Downloads/nats-storage/cluster/jetstream-c2:/tmp/nats/jetstream
    - ./config.c.conf:/config.conf
    command: "--config /config.conf --server_name S2"
    depends_on:
    - nats
    networks:
    - nats

  nats-c3:
    image: "nats:2.8.4-alpine3.15"
    container_name: "nats-c3"
    ports:
    - "6222:4222"
    volumes:
    - ~/Downloads/nats-storage/cluster/jetstream-c3:/tmp/nats/jetstream
    - ./config.c.conf:/config.conf
    command: "--config /config.conf --server_name S3"
    depends_on:
    - nats
    networks:
    - nats

networks:
  nats:
   external: true

config.c.conf

debug: true
trace: false

http_port: 8222
listen: 0.0.0.0:4222

jetstream = {}

cluster = {
    name: "NATS"

    listen: "0.0.0.0:4123"

    authorization = {
      user: route_user
      password: pwd
      timeout: 5
    }

    routes = [
        nats://route_user:pwd@nats:4123
        nats://route_user:pwd@nats-c2:4123
        nats://route_user:pwd@nats-c3:4123
    ]
}

I'm using Golang client version v1.16.0.

[kozlovic]

@AhmedAbouelkher Could you describe a bit more the issue you are having? If by "Cluster can't tolerate more than one service failure" you mean that the cluster will stop serving clients if 2 out of the 3 servers are down, then yes, this is correct as in all distributed systems that rely on a RAFT consensus that need a quorum, which in the case of a cluster size of 3 means that it can survive only 1 failure: https://docs.nats.io/running-a-nats-service/configuration/clustering/jetstream_clustering#the-quorum

[AhmedAbouelkher]

@kozlovic Thanks for your interest.

• I want to be sure if my configs are correct or need to be edited.

• The behavior I got was bizarre

  • When stopping service nats-c3 with nats-c2 & nats running, the cluster does run with no issue
  • When stopping service nats-c2 with nats-c3 & nats running, the cluster does completely crash
  • When stopping service nats with nats-c3 & nats-c2 running, the cluster does completely crash

I tried many configs with the same results as above.
I don't have any issue with one service down. but the seen behavior is not consistent.

[kozlovic]

Make sure that the stream you created is an R3 because if it is R1 and hosted by the server you are shutting down, then it is normal that this stream is unavailable.

[AhmedAbouelkher]

I got it now.
I have one minor question, do you mean by R3 stream the number of message replicas?

I changed my code to be as so

info, err := js.AddStream(&nats.StreamConfig{
	  Name:     "stream",
	  Subjects: []string{"subject.>", "subject"},
	  Replicas: 3,
})

What do you think?

I found these resources as well https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/disaster_recovery#automatic-recovery and https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/disaster_recovery#automatic-recovery but I'm not sure

[AhmedAbouelkher]

And what happens if the nats leader service is down and I'm already connected to nats.DefaultURL?

[AhmedAbouelkher]

@kozlovic Tested creating a stream with Replicas: 3 and it works 🎉, but if the leader nats is down, the whole service collapses like before

[kozlovic]

You need to have the client be able to reconnect to the other servers. Make sure you provide the list of the 3 servers in the connect URL. Normally servers will gossip the cluster client connect URLs to the client libraries, but the way it is running right now I suspect that the server sends the internal ports that are not visible from the client library. So easier to just provide the 3 ports to your app ("nats://localhost:4222,nats://localhost:5222,nats://localhost:6222")