Not able to set up a cluster with more than two nodes locally(Mac)

[longquanzheng]

Hi,
I want to set up a local cluster with three nodes locally in my laptop for debugging.
I was following this instruction "Make a cluster" of https://hub.docker.com/_/cassandra/
Only difference is I want every node to expose CQL/thrift ports.

But every time I added the 3rd node, it is broken. The first node (seed node) would exit without any useful information.

First command to spin up 1st node:

docker run --name cass3.11-1 -p 9242:9042 -p 9260:9160 --log-opt max-size=5g cassandra:3.11

Second command to spin up 2nd node:

docker run --name cass3.11-2  -p 9342:9042 -p 9360:9160 -e CASSANDRA_SEEDS="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' cass3.11-1 )" cassandra:3.11

After everything looks fine in the console output of the above two commands, I started the 3rd command:

docker run --name cass3.11-3  -p 9442:9042 -p 9460:9160 -e CASSANDRA_SEEDS="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' cass3.11-1 )" cassandra:3.11

Then suddenly the first node would exit. Only left information:

INFO  [main] 2018-04-13 21:33:25,334 Server.java:156 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO  [main] 2018-04-13 21:33:25,387 CassandraDaemon.java:527 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO  [OptionalTasks:1] 2018-04-13 21:33:27,313 CassandraRoleManager.java:355 - Created default superuser role 'cassandra'
INFO  [GossipStage:1] 2018-04-13 21:34:49,437 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED
INFO  [HANDSHAKE-/172.17.0.3] 2018-04-13 21:34:49,475 OutboundTcpConnection.java:560 - Handshaking version with /172.17.0.3
INFO  [GossipStage:1] 2018-04-13 21:34:51,372 Gossiper.java:1067 - Node /172.17.0.3 is now part of the cluster
INFO  [RequestResponseStage-1] 2018-04-13 21:34:51,384 Gossiper.java:1031 - InetAddress /172.17.0.3 is now UP
WARN  [GossipTasks:1] 2018-04-13 21:34:51,674 FailureDetector.java:288 - Not marking nodes down due to local pause of 95386755752 > 5000000000
INFO  [HANDSHAKE-/172.17.0.3] 2018-04-13 21:34:51,686 OutboundTcpConnection.java:560 - Handshaking version with /172.17.0.3
INFO  [STREAM-INIT-/172.17.0.3:52154] 2018-04-13 21:35:22,714 StreamResultFuture.java:116 - [Stream #921d2a90-3f62-11e8-a723-f58bd8d3766a ID#0] Creating new streaming plan for Bootstrap
INFO  [STREAM-INIT-/172.17.0.3:52154] 2018-04-13 21:35:22,757 StreamResultFuture.java:123 - [Stream #921d2a90-3f62-11e8-a723-f58bd8d3766a, ID#0] Received streaming plan for Bootstrap
INFO  [STREAM-INIT-/172.17.0.3:52156] 2018-04-13 21:35:22,760 StreamResultFuture.java:123 - [Stream #921d2a90-3f62-11e8-a723-f58bd8d3766a, ID#0] Received streaming plan for Bootstrap
INFO  [STREAM-IN-/172.17.0.3:52156] 2018-04-13 21:35:22,936 StreamResultFuture.java:187 - [Stream #921d2a90-3f62-11e8-a723-f58bd8d3766a] Session with /172.17.0.3 is complete
INFO  [STREAM-IN-/172.17.0.3:52156] 2018-04-13 21:35:22,954 StreamResultFuture.java:219 - [Stream #921d2a90-3f62-11e8-a723-f58bd8d3766a] All sessions completed
INFO  [Service Thread] 2018-04-13 21:35:32,708 GCInspector.java:284 - ParNew GC in 864ms.  CMS Old Gen: 11203864 -> 21608984; Par Eden Space: 208928768 -> 0;
longer@~/cassandra/compose-cassandra-cluster:(master)$

And the 3rd node also exit as well because it couldn't connect to the seed. The 2nd keep alive since it already connected to seed node before.

Some other environment info:

longer@~:$ docker --version
Docker version 17.12.0-ce, build c97c6d6

longer@~:$ uname -a
Darwin longer-C02V60N3HTDG 17.3.0 Darwin Kernel Version 17.3.0: Thu Nov  9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64 x86_64

[yosifkit]

Most likely you are running out of memory. By default Cassandra will use 40-50% of total memory. You'll want to set MAX_HEAP_SIZE and HEAP_NEWSIZE.

Duplicate of #114, #128. See also https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsTuneJVM.html

[longquanzheng]

Thanks, it works after I set MAX_HEAP_SIZE=512m

[supritshah1289]

Cassandra uses this formula to allocate memory to a Cassandra docker instance max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB) 8GB is replaced with total ram size of your server/laptop

So if you have other process comsuming memory and free memory is less than the output of the formula then cassandra docker container will exit with 137 without any logs.

In order to resolve this issue the docker container needs to be start by allocating minimum memory. Here is a command to set enviroment variables for alloating memory.

Command
docker run --name cass-test -e HEAP_NEWSIZE=600M -e MAX_HEAP_SIZE=1024M cassandra:3