Nodetool fails if heap size is specified

[Zerych]

Hello!


When I launch Cassandra container without any env variables,

docker run --name cass --rm cassandra:3.11.3

nodetool works fine.


However, if i specify JVM_OPTS

docker run --name cass --rm -e JVM_OPTS='-Xms5976M -Xmx5976M' cassandra:3.11.3

nodetool fails while executing any command with following error:

Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size

I'm not sure if Cassandra and Nodetool share same config, but in Cassandra startup logs I can see that JVM args are the same (the only difference is in order) for both cases:

Without specifying args:

INFO  [main] 2018-12-14 10:51:09,571 CassandraDaemon.java:490 - JVM Arguments: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms5976M, -Xmx5976M, -Xmn400M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes]

With specifying args:

INFO  [main] 2018-12-14 10:51:48,725 CassandraDaemon.java:490 - JVM Arguments: [-Xms5976M, -Xmx5976M, -Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xmn400M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes]

Do I do anything wrong while specifying JVM args?

Thanks!

[wglambert]

I don't know what would be causing this, setting the Xms to a smaller value gives the same error.

$ docker run --name cass -e JVM_OPTS="-Xms1000M -Xmx1024M" cassandra:3.11.3 nodetool status
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size

[Zerych]

I've checked nodetool script and it looks like a bug is hidden in

if [ "x$MAX_HEAP_SIZE" = "x" ]; then
    MAX_HEAP_SIZE="128m"
fi

"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE 
   ...

If I remove explicit setting of Xmx ($JVM_OPTS -Xmx$MAX_HEAP_SIZE ), then it works fine

[tianon]

Is there somewhere this is already filed upstream so they're aware of the issue? It doesn't look like something we can/should fix in the Docker image.

[tianon]

(Closing since I don't see anything we should fix at this layer -- I'd recommend following up with an issue upstream letting them know that blurb of nodetool that's causing the issue.)

[cnighojkar]

use this it will work
docker run -d --name cass --rm -e JAVA_OPTIONS='-Xms5976M -Xmx5976M' cassandra:3.11

use JAVA_OPTIONS instead of JVM_OPTS

[Jamim]

Hi @cnighojkar,
I just checked your suggestion and it doesn't work.
JAVA_OPTIONS has no effect.

[gkulkarni2020]

I also faced same problem while running nodetool when I was trying to start a 3 node cluster on my windows machine. But when I added another environment variable, it worked for me

-e HEAP_NEWSIZE=475M -e MAX_HEAP_SIZE=475M