Configure java -Xmx for max heap size

[gyuho]

While test VMs have enough RAM (30GB) for all test cases,
we should make it configurable for different machines

java -X
    -Xmixed           mixed mode execution (default)
    -Xint             interpreted mode execution only
    -Xbootclasspath:<directories and zip/jar files separated by :>
                      set search path for bootstrap classes and resources
    -Xbootclasspath/a:<directories and zip/jar files separated by :>
                      append to end of bootstrap class path
    -Xbootclasspath/p:<directories and zip/jar files separated by :>
                      prepend in front of bootstrap class path
    -Xdiag            show additional diagnostic messages
    -Xnoclassgc       disable class garbage collection
    -Xincgc           enable incremental garbage collection
    -Xloggc:<file>    log GC status to a file with time stamps
    -Xbatch           disable background compilation
    -Xms<size>        set initial Java heap size
    -Xmx<size>        set maximum Java heap size
    -Xss<size>        set java thread stack size
    -Xprof            output cpu profiling data
    -Xfuture          enable strictest checks, anticipating future default
    -Xrs              reduce use of OS signals by Java/VM (see documentation)
    -Xcheck:jni       perform additional checks for JNI functions
    -Xshare:off       do not attempt to use shared class data
    -Xshare:auto      use shared class data if possible (default)
    -Xshare:on        require using shared class data, otherwise fail.
    -XshowSettings    show all settings and continue
    -XshowSettings:all
                      show all settings and continue
    -XshowSettings:vm show all vm related settings and continue
    -XshowSettings:properties
                      show all property settings and continue
    -XshowSettings:locale
                      show all locale related settings and continue

The -X options are non-standard and subject to change without notice.

[gyuho]

Turns out it's not enforced...

-Xms, -Xmx does not limit the total amount of memory that the JVM can use.

There's no need to introduce this experimental flag and having to maintain it.
Most test cases (except 3-million writes) do not explode the memory usage...

So don't need this.

https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/optionX.html

[pmoust]

@gyuho please note (in the same docs)

The -Xmx option and -Xms option in combination are used to limit the Java heap size. The Java heap can never grow larger than -Xmx. Also, the -Xms value can be used as “minimum heap size” to set a fixed heap size by setting -Xms = -Xmx when, for example, you want to run benchmark tests.

By not sizing the heap size properly, you are setting up ZK for failure in your tests
Ref: #307

[xiang90]

@pmoust It is great to see you guys have a large zk deployment. Can you share us more about your workload? Like total number of keys, avg size of keys, avg write QPS, avg read QPS, avg Watch QPS? It would be super helpful for us to develop better benchmark test cases.

[gyuho]

@pmoust And what heap size would you set for our test case?

We were creating 1-million keys (1.3 GB), 3-million keys (4 GB).
java memory usage was 3.9GB and 7.4GB (when VM RAM is 30 GB).
Zookeeper logs do not show any java.lang.OutOfMemoryError.
All logs can be found at https://console.cloud.google.com/storage/browser/dbtester-results/2017Q1-00-etcd-zookeeper-consul/?project=etcd-development&authuser=0.

Thanks!

[pmoust]

Hey @xiang90 @gyuho just touching base here.

I'd give it a go with -Djute.maxbuffer=33554432 -Xms20G -Xmx20G according to my notes for a c3.4xlarge x 3 ensemble we maintain (which is also 30G per VM) - we have various ZK setups depending on region/load/expected growth.
This sustains without any strain about 750k znodes, 60k ephemeral znodes, 4m watches with 3.5k alive connections on a 3.5.1 set. Key sizes vary a lot. So does the avg QPS.

I will get back to you with our tests using dbstester by March 6th (pretty packed atm).

Would be wonderful if you could give it a go and let me know how this compares against latest etcd and consul compiled with Go 1.8

[gyuho]

@pmoust I will try that flags for next tests. Probably this week or next week (with latest Consul, etcd + go1.8, and Zookeeper alpha releases)

And if you want to run dbtester on your own and have any questions, you can email me

gyu_ho.lee@coreos.com

I am trying to make the config as easy as possible, but many parts aren't documented yet. Also I am planning to make some break changes in the config format as well to differentiate different versions of etcd, etc. Hope the config format gets stable this or early next week.

Thanks!

[gyuho]

Closing via #308.
Will close #309 when the new test results are ready.

[gyuho]

@pmoust Please check out https://github.com/coreos/dbtester/tree/master/test-results/2017Q1-01-etcd-zookeeper-consul for new results with jvm settings.

[pmoust]

Thanks @gyuho. We are already discussing your results internally - much appreciated!