Tuning for perfomance

Tuning the JVM

• -XX:+UseStringDeduplication, useful for Grok, other objects (events) might not live long enough to be useful. Check with -XX:+PrintStringDeduplicationStatistics, needs -XX:+UseG1GC and gc logging.
• -XX:+UseTransparentHugePages and -XX:+AlwaysPreTouch, see JVM Anatomy Park #2: Transparent Huge Pages
• -XX:+PrintStringTableStatistics to ensure that String.intern is not creating collisions.
• -Xloggc:.../gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps, for gc debugging. And then inspects with Universal GC Log Analyzer.
• -XX:AutoBoxCacheMax=20000, sets java.lang.Integer.IntegerCache.high, that may help the parsing of many strings to integer, and is used by snmpcodec.
• -XX:+AggressiveOpts: check usefulness with java -XX:+PrintFlagsFinal -XX:±AggressiveOpts

On openjdk, version "1.8.0_151":

	-AggressiveOpts	+AggressiveOpts
AutoBoxCacheMax	128	20000
BiasedLockingStartupDelay	4000	500

Tuning the system

Each event is processed in a single thread, and usually few IO are necessary, so memory locality is more important that disk caching. On Linux, vm.zone_reclaim_mode should be set to 2

High performance SSL

The Google library Conscrypt can be used to provide native code for TLS encryption. It's not provided, as it uses native code, and should be downloaded manually. Once done, one can add in the configuration file:

ssl.context: "TLSv1.2"
ssl.providerclass: "org.conscrypt.OpenSSLProvider"
plugins: [
...
"path/to/conscrypt-openjdk-....jar"
]

But when talking to a TLS 1.3 server, it needs its own TrustManager, with the non default algorithm PKIX. Otherwise, it fails with the message java.security.cert.CertificateException: Unknown authType: GENERIC. The bug status is found at Conscrypt's SSLContext fails when given a SunJSSE TrustManager.

So the configuration become:

ssl.context: "TLSv1.3"
ssl.providerclass: "org.conscrypt.OpenSSLProvider"
ssl.trustmanageralgorithm: "PKIX"

A more simple way to define the plugins path can also be:

plugins: "path/to/plugins"

All the jar in that folder will be used.

High performance UDP

As UDP is asynchronous, extra care must be taken to ensure that no packet will be dropped.

First step is to ensure that UDP receiver is correctly tuned:

loghub.receivers.Udp {
    poller: "EPOLL",
    rcvBuf: 1048576,
}

By setting a high value in rcvBuf, it reduce the possibility that packets are dropped during GC pauses.

One Linux, two sysctl settings can be tuned:

• net.core.rmem_max, it should be twice the value of rcvBuf.
• net.ipv4.udp_mem, it defines the maximum number of 4 KB pages than can be used by UDP. It's usually high enough, but if a lot of UDP listener are used, it must be checked.

The settings can be checked using the command ss -lumnep