Significant performance difference across JVMs on Mac OS X

[thyandrecardoso]

Elasticsearch Version

7.17.3

Installed Plugins

All default plugins that come installed in the docker distribution and on the osx brew tap

Java Version

bundled; OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09); OpenJDK Runtime Environment Temurin-17.0.3+7 (build 17.0.3+7); OpenJDK Runtime Environment Temurin-18.0.1+10 (build 18.0.1+10)

OS Version

OSX 12.3.1; Darwin wall-a 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:45:05 PDT 2022; root:xnu-8020.101.4~15/RELEASE_X86_64 x86_64

Problem Description

In OSX (specifically, on a Macbook Pro 2019), I am seeing a very significant difference in indexing performance across different JVMs.

It seems that anything other than Java 8 is quite slower when running directly on the host (vs a docker instance for example).

For the tests I conducted on that machine, I got the following number when trying to index 100 documents across 3 newly created indexes, sequentially:

• elasticsearch on a docker: ~6 seconds
• elasticsearch from brew tap, with java (Temurin)(build 1.8.0_332-b09): ~6 seconds
• elasticsearch from brew tap, with java Temurin-17.0.3+7: ~27 seconds
• elasticsearch from brew tap, with java Temurin-18.0.1+10: ~27 seconds
• elasticsearch from brew tap with bundled java ( 18 ? ): ~27 seconds

The particular numbers are not very important. I think that the magnitude of the difference is what matters.

I expected to see the same performance across every experiment, using any JVM. In particular, I expected that Elasticsearch installed through brew, running with a bundled JDK, would have the same (or slightly better) performance than the one running through a docker container.

Steps to Reproduce

I followed official instructions to install and run elasticsearch using brew and docker without any further customization.

For the brew one, I then ran it using different VMs by relying on JAVA_HOME and asdf (asdf-java) to change the Java version. Among others, I used specifically temurin-18.0.1+10, temurin-17.0.3+7 and temurin-8.0.332+9.

# to run with java 8
asdf shell java temurin-8.0.332+9
elasticsearch

# to run with java 17
asdf shell java temurin-17.0.3+7
elasticsearch

# to run with java 18
asdf shell java temurin-18.0.1+10
elasticsearch

# to run with bundled jdk
unset JAVA_HOME; elasticsearch

For testing, I threw some sequential indexing requests using something like:

port=9200
for indexIdx in {1..3}; do
  curl -sX DELETE "http://localhost:$port/test-index-temp-$indexIdx" >/dev/null
done
start_time=$(date +%s)
for indexIdx in {1..3}; do
  for d in {1..100}; do
    curl -sX POST "http://localhost:$port/test-index-temp-$indexIdx/_doc/" -H 'Content-Type: application/json' -d'{"user": {"id": "'"test-$d"'"}}' >/dev/null
  done
done
end_time=$(date +%s)
elapsed=$((end_time - start_time))
echo "Took $elapsed seconds."

Logs (if relevant)

I don't know exactly what logs are most relevant but, for now, I will paste the startup snippets showing the different JVMs being picked up.

Java 8

warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Future versions of Elasticsearch will require Java 11; your Java version from [/Users/thyandrecardoso/.asdf/installs/java/temurin-8.0.332+9/jre] does not meet this requirement. Consider switching to a distribution of Elasticsearch with a bundled JDK. If you are already using a distribution with a bundled JDK, ensure the JAVA_HOME environment variable is not set.
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Future versions of Elasticsearch will require Java 11; your Java version from [/Users/thyandrecardoso/.asdf/installs/java/temurin-8.0.332+9/jre] does not meet this requirement. Consider switching to a distribution of Elasticsearch with a bundled JDK. If you are already using a distribution with a bundled JDK, ensure the JAVA_HOME environment variable is not set.
[2022-05-10T11:11:58,406][INFO ][o.e.n.Node               ] [wall-a] version[7.17.3], pid[16589], build[default/tar/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z], OS[Mac OS X/10.16/x86_64], JVM[Temurin/OpenJDK 64-Bit Server VM/1.8.0_332/25.332-b09]
[2022-05-10T11:11:58,408][INFO ][o.e.n.Node               ] [wall-a] JVM home [/Users/thyandrecardoso/.asdf/installs/java/temurin-8.0.332+9/jre], using bundled JDK [false]
[2022-05-10T11:11:58,408][INFO ][o.e.n.Node               ] [wall-a] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,JRE, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/var/folders/hz/btcz28l9469116fcgjyyk5sm0000gn/T/elasticsearch-7060214858629589197, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/usr/local/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/local/Cellar/elasticsearch-full/7.17.3/libexec, -Des.path.conf=/usr/local/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-05-10T11:12:01,191][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [aggs-matrix-stats]
[2022-05-10T11:12:01,191][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [analysis-common]
[2022-05-10T11:12:01,191][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [constant-keyword]
[2022-05-10T11:12:01,192][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [frozen-indices]
[2022-05-10T11:12:01,192][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-common]
[2022-05-10T11:12:01,192][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-geoip]
[2022-05-10T11:12:01,192][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-user-agent]
[2022-05-10T11:12:01,192][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [kibana]
[2022-05-10T11:12:01,193][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-expression]
[2022-05-10T11:12:01,193][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-mustache]
[2022-05-10T11:12:01,193][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-painless]
[2022-05-10T11:12:01,193][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [legacy-geo]
[2022-05-10T11:12:01,194][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [mapper-extras]
[2022-05-10T11:12:01,194][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [mapper-version]
[2022-05-10T11:12:01,194][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [parent-join]
[2022-05-10T11:12:01,194][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [percolator]
[2022-05-10T11:12:01,194][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [rank-eval]
[2022-05-10T11:12:01,195][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [reindex]
[2022-05-10T11:12:01,195][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repositories-metering-api]
[2022-05-10T11:12:01,195][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repository-encrypted]
[2022-05-10T11:12:01,195][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repository-url]
[2022-05-10T11:12:01,196][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [runtime-fields-common]
[2022-05-10T11:12:01,196][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [search-business-rules]
[2022-05-10T11:12:01,196][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [searchable-snapshots]
[2022-05-10T11:12:01,196][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [snapshot-repo-test-kit]
[2022-05-10T11:12:01,196][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [spatial]
[2022-05-10T11:12:01,197][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [transform]
[2022-05-10T11:12:01,197][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [transport-netty4]
[2022-05-10T11:12:01,197][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [unsigned-long]
[2022-05-10T11:12:01,197][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [vector-tile]
[2022-05-10T11:12:01,197][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [vectors]
[2022-05-10T11:12:01,197][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [wildcard]
[2022-05-10T11:12:01,198][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-aggregate-metric]
[2022-05-10T11:12:01,198][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-analytics]
[2022-05-10T11:12:01,198][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-async]
[2022-05-10T11:12:01,198][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-async-search]
[2022-05-10T11:12:01,198][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-autoscaling]
[2022-05-10T11:12:01,199][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ccr]
[2022-05-10T11:12:01,199][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-core]
[2022-05-10T11:12:01,199][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-data-streams]
[2022-05-10T11:12:01,199][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-deprecation]
[2022-05-10T11:12:01,199][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-enrich]
[2022-05-10T11:12:01,199][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-eql]
[2022-05-10T11:12:01,199][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-fleet]
[2022-05-10T11:12:01,200][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-graph]
[2022-05-10T11:12:01,200][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-identity-provider]
[2022-05-10T11:12:01,200][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ilm]
[2022-05-10T11:12:01,200][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-logstash]
[2022-05-10T11:12:01,200][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ml]
[2022-05-10T11:12:01,200][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-monitoring]
[2022-05-10T11:12:01,201][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ql]
[2022-05-10T11:12:01,201][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-rollup]
[2022-05-10T11:12:01,201][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-security]
[2022-05-10T11:12:01,201][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-shutdown]
[2022-05-10T11:12:01,201][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-sql]
[2022-05-10T11:12:01,201][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-stack]
[2022-05-10T11:12:01,201][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-text-structure]
[2022-05-10T11:12:01,202][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-voting-only-node]
[2022-05-10T11:12:01,202][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-watcher]
[2022-05-10T11:12:01,202][INFO ][o.e.p.PluginsService     ] [wall-a] no plugins loaded
[2022-05-10T11:12:01,234][INFO ][o.e.e.NodeEnvironment    ] [wall-a] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s2)]], net usable_space [184.4gb], net total_space [465.6gb], types [apfs]
[2022-05-10T11:12:01,234][INFO ][o.e.e.NodeEnvironment    ] [wall-a] heap size [989.8mb], compressed ordinary object pointers [true]
[2022-05-10T11:12:01,329][INFO ][o.e.n.Node               ] [wall-a] node name [wall-a], node ID [vg6sBB6NQEivmkCmCuCWdA], cluster name [elasticsearch_thyandrecardoso], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-05-10T11:12:06,145][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [wall-a] [controller/16621] [Main.cc@122] controller (64 bit): Version 7.17.3 (Build b3cb07de236c11) Copyright (c) 2022 Elasticsearch BV
[2022-05-10T11:12:06,627][INFO ][o.e.x.s.a.s.FileRolesStore] [wall-a] parsed [0] roles from file [/usr/local/etc/elasticsearch/roles.yml]
[2022-05-10T11:12:07,521][INFO ][o.e.i.g.ConfigDatabases  ] [wall-a] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/local/etc/elasticsearch/ingest-geoip] for changes
[2022-05-10T11:12:07,523][INFO ][o.e.i.g.DatabaseNodeService] [wall-a] initialized database registry, using geoip-databases directory [/var/folders/hz/btcz28l9469116fcgjyyk5sm0000gn/T/elasticsearch-7060214858629589197/geoip-databases/vg6sBB6NQEivmkCmCuCWdA]
[2022-05-10T11:12:08,376][INFO ][o.e.t.NettyAllocator     ] [wall-a] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=false, g1gc_region_size=0b, heap_size=989.8mb}]
[2022-05-10T11:12:08,410][INFO ][o.e.i.r.RecoverySettings ] [wall-a] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-05-10T11:12:08,457][INFO ][o.e.d.DiscoveryModule    ] [wall-a] using discovery type [zen] and seed hosts providers [settings]
[2022-05-10T11:12:08,997][INFO ][o.e.g.DanglingIndicesState] [wall-a] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-05-10T11:12:09,564][INFO ][o.e.n.Node               ] [wall-a] initialized
[2022-05-10T11:12:09,565][INFO ][o.e.n.Node               ] [wall-a] starting ...

Java 17

warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
[2022-05-10T11:16:04,863][INFO ][o.e.n.Node               ] [wall-a] version[7.17.3], pid[17246], build[default/tar/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z], OS[Mac OS X/12.3.1/x86_64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.3/17.0.3+7]
[2022-05-10T11:16:04,869][INFO ][o.e.n.Node               ] [wall-a] JVM home [/Users/thyandrecardoso/.asdf/installs/java/temurin-17.0.3+7], using bundled JDK [false]
[2022-05-10T11:16:04,869][INFO ][o.e.n.Node               ] [wall-a] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/var/folders/hz/btcz28l9469116fcgjyyk5sm0000gn/T/elasticsearch-1387869772179848150, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/usr/local/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8192m, -Xmx8192m, -XX:MaxDirectMemorySize=4294967296, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/local/Cellar/elasticsearch-full/7.17.3/libexec, -Des.path.conf=/usr/local/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-05-10T11:16:07,459][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [aggs-matrix-stats]
[2022-05-10T11:16:07,459][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [analysis-common]
[2022-05-10T11:16:07,460][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [constant-keyword]
[2022-05-10T11:16:07,460][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [frozen-indices]
[2022-05-10T11:16:07,461][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-common]
[2022-05-10T11:16:07,461][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-geoip]
[2022-05-10T11:16:07,461][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-user-agent]
[2022-05-10T11:16:07,461][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [kibana]
[2022-05-10T11:16:07,461][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-expression]
[2022-05-10T11:16:07,462][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-mustache]
[2022-05-10T11:16:07,462][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-painless]
[2022-05-10T11:16:07,462][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [legacy-geo]
[2022-05-10T11:16:07,462][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [mapper-extras]
[2022-05-10T11:16:07,463][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [mapper-version]
[2022-05-10T11:16:07,463][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [parent-join]
[2022-05-10T11:16:07,463][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [percolator]
[2022-05-10T11:16:07,463][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [rank-eval]
[2022-05-10T11:16:07,463][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [reindex]
[2022-05-10T11:16:07,464][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repositories-metering-api]
[2022-05-10T11:16:07,464][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repository-encrypted]
[2022-05-10T11:16:07,464][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repository-url]
[2022-05-10T11:16:07,464][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [runtime-fields-common]
[2022-05-10T11:16:07,465][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [search-business-rules]
[2022-05-10T11:16:07,465][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [searchable-snapshots]
[2022-05-10T11:16:07,465][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [snapshot-repo-test-kit]
[2022-05-10T11:16:07,465][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [spatial]
[2022-05-10T11:16:07,465][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [transform]
[2022-05-10T11:16:07,466][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [transport-netty4]
[2022-05-10T11:16:07,466][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [unsigned-long]
[2022-05-10T11:16:07,466][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [vector-tile]
[2022-05-10T11:16:07,466][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [vectors]
[2022-05-10T11:16:07,467][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [wildcard]
[2022-05-10T11:16:07,467][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-aggregate-metric]
[2022-05-10T11:16:07,467][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-analytics]
[2022-05-10T11:16:07,467][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-async]
[2022-05-10T11:16:07,468][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-async-search]
[2022-05-10T11:16:07,468][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-autoscaling]
[2022-05-10T11:16:07,469][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ccr]
[2022-05-10T11:16:07,471][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-core]
[2022-05-10T11:16:07,471][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-data-streams]
[2022-05-10T11:16:07,471][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-deprecation]
[2022-05-10T11:16:07,472][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-enrich]
[2022-05-10T11:16:07,472][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-eql]
[2022-05-10T11:16:07,472][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-fleet]
[2022-05-10T11:16:07,472][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-graph]
[2022-05-10T11:16:07,472][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-identity-provider]
[2022-05-10T11:16:07,473][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ilm]
[2022-05-10T11:16:07,473][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-logstash]
[2022-05-10T11:16:07,473][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ml]
[2022-05-10T11:16:07,474][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-monitoring]
[2022-05-10T11:16:07,474][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ql]
[2022-05-10T11:16:07,474][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-rollup]
[2022-05-10T11:16:07,474][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-security]
[2022-05-10T11:16:07,475][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-shutdown]
[2022-05-10T11:16:07,475][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-sql]
[2022-05-10T11:16:07,475][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-stack]
[2022-05-10T11:16:07,475][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-text-structure]
[2022-05-10T11:16:07,475][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-voting-only-node]
[2022-05-10T11:16:07,476][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-watcher]
[2022-05-10T11:16:07,476][INFO ][o.e.p.PluginsService     ] [wall-a] no plugins loaded
[2022-05-10T11:16:07,508][INFO ][o.e.e.NodeEnvironment    ] [wall-a] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s2)]], net usable_space [184.3gb], net total_space [465.6gb], types [apfs]
[2022-05-10T11:16:07,509][INFO ][o.e.e.NodeEnvironment    ] [wall-a] heap size [8gb], compressed ordinary object pointers [true]
[2022-05-10T11:16:07,585][INFO ][o.e.n.Node               ] [wall-a] node name [wall-a], node ID [vg6sBB6NQEivmkCmCuCWdA], cluster name [elasticsearch_thyandrecardoso], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-05-10T11:16:13,111][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [wall-a] [controller/17283] [Main.cc@122] controller (64 bit): Version 7.17.3 (Build b3cb07de236c11) Copyright (c) 2022 Elasticsearch BV
[2022-05-10T11:16:13,522][INFO ][o.e.x.s.a.s.FileRolesStore] [wall-a] parsed [0] roles from file [/usr/local/etc/elasticsearch/roles.yml]
[2022-05-10T11:16:14,340][INFO ][o.e.i.g.ConfigDatabases  ] [wall-a] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/local/etc/elasticsearch/ingest-geoip] for changes
[2022-05-10T11:16:14,341][INFO ][o.e.i.g.DatabaseNodeService] [wall-a] initialized database registry, using geoip-databases directory [/var/folders/hz/btcz28l9469116fcgjyyk5sm0000gn/T/elasticsearch-1387869772179848150/geoip-databases/vg6sBB6NQEivmkCmCuCWdA]
[2022-05-10T11:16:15,262][INFO ][o.e.t.NettyAllocator     ] [wall-a] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-05-10T11:16:15,297][INFO ][o.e.i.r.RecoverySettings ] [wall-a] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-05-10T11:16:15,340][INFO ][o.e.d.DiscoveryModule    ] [wall-a] using discovery type [zen] and seed hosts providers [settings]
[2022-05-10T11:16:15,759][INFO ][o.e.g.DanglingIndicesState] [wall-a] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-05-10T11:16:16,250][INFO ][o.e.n.Node               ] [wall-a] initialized
[2022-05-10T11:16:16,250][INFO ][o.e.n.Node               ] [wall-a] starting ...

Java 18

warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
[2022-05-10T12:17:21,748][INFO ][o.e.n.Node               ] [wall-a] version[7.17.3], pid[25924], build[default/tar/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z], OS[Mac OS X/12.3.1/x86_64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/18.0.1/18.0.1+10]
[2022-05-10T12:17:21,752][INFO ][o.e.n.Node               ] [wall-a] JVM home [/Users/thyandrecardoso/.asdf/installs/java/temurin-18.0.1+10], using bundled JDK [false]
[2022-05-10T12:17:21,752][INFO ][o.e.n.Node               ] [wall-a] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/var/folders/hz/btcz28l9469116fcgjyyk5sm0000gn/T/elasticsearch-1743142841912937309, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/usr/local/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8192m, -Xmx8192m, -XX:MaxDirectMemorySize=4294967296, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/local/Cellar/elasticsearch-full/7.17.3/libexec, -Des.path.conf=/usr/local/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-05-10T12:17:23,749][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [aggs-matrix-stats]
[2022-05-10T12:17:23,750][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [analysis-common]
[2022-05-10T12:17:23,750][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [constant-keyword]
[2022-05-10T12:17:23,750][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [frozen-indices]
[2022-05-10T12:17:23,751][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-common]
[2022-05-10T12:17:23,751][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-geoip]
[2022-05-10T12:17:23,751][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-user-agent]
[2022-05-10T12:17:23,751][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [kibana]
[2022-05-10T12:17:23,751][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-expression]
[2022-05-10T12:17:23,752][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-mustache]
[2022-05-10T12:17:23,752][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-painless]
[2022-05-10T12:17:23,752][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [legacy-geo]
[2022-05-10T12:17:23,752][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [mapper-extras]
[2022-05-10T12:17:23,753][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [mapper-version]
[2022-05-10T12:17:23,753][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [parent-join]
[2022-05-10T12:17:23,753][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [percolator]
[2022-05-10T12:17:23,753][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [rank-eval]
[2022-05-10T12:17:23,753][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [reindex]
[2022-05-10T12:17:23,753][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repositories-metering-api]
[2022-05-10T12:17:23,754][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repository-encrypted]
[2022-05-10T12:17:23,754][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repository-url]
[2022-05-10T12:17:23,754][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [runtime-fields-common]
[2022-05-10T12:17:23,754][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [search-business-rules]
[2022-05-10T12:17:23,754][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [searchable-snapshots]
[2022-05-10T12:17:23,754][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [snapshot-repo-test-kit]
[2022-05-10T12:17:23,755][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [spatial]
[2022-05-10T12:17:23,755][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [transform]
[2022-05-10T12:17:23,755][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [transport-netty4]
[2022-05-10T12:17:23,755][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [unsigned-long]
[2022-05-10T12:17:23,755][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [vector-tile]
[2022-05-10T12:17:23,755][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [vectors]
[2022-05-10T12:17:23,756][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [wildcard]
[2022-05-10T12:17:23,756][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-aggregate-metric]
[2022-05-10T12:17:23,756][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-analytics]
[2022-05-10T12:17:23,756][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-async]
[2022-05-10T12:17:23,756][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-async-search]
[2022-05-10T12:17:23,756][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-autoscaling]
[2022-05-10T12:17:23,756][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ccr]
[2022-05-10T12:17:23,757][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-core]
[2022-05-10T12:17:23,757][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-data-streams]
[2022-05-10T12:17:23,757][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-deprecation]
[2022-05-10T12:17:23,757][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-enrich]
[2022-05-10T12:17:23,757][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-eql]
[2022-05-10T12:17:23,757][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-fleet]
[2022-05-10T12:17:23,758][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-graph]
[2022-05-10T12:17:23,758][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-identity-provider]
[2022-05-10T12:17:23,758][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ilm]
[2022-05-10T12:17:23,758][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-logstash]
[2022-05-10T12:17:23,758][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ml]
[2022-05-10T12:17:23,758][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-monitoring]
[2022-05-10T12:17:23,758][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ql]
[2022-05-10T12:17:23,759][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-rollup]
[2022-05-10T12:17:23,759][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-security]
[2022-05-10T12:17:23,759][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-shutdown]
[2022-05-10T12:17:23,759][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-sql]
[2022-05-10T12:17:23,759][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-stack]
[2022-05-10T12:17:23,759][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-text-structure]
[2022-05-10T12:17:23,760][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-voting-only-node]
[2022-05-10T12:17:23,760][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-watcher]
[2022-05-10T12:17:23,760][INFO ][o.e.p.PluginsService     ] [wall-a] no plugins loaded
[2022-05-10T12:17:23,784][INFO ][o.e.e.NodeEnvironment    ] [wall-a] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s2)]], net usable_space [181.6gb], net total_space [465.6gb], types [apfs]
[2022-05-10T12:17:23,784][INFO ][o.e.e.NodeEnvironment    ] [wall-a] heap size [8gb], compressed ordinary object pointers [true]
[2022-05-10T12:17:23,838][INFO ][o.e.n.Node               ] [wall-a] node name [wall-a], node ID [vg6sBB6NQEivmkCmCuCWdA], cluster name [elasticsearch_thyandrecardoso], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-05-10T12:17:27,484][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [wall-a] [controller/25956] [Main.cc@122] controller (64 bit): Version 7.17.3 (Build b3cb07de236c11) Copyright (c) 2022 Elasticsearch BV
[2022-05-10T12:17:28,073][INFO ][o.e.x.s.a.s.FileRolesStore] [wall-a] parsed [0] roles from file [/usr/local/etc/elasticsearch/roles.yml]
[2022-05-10T12:17:28,562][INFO ][o.e.i.g.ConfigDatabases  ] [wall-a] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/local/etc/elasticsearch/ingest-geoip] for changes
[2022-05-10T12:17:28,563][INFO ][o.e.i.g.DatabaseNodeService] [wall-a] initialized database registry, using geoip-databases directory [/var/folders/hz/btcz28l9469116fcgjyyk5sm0000gn/T/elasticsearch-1743142841912937309/geoip-databases/vg6sBB6NQEivmkCmCuCWdA]
[2022-05-10T12:17:29,173][INFO ][o.e.t.NettyAllocator     ] [wall-a] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-05-10T12:17:29,200][INFO ][o.e.i.r.RecoverySettings ] [wall-a] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-05-10T12:17:29,236][INFO ][o.e.d.DiscoveryModule    ] [wall-a] using discovery type [zen] and seed hosts providers [settings]
[2022-05-10T12:17:29,641][INFO ][o.e.g.DanglingIndicesState] [wall-a] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-05-10T12:17:30,212][INFO ][o.e.n.Node               ] [wall-a] initialized
[2022-05-10T12:17:30,212][INFO ][o.e.n.Node               ] [wall-a] starting ...

Bundled JDK

[2022-05-10T11:17:25,327][INFO ][o.e.n.Node               ] [wall-a] version[7.17.3], pid[17395], build[default/tar/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z], OS[Mac OS X/12.3.1/x86_64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/18/18+36]
[2022-05-10T11:17:25,332][INFO ][o.e.n.Node               ] [wall-a] JVM home [/usr/local/Cellar/elasticsearch-full/7.17.3/libexec/jdk.app/Contents/Home], using bundled JDK [true]
[2022-05-10T11:17:25,333][INFO ][o.e.n.Node               ] [wall-a] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/var/folders/hz/btcz28l9469116fcgjyyk5sm0000gn/T/elasticsearch-9701995722596221625, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/usr/local/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8192m, -Xmx8192m, -XX:MaxDirectMemorySize=4294967296, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/local/Cellar/elasticsearch-full/7.17.3/libexec, -Des.path.conf=/usr/local/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-05-10T11:17:27,767][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [aggs-matrix-stats]
[2022-05-10T11:17:27,767][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [analysis-common]
[2022-05-10T11:17:27,767][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [constant-keyword]
[2022-05-10T11:17:27,768][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [frozen-indices]
[2022-05-10T11:17:27,768][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-common]
[2022-05-10T11:17:27,768][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-geoip]
[2022-05-10T11:17:27,769][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [ingest-user-agent]
[2022-05-10T11:17:27,769][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [kibana]
[2022-05-10T11:17:27,769][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-expression]
[2022-05-10T11:17:27,769][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-mustache]
[2022-05-10T11:17:27,770][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [lang-painless]
[2022-05-10T11:17:27,770][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [legacy-geo]
[2022-05-10T11:17:27,770][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [mapper-extras]
[2022-05-10T11:17:27,770][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [mapper-version]
[2022-05-10T11:17:27,770][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [parent-join]
[2022-05-10T11:17:27,771][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [percolator]
[2022-05-10T11:17:27,771][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [rank-eval]
[2022-05-10T11:17:27,771][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [reindex]
[2022-05-10T11:17:27,771][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repositories-metering-api]
[2022-05-10T11:17:27,772][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repository-encrypted]
[2022-05-10T11:17:27,772][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [repository-url]
[2022-05-10T11:17:27,772][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [runtime-fields-common]
[2022-05-10T11:17:27,772][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [search-business-rules]
[2022-05-10T11:17:27,773][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [searchable-snapshots]
[2022-05-10T11:17:27,773][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [snapshot-repo-test-kit]
[2022-05-10T11:17:27,773][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [spatial]
[2022-05-10T11:17:27,773][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [transform]
[2022-05-10T11:17:27,773][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [transport-netty4]
[2022-05-10T11:17:27,774][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [unsigned-long]
[2022-05-10T11:17:27,774][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [vector-tile]
[2022-05-10T11:17:27,774][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [vectors]
[2022-05-10T11:17:27,774][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [wildcard]
[2022-05-10T11:17:27,774][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-aggregate-metric]
[2022-05-10T11:17:27,775][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-analytics]
[2022-05-10T11:17:27,775][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-async]
[2022-05-10T11:17:27,775][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-async-search]
[2022-05-10T11:17:27,775][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-autoscaling]
[2022-05-10T11:17:27,775][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ccr]
[2022-05-10T11:17:27,775][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-core]
[2022-05-10T11:17:27,776][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-data-streams]
[2022-05-10T11:17:27,776][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-deprecation]
[2022-05-10T11:17:27,776][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-enrich]
[2022-05-10T11:17:27,776][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-eql]
[2022-05-10T11:17:27,776][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-fleet]
[2022-05-10T11:17:27,777][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-graph]
[2022-05-10T11:17:27,777][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-identity-provider]
[2022-05-10T11:17:27,777][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ilm]
[2022-05-10T11:17:27,777][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-logstash]
[2022-05-10T11:17:27,777][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ml]
[2022-05-10T11:17:27,778][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-monitoring]
[2022-05-10T11:17:27,778][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-ql]
[2022-05-10T11:17:27,778][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-rollup]
[2022-05-10T11:17:27,778][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-security]
[2022-05-10T11:17:27,778][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-shutdown]
[2022-05-10T11:17:27,779][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-sql]
[2022-05-10T11:17:27,779][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-stack]
[2022-05-10T11:17:27,779][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-text-structure]
[2022-05-10T11:17:27,779][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-voting-only-node]
[2022-05-10T11:17:27,779][INFO ][o.e.p.PluginsService     ] [wall-a] loaded module [x-pack-watcher]
[2022-05-10T11:17:27,780][INFO ][o.e.p.PluginsService     ] [wall-a] no plugins loaded
[2022-05-10T11:17:27,814][INFO ][o.e.e.NodeEnvironment    ] [wall-a] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s2)]], net usable_space [182.8gb], net total_space [465.6gb], types [apfs]
[2022-05-10T11:17:27,815][INFO ][o.e.e.NodeEnvironment    ] [wall-a] heap size [8gb], compressed ordinary object pointers [true]
[2022-05-10T11:17:27,881][INFO ][o.e.n.Node               ] [wall-a] node name [wall-a], node ID [vg6sBB6NQEivmkCmCuCWdA], cluster name [elasticsearch_thyandrecardoso], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-05-10T11:17:31,593][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [wall-a] [controller/17431] [Main.cc@122] controller (64 bit): Version 7.17.3 (Build b3cb07de236c11) Copyright (c) 2022 Elasticsearch BV
[2022-05-10T11:17:32,081][INFO ][o.e.x.s.a.s.FileRolesStore] [wall-a] parsed [0] roles from file [/usr/local/etc/elasticsearch/roles.yml]
[2022-05-10T11:17:32,676][INFO ][o.e.i.g.ConfigDatabases  ] [wall-a] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/local/etc/elasticsearch/ingest-geoip] for changes
[2022-05-10T11:17:32,677][INFO ][o.e.i.g.DatabaseNodeService] [wall-a] initialized database registry, using geoip-databases directory [/var/folders/hz/btcz28l9469116fcgjyyk5sm0000gn/T/elasticsearch-9701995722596221625/geoip-databases/vg6sBB6NQEivmkCmCuCWdA]
[2022-05-10T11:17:33,315][INFO ][o.e.t.NettyAllocator     ] [wall-a] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-05-10T11:17:33,345][INFO ][o.e.i.r.RecoverySettings ] [wall-a] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-05-10T11:17:33,387][INFO ][o.e.d.DiscoveryModule    ] [wall-a] using discovery type [zen] and seed hosts providers [settings]
[2022-05-10T11:17:33,820][INFO ][o.e.g.DanglingIndicesState] [wall-a] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-05-10T11:17:34,420][INFO ][o.e.n.Node               ] [wall-a] initialized
[2022-05-10T11:17:34,420][INFO ][o.e.n.Node               ] [wall-a] starting ...

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticmachine]

Pinging @elastic/es-perf (Team:Performance)

[mark-vieira]

So this is actually due to the fact that fsync performance on modern Macs is very poor as described in this Twitter thread. The reason we don't see the performance issue in Java 8 is because in later versions (starting with 9), the JDK was fixed to do a proper real fsync on MacOS.

I actually did some experimentation with https://github.com/stewartsmith/libeatmydata locally to try and disable fsync but with limited success. It seemed like the JVM wasn't using my dynamic library. Perhaps someone better versed in native code would have a better chance here. I did confirm though that this is an IO problem as even with modern JDKs on the latest Macs, the performance drop disappears if you run Elasticsearch backed by a ramdisk.

[thyandrecardoso]

So this is actually due to the fact that fsync performance on modern Macs is very poor as described in this Twitter thread. The reason we don't see the performance issue in Java 8 is because in later versions (starting with 9), the JDK was fixed to do a proper real fsync on MacOS.

That seems to make sense. But I am a bit confused with the docker results (which are on par with JVM 8). Filesystem access in docker is virtualized or something and that layer does not implement "proper" fsync?

[mark-vieira]

Filesystem access in docker is virtualized or something and that layer does not implement "proper" fsync?

Yep, basically this. Depending on the underlying fileystem driver being used, more than likely a "proper" fsync is not happening. To the JVM, since we're running in a Docker container, the host is Linux, so it's just doing a normal fsync, when when finally translated to the MacOS filesystem is not doing a real fsync, thus the perceived better performance.

[mark-vieira]

I'm closing this as it's not actually an issue with Elasticsearch, but a fundamental issue with running any system that does real fsyncon modern Macs.