org.apache.lucene.util.MergedIterator.pullTop() Crash #31425

WUMUXIAN · 2018-06-19T10:27:08Z

Elasticsearch version (bin/elasticsearch --version):
6.3.0

Plugins installed: [ingest-geoip, ingest-user-agent]

JVM version (java -version):
openjdk version "10.0.1" 2018-04-17
OpenJDK Runtime Environment (build 10.0.1+10)
OpenJDK 64-Bit Server VM (build 10.0.1+10, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux es-data-us-east-1c-0 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I am running a elasticsearch data node using this docker image:

docker.elastic.co/elasticsearch/elasticsearch:6.3.0

Somehow it always crashes after running for ~5 mins with error like this:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fc7ebc050eb, pid=1, tid=151
#
# JRE version: OpenJDK Runtime Environment (10.0.1+10) (build 10.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM (10.0.1+10, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# J 14088 c2 org.apache.lucene.util.MergedIterator.pullTop()V (135 bytes) @ 0x00007fc7ebc050eb [0x00007fc7ebc041e0+0x0000000000000f0b]
#
# Core dump will be written. Default location: /usr/share/elasticsearch/core
#
# An error report file with more information is saved as:
# logs/hs_err_pid1.log
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]

[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

Deploy 3 master nodes
Deploy 2 data nodes
One of the data nodes always crashes with the above error.

The text was updated successfully, but these errors were encountered:

vladimirdolzhenko · 2018-06-19T11:01:44Z

@elastic/es-core-infra could you pls have a look ?

ywelsch · 2018-06-19T12:38:11Z

@WUMUXIAN can you share more information on the machine type that you're using, in particular the output of cat /proc/cpuinfo?

WUMUXIAN · 2018-06-20T02:15:22Z

@yevhen Here you go:

cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

A few more information:

I ran two docker containers with the same images on the same OS (m5.large AWS EC2 instances, Debian 1.9). But on one of the machine, it crashes always, on another one it does not.
I updated the -Xms and -Xmx to 2g for the one that always crashes, it still crashed once, but somehow it stopped crashing and is up and running since 12 hours ago.

I am not sure why this is the behaviour, very strange to me, hope it can help you.

Let me know if you need more information.

jpountz · 2018-06-20T09:04:40Z

Thank you @WUMUXIAN. Can you also share the content of logs/hs_err_pid1.log?

WUMUXIAN · 2018-06-20T10:23:48Z

@jpountz Sure, see it below:

cat elasticsearch/logs/hs_err_pid1.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8c43de5371, pid=1, tid=140
#
# JRE version: OpenJDK Runtime Environment (10.0.1+10) (build 10.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM (10.0.1+10, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# J 13039 c2 org.apache.lucene.util.MergedIterator.pushTop()V (92 bytes) @ 0x00007f8c43de5371 [0x00007f8c43de4c40+0x0000000000000731]
#
# Core dump will be written. Default location: /usr/share/elasticsearch/core
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.9Ts2lgAF -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Des.cgroups.hierarchy.override=/ -Xms2g -Xmx2g -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=tar org.elasticsearch.bootstrap.Elasticsearch -Ecluster.name=tds-es-db -Expack.watcher.enabled=false -Expack.ml.enabled=false -Expack.logstash.enabled=false -Ediscovery.zen.ping.unicast.hosts=elasticsearch-discovery -Ehttp.enabled=false -Epath.data=/data -Enode.data=true -Enode.master=false -Enode.name=es-data-us-east-1c-0 -Enetwork.host=_eth0:ipv4_ -Expack.graph.enabled=false -Ediscovery.zen.minimum_master_nodes=2 -Expack.security.enabled=false -Enode.ingest=false

Host: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz, 2 cores, 7G, CentOS Linux release 7.5.1804 (Core)
Time: Tue Jun 19 10:15:27 2018 UTC elapsed time: 247 seconds (0d 0h 4m 7s)

---------------  T H R E A D  ---------------

Current thread (0x00007f8c18125000):  JavaThread "elasticsearch[es-data-us-east-1c-0][[logstash-2018.06.19][2]: Lucene Merge Thread #8]" daemon [_thread_in_Java, id=140, stack(0x00007f8bf12c1000,0x00007f8bf13c2000)]

Stack: [0x00007f8bf12c1000,0x00007f8bf13c2000],  sp=0x00007f8bf13c01e8,  free space=1020k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 13039 c2 org.apache.lucene.util.MergedIterator.pushTop()V (92 bytes) @ 0x00007f8c43de5371 [0x00007f8c43de4c40+0x0000000000000731]

[error occurred during error reporting (printing native stack), id 0xb]


siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000007fffffc3

Register to memory mapping:

RAX=0xffffffffffffffe0 is an unknown value
RBX=0x000000008015d6f8 is an oop
org.apache.lucene.util.MergedIterator
{0x000000008015d6f8} - klass: 'org/apache/lucene/util/MergedIterator'
RCX=0x0000000000000005 is an unknown value
RDX=0x0000000000000020 is an unknown value
RSP=0x00007f8bf13c01e8 is pointing into the stack for thread: 0x00007f8c18125000
RBP=0x0000000000000023 is an unknown value
RSI=0x000000007fffffe3 is an unknown value
RDI=0x000000008f2bee53 is pointing into object: 0x000000008f2bee40
org.apache.lucene.analysis.tokenattributes.BytesTermAttributeImpl
{0x000000008f2bee40} - klass: 'org/apache/lucene/analysis/tokenattributes/BytesTermAttributeImpl'
R8 =0x0000000093598e60 is an oop
[B
{0x0000000093598e60} - klass: {type array byte}
 - length: 35
R9 =0x0000000000000000 is an unknown value
R10=0x00000000842d9ff0 is an oop
[B
{0x00000000842d9ff0} - klass: {type array byte}
 - length: 42
R11=0x00000000842d9ff0 is an oop
[B
{0x00000000842d9ff0} - klass: {type array byte}
 - length: 42
R12=0x0000000000000000 is an unknown value
R13=0x000000000000000b is an unknown value
R14=0x000000008015d7c0 is an oop
org.apache.lucene.util.MergedIterator$SubIterator
{0x000000008015d7c0} - klass: 'org/apache/lucene/util/MergedIterator$SubIterator'
R15=0x00007f8c18125000 is a thread


Registers:
RAX=0xffffffffffffffe0, RBX=0x000000008015d6f8, RCX=0x0000000000000005, RDX=0x0000000000000020
RSP=0x00007f8bf13c01e8, RBP=0x0000000000000023, RSI=0x000000007fffffe3, RDI=0x000000008f2bee53
R8 =0x0000000093598e60, R9 =0x0000000000000000, R10=0x00000000842d9ff0, R11=0x00000000842d9ff0
R12=0x0000000000000000, R13=0x000000000000000b, R14=0x000000008015d7c0, R15=0x00007f8c18125000
RIP=0x00007f8c43de5371, EFLAGS=0x0000000000010283, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f8bf13c01e8)
0x00007f8bf13c01e8:   00000000fffffff9 8015d7a800000000
0x00007f8bf13c01f8:   0000000000000002 000000008015d6f8
0x00007f8bf13c0208:   000000008015d6f8 00007f8bf13c0250
0x00007f8bf13c0218:   00007f8bf13c0240 00007f8c18125000

Instructions: (pc=0x00007f8c43de5371)
0x00007f8c43de5351:   98 ff 0f 83 14 01 00 00 48 83 c0 40 83 ea 40 75
0x00007f8c43de5361:   de 62 a1 7d a1 ef c0 eb 64 62 e1 7e a9 6f 04 07
0x00007f8c43de5371:   62 e1 7d a1 ef 04 06 48 83 ec 40 62 f1 7e 49 7f
0x00007f8c43de5381:   04 24 48 83 ec 40 62 f1 7e 49 7f 0c 24 62 b1 7e



---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x00007f8c20004670, length=38, elements={
0x00007f8c540c7000, 0x00007f8c540c9000, 0x00007f8c540e0800, 0x00007f8c540e2800,
0x00007f8c540e4800, 0x00007f8c540e6800, 0x00007f8c54157800, 0x00007f8c54169800,
0x00007f8c54729800, 0x00007f8c55090000, 0x00007f8c55687800, 0x00007f8c55cf6000,
0x00007f8c55d17800, 0x00007f8c55d1f800, 0x00007f8c08008000, 0x00007f8c10013800,
0x00007f8c14928000, 0x00007f8c1c005800, 0x00007f8c143eb800, 0x00007f8c20003800,
0x00007f8c0800c800, 0x00007f8bfc014800, 0x00007f8c55d32800, 0x00007f8bfc02f800,
0x00007f8c04002000, 0x00007f8c04003800, 0x00007f8c04005800, 0x00007f8c04083000,
0x00007f8c54a81000, 0x00007f8c54012000, 0x00007f8c0000e800, 0x00007f8c55974800,
0x00007f8bfc00e000, 0x00007f8c549bf000, 0x00007f8c554ae800, 0x00007f8c549c0000,
0x00007f8c24244800, 0x00007f8c18125000
}

Java Threads: ( => current thread )
  0x00007f8c540c7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=52, stack(0x00007f8c58dab000,0x00007f8c58eac000)]
  0x00007f8c540c9000 JavaThread "Finalizer" daemon [_thread_blocked, id=53, stack(0x00007f8c58caa000,0x00007f8c58dab000)]
  0x00007f8c540e0800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=54, stack(0x00007f8c58ba9000,0x00007f8c58caa000)]
  0x00007f8c540e2800 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=55, stack(0x00007f8c58aa8000,0x00007f8c58ba9000)]
  0x00007f8c540e4800 JavaThread "C1 CompilerThread1" daemon [_thread_blocked, id=56, stack(0x00007f8c589a7000,0x00007f8c58aa8000)]
  0x00007f8c540e6800 JavaThread "Sweeper thread" daemon [_thread_blocked, id=57, stack(0x00007f8c588a6000,0x00007f8c589a7000)]
  0x00007f8c54157800 JavaThread "Service Thread" daemon [_thread_blocked, id=58, stack(0x00007f8c587a5000,0x00007f8c588a6000)]
  0x00007f8c54169800 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=60, stack(0x00007f8c585a2000,0x00007f8c586a3000)]
  0x00007f8c54729800 JavaThread "process reaper" daemon [_thread_in_native, id=62, stack(0x00007f8c58158000,0x00007f8c5817a000)]
  0x00007f8c55090000 JavaThread "elasticsearch[es-data-us-east-1c-0][[timer]]" daemon [_thread_blocked, id=66, stack(0x00007f8c58035000,0x00007f8c58136000)]
  0x00007f8c55687800 JavaThread "elasticsearch[es-data-us-east-1c-0][scheduler][T#1]" daemon [_thread_blocked, id=67, stack(0x00007f8c39319000,0x00007f8c3941a000)]
  0x00007f8c55cf6000 JavaThread "elasticsearch[es-data-us-east-1c-0][[transport_server_worker.default]][T#1]" daemon [_thread_in_native, id=68, stack(0x00007f8c300ff000,0x00007f8c30200000)]
  0x00007f8c55d17800 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#1]" daemon [_thread_blocked, id=69, stack(0x00007f8bfbeff000,0x00007f8bfc000000)]
  0x00007f8c55d1f800 JavaThread "elasticsearch[es-data-us-east-1c-0][clusterApplierService#updateTask][T#1]" daemon [_thread_blocked, id=70, stack(0x00007f8bfbdfe000,0x00007f8bfbeff000)]
  0x00007f8c08008000 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#2]" daemon [_thread_blocked, id=72, stack(0x00007f8bfb7dc000,0x00007f8bfb8dd000)]
  0x00007f8c10013800 JavaThread "elasticsearch[es-data-us-east-1c-0][transport_client_boss][T#1]" daemon [_thread_in_native, id=73, stack(0x00007f8bfb6db000,0x00007f8bfb7dc000)]
  0x00007f8c14928000 JavaThread "elasticsearch[es-data-us-east-1c-0][transport_client_boss][T#2]" daemon [_thread_in_native, id=76, stack(0x00007f8bfb3d8000,0x00007f8bfb4d9000)]
  0x00007f8c1c005800 JavaThread "threadDeathWatcher-2-1" daemon [_thread_blocked, id=77, stack(0x00007f8bfb2d7000,0x00007f8bfb3d8000)]
  0x00007f8c143eb800 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#3]" daemon [_thread_blocked, id=78, stack(0x00007f8bf91d4000,0x00007f8bf92d5000)]
  0x00007f8c20003800 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#4]" daemon [_thread_blocked, id=79, stack(0x00007f8bf90d3000,0x00007f8bf91d4000)]
  0x00007f8c0800c800 JavaThread "elasticsearch[es-data-us-east-1c-0][masterService#updateTask][T#1]" daemon [_thread_blocked, id=80, stack(0x00007f8bf8fd2000,0x00007f8bf90d3000)]
  0x00007f8bfc014800 JavaThread "elasticsearch[es-data-us-east-1c-0][[transport_server_worker.default]][T#2]" daemon [_thread_in_native, id=81, stack(0x00007f8bf8cd1000,0x00007f8bf8dd2000)]
  0x00007f8c55d32800 JavaThread "elasticsearch[es-data-us-east-1c-0][fetch_shard_store][T#1]" daemon [_thread_blocked, id=82, stack(0x00007f8bf8bd0000,0x00007f8bf8cd1000)]
  0x00007f8bfc02f800 JavaThread "elasticsearch[es-data-us-east-1c-0][fetch_shard_store][T#2]" daemon [_thread_blocked, id=83, stack(0x00007f8bf8acf000,0x00007f8bf8bd0000)]
  0x00007f8c04002000 JavaThread "elasticsearch[es-data-us-east-1c-0][management][T#1]" daemon [_thread_blocked, id=84, stack(0x00007f8bf89ce000,0x00007f8bf8acf000)]
  0x00007f8c04003800 JavaThread "elasticsearch[es-data-us-east-1c-0][management][T#2]" daemon [_thread_blocked, id=85, stack(0x00007f8bf88cd000,0x00007f8bf89ce000)]
  0x00007f8c04005800 JavaThread "elasticsearch[es-data-us-east-1c-0][management][T#3]" daemon [_thread_blocked, id=86, stack(0x00007f8bf87cc000,0x00007f8bf88cd000)]
  0x00007f8c04083000 JavaThread "trigger_engine_scheduler[T#1]" daemon [_thread_blocked, id=87, stack(0x00007f8bf84cb000,0x00007f8bf85cc000)]
  0x00007f8c54a81000 JavaThread "elasticsearch[keepAlive/6.3.0]" [_thread_blocked, id=88, stack(0x00007f8bf83ca000,0x00007f8bf84cb000)]
  0x00007f8c54012000 JavaThread "DestroyJavaVM" [_thread_blocked, id=48, stack(0x00007f8c5d669000,0x00007f8c5d76a000)]
  0x00007f8c0000e800 JavaThread "elasticsearch[es-data-us-east-1c-0][refresh][T#1]" daemon [_thread_blocked, id=89, stack(0x00007f8bf80c9000,0x00007f8bf81ca000)]
  0x00007f8c55974800 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#5]" daemon [_thread_blocked, id=90, stack(0x00007f8bf7fc8000,0x00007f8bf80c9000)]
  0x00007f8bfc00e000 JavaThread "elasticsearch[es-data-us-east-1c-0][write][T#1]" daemon [_thread_blocked, id=91, stack(0x00007f8bf3b04000,0x00007f8bf3c05000)]
  0x00007f8c549bf000 JavaThread "elasticsearch[es-data-us-east-1c-0][search][T#1]" daemon [_thread_blocked, id=94, stack(0x00007f8bf366c000,0x00007f8bf376d000)]
  0x00007f8c554ae800 JavaThread "elasticsearch[es-data-us-east-1c-0][get][T#1]" daemon [_thread_blocked, id=95, stack(0x00007f8bf13c2000,0x00007f8bf14c3000)]
  0x00007f8c549c0000 JavaThread "elasticsearch[es-data-us-east-1c-0][search][T#2]" daemon [_thread_blocked, id=97, stack(0x00007f8bf356b000,0x00007f8bf366c000)]
  0x00007f8c24244800 JavaThread "elasticsearch[es-data-us-east-1c-0][flush][T#1]" daemon [_thread_blocked, id=101, stack(0x00007f8bfb4d9000,0x00007f8bfb5da000)]
=>0x00007f8c18125000 JavaThread "elasticsearch[es-data-us-east-1c-0][[logstash-2018.06.19][2]: Lucene Merge Thread #8]" daemon [_thread_in_Java, id=140, stack(0x00007f8bf12c1000,0x00007f8bf13c2000)]

Other Threads:
  0x00007f8c540bd000 VMThread "VM Thread" [stack: 0x00007f8c58eae000,0x00007f8c58fae000] [id=51]
  0x00007f8c5415a800 WatcherThread [stack: 0x00007f8c586a5000,0x00007f8c587a5000] [id=59]
  0x00007f8c54023800 GCTaskThread "GC Thread#0" [stack: 0x00007f8c5a35d000,0x00007f8c5a45d000] [id=49]
  0x00007f8c54046000 ConcurrentGCThread "CMS Main Thread" [stack: 0x00007f8c59881000,0x00007f8c59981000] [id=50]

Threads with active compile tasks:
C2 CompilerThread013047 %     4       org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexWriter::writeBlock @ 326 (496 bytes)

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000

Heap:
 par new generation   total 76672K, used 7596K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   9% used [0x0000000080000000, 0x000000008060de20, 0x0000000084290000)
  from space 8512K,  16% used [0x0000000084290000, 0x00000000843ed2e8, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 177343K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73262K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K

Card table byte_map: [0x00007f8c59f5a000,0x00007f8c5a35b000] byte_map_base: 0x00007f8c59b5a000

Marking Bits: (CMSBitMap*) 0x00007f8c54044ee8
 Bits: [0x00007f8c39c73000, 0x00007f8c3bb26400)

Mod Union Table: (CMSBitMap*) 0x00007f8c54044f88
 Bits: [0x00007f8c5d5da000, 0x00007f8c5d654cd0)

Polling page: 0x00007f8c5d770000

CodeHeap 'non-profiled nmethods': size=120036Kb used=8163Kb max_used=8163Kb free=111872Kb
 bounds [0x00007f8c435ee000, 0x00007f8c43dee000, 0x00007f8c4ab27000]
CodeHeap 'profiled nmethods': size=120032Kb used=23258Kb max_used=23258Kb free=96773Kb
 bounds [0x00007f8c3c0b6000, 0x00007f8c3d786000, 0x00007f8c435ee000]
CodeHeap 'non-nmethods': size=5692Kb used=1352Kb max_used=1433Kb free=4339Kb
 bounds [0x00007f8c3bb27000, 0x00007f8c3bd97000, 0x00007f8c3c0b6000]
 total_blobs=16241 nmethods=11803 adapters=716
 compilation: enabled

Compilation events (10 events):
Event: 246.993 Thread 0x00007f8c540e4800 13045 %     3       org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter::merge @ 913 (1010 bytes)
Event: 247.019 Thread 0x00007f8c540e4800 nmethod 13045% 0x00007f8c3d76d390 code [0x00007f8c3d76dc80, 0x00007f8c3d772d58]
Event: 247.020 Thread 0x00007f8c540e2800 nmethod 13039 0x00007f8c43de4a10 code [0x00007f8c43de4c40, 0x00007f8c43de5c18]
Event: 247.020 Thread 0x00007f8c540e2800 13042       4       org.apache.lucene.util.bkd.BKDWriter$1::visit (10 bytes)
Event: 247.023 Thread 0x00007f8c540e2800 nmethod 13042 0x00007f8c43de6490 code [0x00007f8c43de6640, 0x00007f8c43de68d8]
Event: 247.023 Thread 0x00007f8c540e2800 13046       4       org.apache.lucene.index.IndexWriter::ensureOpen (33 bytes)
Event: 247.093 Thread 0x00007f8c540e2800 nmethod 13046 0x00007f8c43de6b90 code [0x00007f8c43de6d20, 0x00007f8c43de6db8]
Event: 247.101 Thread 0x00007f8c540e4800 13048       3       java.util.Spliterators::checkFromToBounds (74 bytes)
Event: 247.103 Thread 0x00007f8c540e4800 nmethod 13048 0x00007f8c3d774f10 code [0x00007f8c3d7751e0, 0x00007f8c3d775e48]
Event: 247.103 Thread 0x00007f8c540e2800 13047 %     4       org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexWriter::writeBlock @ 326 (496 bytes)

GC Heap History (10 events):
Event: 245.405 GC heap before
{Heap before GC invocations=219 (full 3):
 par new generation   total 76672K, used 75988K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,  99% used [0x0000000080000000, 0x000000008428b0f0, 0x0000000084290000)
  from space 8512K,  92% used [0x0000000084ae0000, 0x000000008528a288, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 171228K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73245K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 245.421 GC heap after
{Heap after GC invocations=220 (full 3):
 par new generation   total 76672K, used 5001K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  58% used [0x0000000084290000, 0x0000000084772578, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73245K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 245.593 GC heap before
{Heap before GC invocations=220 (full 3):
 par new generation   total 76672K, used 72835K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,  99% used [0x0000000080000000, 0x000000008423e860, 0x0000000084290000)
  from space 8512K,  58% used [0x0000000084290000, 0x0000000084772578, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73251K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 245.606 GC heap after
{Heap after GC invocations=221 (full 3):
 par new generation   total 76672K, used 1307K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  15% used [0x0000000084ae0000, 0x0000000084c26c18, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73251K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 246.598 GC heap before
{Heap before GC invocations=221 (full 3):
 par new generation   total 76672K, used 69266K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,  99% used [0x0000000080000000, 0x000000008425df90, 0x0000000084290000)
  from space 8512K,  15% used [0x0000000084ae0000, 0x0000000084c26c18, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73253K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 246.612 GC heap after
{Heap after GC invocations=222 (full 3):
 par new generation   total 76672K, used 7361K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  86% used [0x0000000084290000, 0x00000000849c0740, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73253K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 246.706 GC heap before
{Heap before GC invocations=222 (full 3):
 par new generation   total 76672K, used 75521K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K, 100% used [0x0000000080000000, 0x0000000084290000, 0x0000000084290000)
  from space 8512K,  86% used [0x0000000084290000, 0x00000000849c0740, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73253K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 246.722 GC heap after
{Heap after GC invocations=223 (full 3):
 par new generation   total 76672K, used 5070K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  59% used [0x0000000084ae0000, 0x0000000084fd3a38, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 177343K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73253K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 247.002 GC heap before
{Heap before GC invocations=223 (full 3):
 par new generation   total 76672K, used 73183K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,  99% used [0x0000000080000000, 0x0000000084284450, 0x0000000084290000)
  from space 8512K,  59% used [0x0000000084ae0000, 0x0000000084fd3a38, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 177343K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73262K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 247.016 GC heap after
{Heap after GC invocations=224 (full 3):
 par new generation   total 76672K, used 1396K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  16% used [0x0000000084290000, 0x00000000843ed2e8, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 177343K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73262K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 199.208 Thread 0x00007f8c1860c000 Uncommon trap: reason=bimorphic_or_optimized_type_check action=maybe_recompile pc=0x00007f8c43d0d028 method=org.apache.lucene.util.packed.MonotonicLongValues.get(II)J @ 23 c2
Event: 199.208 Thread 0x00007f8c1860c000 Uncommon trap: reason=bimorphic_or_optimized_type_check action=maybe_recompile pc=0x00007f8c43d0dda8 method=org.apache.lucene.util.packed.MonotonicLongValues.get(II)J @ 23 c2
Event: 201.194 Thread 0x00007f8c549bf000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f8c43caceb4 method=java.util.Comparators$NaturalOrderComparator.compare(Ljava/lang/Object;Ljava/lang/Object;)I @ 2 c2
Event: 201.194 Thread 0x00007f8c549bf000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f8c43caceb4 method=java.util.Comparators$NaturalOrderComparator.compare(Ljava/lang/Object;Ljava/lang/Object;)I @ 2 c2
Event: 201.194 Thread 0x00007f8c549bf000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f8c43caceb4 method=java.util.Comparators$NaturalOrderComparator.compare(Ljava/lang/Object;Ljava/lang/Object;)I @ 2 c2
Event: 201.194 Thread 0x00007f8c549bf000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f8c43caceb4 method=java.util.Comparators$NaturalOrderComparator.compare(Ljava/lang/Object;Ljava/lang/Object;)I @ 2 c2
Event: 214.831 Thread 0x00007f8c186fe000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c43d90c14 method=org.apache.lucene.store.GrowableByteArrayDataOutput.writeBytes([BII)V @ 15 c2
Event: 215.501 Thread 0x00007f8c186ff000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c43c552e0 method=org.apache.lucene.store.RateLimitedIndexOutput.checkRate()V @ 9 c2
Event: 220.893 Thread 0x00007f8bfc014800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c43d52988 method=io.netty.util.internal.shaded.org.jctools.queues.atomic.BaseMpscLinkedAtomicArrayQueue.isEmpty()Z @ 9 c2
Event: 245.900 Thread 0x00007f8c55cf6000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c43ddc2c0 method=java.util.HashSet.remove(Ljava/lang/Object;)Z @ 11 c2

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 122.204 Thread 0x00007f8c55974800 Implicit null exception at 0x00007f8c43a97d28 to 0x00007f8c43a98302
Event: 124.917 Thread 0x00007f8c55d1f800 Exception <a 'sun/nio/fs/UnixException'{0x0000000080912250}> (0x0000000080912250) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 125.000 Thread 0x00007f8c55d1f800 Exception <a 'sun/nio/fs/UnixException'{0x0000000080919e20}> (0x0000000080919e20) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 129.817 Thread 0x00007f8c55d1f800 Exception <a 'sun/nio/fs/UnixException'{0x0000000083fb12e0}> (0x0000000083fb12e0) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 129.824 Thread 0x00007f8c55d1f800 Exception <a 'sun/nio/fs/UnixException'{0x0000000083fb8eb0}> (0x0000000083fb8eb0) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 131.820 Thread 0x00007f8c18063800 Implicit null exception at 0x00007f8c43abc2bc to 0x00007f8c43ac8e5a
Event: 144.216 Thread 0x00007f8c2c00b000 Implicit null exception at 0x00007f8c43b023b0 to 0x00007f8c43b0d8ae
Event: 163.709 Thread 0x00007f8c18357000 Implicit null exception at 0x00007f8c43c0f8d4 to 0x00007f8c43c0fb1e
Event: 184.106 Thread 0x00007f8c04005800 Exception <a 'sun/nio/fs/UnixException'{0x0000000081fba9d0}> (0x0000000081fba9d0) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 245.899 Thread 0x00007f8c55cf6000 Implicit null exception at 0x00007f8c43ddc25c to 0x00007f8c43ddc2ae

Events (10 events):
Event: 246.622 Thread 0x00007f8c18123000 DEOPT PACKING pc=0x00007f8c3d0e4dfb sp=0x00007f8bfbdfc3f0
Event: 246.622 Thread 0x00007f8c18123000 DEOPT UNPACKING pc=0x00007f8c3bb4fe4a sp=0x00007f8bfbdfb918 mode 0
Event: 246.702 Thread 0x00007f8c18125000 Thread added: 0x00007f8c18125000
Event: 246.706 Executing VM operation: GenCollectForAllocation
Event: 246.722 Executing VM operation: GenCollectForAllocation done
Event: 246.793 Executing VM operation: RevokeBias
Event: 246.793 Executing VM operation: RevokeBias done
Event: 247.002 Executing VM operation: GenCollectForAllocation
Event: 247.016 Executing VM operation: GenCollectForAllocation done
Event: 247.097 Thread 0x00007f8c18123000 Thread exited: 0x00007f8c18123000


Dynamic libraries:
00400000-00401000 r-xp 00000000 103:02 2625764                           /opt/jdk-10.0.1/bin/java
00600000-00601000 rw-p 00000000 103:02 2625764                           /opt/jdk-10.0.1/bin/java
006f8000-0073e000 rw-p 00000000 00:00 0                                  [heap]
80000000-100a18000 rw-p 00000000 00:00 0
100a18000-140000000 ---p 00000000 00:00 0
7f8be8fda000-7f8be97ff000 r--s 00000000 103:03 2752559                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc_Lucene70_0.dvd
7f8be97ff000-7f8bea36d000 r--s 00000000 103:03 2752831                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc.fdt
7f8bea36d000-7f8bead09000 r--s 00000000 103:03 2752840                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc_Lucene50_0.tim
7f8bead09000-7f8beb112000 r--s 00000000 103:03 2752563                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc_Lucene50_0.doc
7f8beb311000-7f8beb7b9000 r--s 00000000 103:03 2752698                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r_Lucene70_0.dvd
7f8beb7b9000-7f8bebc77000 r--s 00000000 103:03 2752851                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r.fdt
7f8bebc77000-7f8bec210000 r--s 00000000 103:03 2752878                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r_Lucene50_0.tim
7f8bec23b000-7f8bec403000 r--s 00000000 103:03 2752832                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc_Lucene50_0.pos
7f8bee107000-7f8bee10b000 ---p 00000000 00:00 0
7f8bee10b000-7f8bee208000 rw-p 00000000 00:00 0
7f8bee685000-7f8bee7c1000 r--s 00000000 103:03 2752828                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n_Lucene70_0.dvd
7f8bee7c1000-7f8bee936000 r--s 00000000 103:03 2753020                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n_Lucene50_0.tim
7f8beea6c000-7f8bef277000 r--s 00000000 103:03 2752727                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt_Lucene70_0.dvd
7f8bef277000-7f8befd4f000 r--s 00000000 103:03 2752908                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt.fdt
7f8befd4f000-7f8bf0213000 r--s 00000000 103:03 2752774                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz_Lucene70_0.dvd
7f8bf0213000-7f8bf0271000 r--s 00000000 103:03 2752803                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz.dim
7f8bf0271000-7f8bf08e7000 r--s 00000000 103:03 2752781                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz.fdt
7f8bf08e7000-7f8bf0956000 r--s 00000000 103:03 2752871                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz.nvd
7f8bf0956000-7f8bf0f0d000 r--s 00000000 103:03 2752859                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz_Lucene50_0.tim
7f8bf0f0d000-7f8bf1025000 r--s 00000000 103:03 2752855                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz_Lucene50_0.pos
7f8bf1025000-7f8bf128e000 r--s 00000000 103:03 2752854                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz_Lucene50_0.doc
7f8bf12c1000-7f8bf12c5000 ---p 00000000 00:00 0
7f8bf12c5000-7f8bf13c2000 rw-p 00000000 00:00 0
7f8bf13c2000-7f8bf13c6000 ---p 00000000 00:00 0
7f8bf13c6000-7f8bf14c3000 rw-p 00000000 00:00 0
7f8bf14c3000-7f8bf1640000 rw-p 00000000 00:00 0
7f8bf1640000-7f8bf16c3000 ---p 00000000 00:00 0
7f8bf16c3000-7f8bf2671000 r--s 00000000 103:03 2752860                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_1.cfs
7f8bf2671000-7f8bf356b000 r--s 00000000 103:03 2752863                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_0.cfs
7f8bf356b000-7f8bf356f000 ---p 00000000 00:00 0
7f8bf356f000-7f8bf366c000 rw-p 00000000 00:00 0
7f8bf366c000-7f8bf3670000 ---p 00000000 00:00 0
7f8bf3670000-7f8bf376d000 rw-p 00000000 00:00 0
7f8bf3773000-7f8bf386f000 r--s 00000000 103:03 2752989                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n.fdt
7f8bf386f000-7f8bf3904000 r--s 00000000 103:03 2752947                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc.dim
7f8bf3904000-7f8bf3b04000 rw-p 00000000 00:00 0
7f8bf3b04000-7f8bf3b08000 ---p 00000000 00:00 0
7f8bf3b08000-7f8bf3c05000 rw-p 00000000 00:00 0
7f8bf3c05000-7f8bf3e05000 rw-p 00000000 00:00 0
7f8bf3e2b000-7f8bf3ee7000 r--s 00000000 103:03 2752735                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc.nvd
7f8bf40ec000-7f8bf4a6b000 r--s 00000000 103:03 2753085                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt_Lucene50_0.tim
7f8bf4a6b000-7f8bf5040000 r--s 00000000 103:03 2752644                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_hk.cfs
7f8bf521e000-7f8bf529a000 r--s 00000000 103:03 2752970                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_d5.cfs
7f8bf53f1000-7f8bf54ee000 r--s 00000000 103:03 2753045                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_qp.cfs
7f8bf54ee000-7f8bf555f000 r--s 00000000 103:03 2753018                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n_Lucene50_0.doc
7f8bf555f000-7f8bf55f9000 r--s 00000000 103:03 2753091                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5p.cfs
7f8bf560b000-7f8bf568c000 r--s 00000000 103:03 2752967                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_df.cfs
7f8bf56d6000-7f8bf5767000 r--s 00000000 103:03 2752745                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt.dim
7f8bf5767000-7f8bf57a6000 r--s 00000000 103:03 2753005                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r.dim
7f8bf57a6000-7f8bf57f9000 r--s 00000000 103:03 2752973                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r.nvd
7f8bf57f9000-7f8bf58a9000 r--s 00000000 103:03 2752877                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r_Lucene50_0.pos
7f8bf58a9000-7f8bf5a60000 r--s 00000000 103:03 2752847                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r_Lucene50_0.doc
7f8bf5a6c000-7f8bf6b0d000 r--s 00000000 103:03 2752714                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_0.cfs
7f8bf6b0d000-7f8bf7785000 r--s 00000000 103:03 2752639                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_hb.cfs
7f8bf77b5000-7f8bf77b9000 ---p 00000000 00:00 0
7f8bf77b9000-7f8bf78b6000 rw-p 00000000 00:00 0
7f8bf78f8000-7f8bf79aa000 r--s 00000000 103:03 2753010                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt.nvd
7f8bf79aa000-7f8bf7b5c000 r--s 00000000 103:03 2753084                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt_Lucene50_0.pos
7f8bf7b5c000-7f8bf7f38000 r--s 00000000 103:03 2753083                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt_Lucene50_0.doc
7f8bf7f6d000-7f8bf7fc8000 r--s 00000000 103:03 2753015                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_h9.cfs
7f8bf7fc8000-7f8bf7fcc000 ---p 00000000 00:00 0
7f8bf7fcc000-7f8bf80c9000 rw-p 00000000 00:00 0
7f8bf80c9000-7f8bf80cd000 ---p 00000000 00:00 0
7f8bf80cd000-7f8bf81ca000 rw-p 00000000 00:00 0
7f8bf81ca000-7f8bf83ca000 rw-p 00000000 00:00 0
7f8bf83ca000-7f8bf83ce000 ---p 00000000 00:00 0
7f8bf83ce000-7f8bf84cb000 rw-p 00000000 00:00 0
7f8bf84cb000-7f8bf84cf000 ---p 00000000 00:00 0
7f8bf84cf000-7f8bf85cc000 rw-p 00000000 00:00 0
7f8bf85cc000-7f8bf87cc000 rw-p 00000000 00:00 0
7f8bf87cc000-7f8bf87d0000 ---p 00000000 00:00 0
7f8bf87d0000-7f8bf88cd000 rw-p 00000000 00:00 0
7f8bf88cd000-7f8bf88d1000 ---p 00000000 00:00 0
7f8bf88d1000-7f8bf89ce000 rw-p 00000000 00:00 0
7f8bf89ce000-7f8bf89d2000 ---p 00000000 00:00 0
7f8bf89d2000-7f8bf8acf000 rw-p 00000000 00:00 0
7f8bf8acf000-7f8bf8ad3000 ---p 00000000 00:00 0
7f8bf8ad3000-7f8bf8bd0000 rw-p 00000000 00:00 0
7f8bf8bd0000-7f8bf8bd4000 ---p 00000000 00:00 0
7f8bf8bd4000-7f8bf8cd1000 rw-p 00000000 00:00 0
7f8bf8cd1000-7f8bf8cd5000 ---p 00000000 00:00 0
7f8bf8cd5000-7f8bf8dd2000 rw-p 00000000 00:00 0
7f8bf8dd2000-7f8bf8fd2000 rw-p 00000000 00:00 0
7f8bf8fd2000-7f8bf8fd6000 ---p 00000000 00:00 0
7f8bf8fd6000-7f8bf90d3000 rw-p 00000000 00:00 0
7f8bf90d3000-7f8bf90d7000 ---p 00000000 00:00 0
7f8bf90d7000-7f8bf91d4000 rw-p 00000000 00:00 0
7f8bf91d4000-7f8bf91d8000 ---p 00000000 00:00 0
7f8bf91d8000-7f8bfb2d7000 rw-p 00000000 00:00 0
7f8bfb2d7000-7f8bfb2db000 ---p 00000000 00:00 0
7f8bfb2db000-7f8bfb3d8000 rw-p 00000000 00:00 0
7f8bfb3d8000-7f8bfb3dc000 ---p 00000000 00:00 0
7f8bfb3dc000-7f8bfb4d9000 rw-p 00000000 00:00 0
7f8bfb4d9000-7f8bfb4dd000 ---p 00000000 00:00 0
7f8bfb4dd000-7f8bfb5da000 rw-p 00000000 00:00 0
7f8bfb5da000-7f8bfb5de000 ---p 00000000 00:00 0
7f8bfb5de000-7f8bfb6db000 rw-p 00000000 00:00 0
7f8bfb6db000-7f8bfb6df000 ---p 00000000 00:00 0
7f8bfb6df000-7f8bfb7dc000 rw-p 00000000 00:00 0
7f8bfb7dc000-7f8bfb7e0000 ---p 00000000 00:00 0
7f8bfb7e0000-7f8bfb8dd000 rw-p 00000000 00:00 0
7f8bfb8dd000-7f8bfb8f3000 r-xp 00000000 103:02 2107384                   /usr/lib64/libresolv-2.17.so
7f8bfb8f3000-7f8bfbaf2000 ---p 00016000 103:02 2107384                   /usr/lib64/libresolv-2.17.so
7f8bfbaf2000-7f8bfbaf3000 r--p 00015000 103:02 2107384                   /usr/lib64/libresolv-2.17.so
7f8bfbaf3000-7f8bfbaf4000 rw-p 00016000 103:02 2107384                   /usr/lib64/libresolv-2.17.so
7f8bfbaf4000-7f8bfbaf6000 rw-p 00000000 00:00 0
7f8bfbaf6000-7f8bfbafb000 r-xp 00000000 103:02 2107325                   /usr/lib64/libnss_dns-2.17.so
7f8bfbafb000-7f8bfbcfb000 ---p 00005000 103:02 2107325                   /usr/lib64/libnss_dns-2.17.so
7f8bfbcfb000-7f8bfbcfc000 r--p 00005000 103:02 2107325                   /usr/lib64/libnss_dns-2.17.so
7f8bfbcfc000-7f8bfbcfd000 rw-p 00006000 103:02 2107325                   /usr/lib64/libnss_dns-2.17.so
7f8bfbcfd000-7f8bfbd01000 ---p 00000000 00:00 0
7f8bfbd01000-7f8bfbdfe000 rw-p 00000000 00:00 0
7f8bfbdfe000-7f8bfbe02000 ---p 00000000 00:00 0
7f8bfbe02000-7f8bfbeff000 rw-p 00000000 00:00 0
7f8bfbeff000-7f8bfbf03000 ---p 00000000 00:00 0
7f8bfbf03000-7f8bfc000000 rw-p 00000000 00:00 0
7f8bfc000000-7f8bfc052000 rw-p 00000000 00:00 0
7f8bfc052000-7f8c00000000 ---p 00000000 00:00 0
7f8c00000000-7f8c00026000 rw-p 00000000 00:00 0
7f8c00026000-7f8c04000000 ---p 00000000 00:00 0
7f8c04000000-7f8c0412f000 rw-p 00000000 00:00 0
7f8c0412f000-7f8c08000000 ---p 00000000 00:00 0
7f8c08000000-7f8c08058000 rw-p 00000000 00:00 0
7f8c08058000-7f8c0c000000 ---p 00000000 00:00 0
7f8c0c000000-7f8c0c094000 rw-p 00000000 00:00 0
7f8c0c094000-7f8c10000000 ---p 00000000 00:00 0
7f8c10000000-7f8c10026000 rw-p 00000000 00:00 0
7f8c10026000-7f8c14000000 ---p 00000000 00:00 0
7f8c14000000-7f8c172f3000 rw-p 00000000 00:00 0
7f8c172f3000-7f8c18000000 ---p 00000000 00:00 0
7f8c18000000-7f8c18a77000 rw-p 00000000 00:00 0
7f8c18a77000-7f8c1c000000 ---p 00000000 00:00 0
7f8c1c000000-7f8c1c042000 rw-p 00000000 00:00 0
7f8c1c042000-7f8c20000000 ---p 00000000 00:00 0
7f8c20000000-7f8c20028000 rw-p 00000000 00:00 0
7f8c20028000-7f8c24000000 ---p 00000000 00:00 0
7f8c24000000-7f8c2456a000 rw-p 00000000 00:00 0
7f8c2456a000-7f8c28000000 ---p 00000000 00:00 0
7f8c28000000-7f8c28070000 rw-p 00000000 00:00 0
7f8c28070000-7f8c2c000000 ---p 00000000 00:00 0
7f8c2c000000-7f8c2c021000 rw-p 00000000 00:00 0
7f8c2c021000-7f8c30000000 ---p 00000000 00:00 0
7f8c3002a000-7f8c30076000 r--s 00000000 103:03 2752898                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_gz.cfs
7f8c30098000-7f8c300a8000 r--s 00000000 103:03 2752834                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n.dim
7f8c300a8000-7f8c300cc000 r--s 00000000 103:03 2752880                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n.nvd
7f8c300d4000-7f8c300e2000 r--s 00000000 103:03 2752893                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dm.cfs
7f8c300e2000-7f8c300f0000 r--s 00000000 103:03 2753078                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_ha.cfs
7f8c300f0000-7f8c300ff000 r--s 00000000 103:03 2753054                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5w.cfs
7f8c300ff000-7f8c30103000 ---p 00000000 00:00 0
7f8c30103000-7f8c30200000 rw-p 00000000 00:00 0
7f8c30200000-7f8c30400000 rw-p 00000000 00:00 0
7f8c30400000-7f8c30600000 rw-p 00000000 00:00 0
7f8c30600000-7f8c30800000 rw-p 00000000 00:00 0
7f8c30800000-7f8c30a00000 rw-p 00000000 00:00 0
7f8c30a00000-7f8c30c00000 rw-p 00000000 00:00 0
7f8c30c00000-7f8c30e00000 rw-p 00000000 00:00 0
7f8c30e00000-7f8c31000000 rw-p 00000000 00:00 0
7f8c31000000-7f8c31200000 rw-p 00000000 00:00 0
7f8c31200000-7f8c313f5000 rw-p 00000000 00:00 0
7f8c313f5000-7f8c31400000 ---p 00000000 00:00 0
7f8c31400000-7f8c31600000 rw-p 00000000 00:00 0
7f8c31600000-7f8c31800000 rw-p 00000000 00:00 0
7f8c31800000-7f8c31a00000 rw-p 00000000 00:00 0
7f8c31a00000-7f8c31c00000 rw-p 00000000 00:00 0
7f8c31c00000-7f8c31e00000 rw-p 00000000 00:00 0
7f8c31e00000-7f8c32000000 rw-p 00000000 00:00 0
7f8c32000000-7f8c34000000 rw-p 00000000 00:00 0
7f8c34000000-7f8c34021000 rw-p 00000000 00:00 0
7f8c34021000-7f8c38000000 ---p 00000000 00:00 0
7f8c38003000-7f8c3802c000 r--s 00000000 103:03 2753019                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n_Lucene50_0.pos
7f8c3802f000-7f8c3803c000 r--s 00000000 103:03 2753048                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5y.cfs
7f8c3803c000-7f8c38049000 r--s 00000000 103:03 2753050                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_qr.cfs
7f8c38049000-7f8c3804e000 r--s 00000000 103:03 2753068                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_qq.cfs
7f8c3804e000-7f8c38054000 r--s 00000000 103:03 2753074                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dk.cfs
7f8c38059000-7f8c3805e000 r--s 00000000 103:03 2753060                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5w.cfs
7f8c38064000-7f8c3806c000 r--s 00000000 103:03 2753026                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dj.cfs
7f8c38073000-7f8c3807a000 r--s 00000000 103:03 2753031                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5t.cfs
7f8c3807a000-7f8c38080000 r--s 00000000 103:03 2753033                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5v.cfs
7f8c3808c000-7f8c38093000 r--s 00000000 103:03 2752969                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_di.cfs
7f8c38099000-7f8c3809f000 r--s 00000000 103:03 2752995                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5s.cfs
7f8c3809f000-7f8c380a5000 r--s 00000000 103:03 2752997                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5u.cfs
7f8c380a5000-7f8c380ac000 r--s 00000000 103:03 2752975                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dg.cfs
7f8c380b9000-7f8c380c3000 r--s 00000000 103:03 2752983                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5s.cfs
7f8c380c3000-7f8c382c3000 rw-p 00000000 00:00 0
7f8c382c3000-7f8c382c4000 r-xp 00000000 103:02 2626260                   /opt/jdk-10.0.1/lib/libextnet.so
7f8c382c4000-7f8c384c3000 ---p 00001000 103:02 2626260                   /opt/jdk-10.0.1/lib/libextnet.so
7f8c384c3000-7f8c384c4000 rw-p 00000000 103:02 2626260                   /opt/jdk-10.0.1/lib/libextnet.so
7f8c384c4000-7f8c386c4000 rw-p 00000000 00:00 0
7f8c386c4000-7f8c388c4000 rw-p 00000000 00:00 0
7f8c388c4000-7f8c38ac4000 rw-p 00000000 00:00 0
7f8c38ac4000-7f8c38adc000 r-xp 00000000 103:02 6292735                   /tmp/elasticsearch.9Ts2lgAF/jna--1985354563/jna2246371085384706757.tmp (deleted)
7f8c38adc000-7f8c38cdc000 ---p 00018000 103:02 6292735                   /tmp/elasticsearch.9Ts2lgAF/jna--1985354563/jna2246371085384706757.tmp (deleted)
7f8c38cdc000-7f8c38cdd000 rw-p 00018000 103:02 6292735                   /tmp/elasticsearch.9Ts2lgAF/jna--1985354563/jna2246371085384706757.tmp (deleted)
7f8c38cdd000-7f8c38edd000 rw-p 00000000 00:00 0
7f8c38edd000-7f8c390dd000 rw-p 00000000 00:00 0
7f8c390dd000-7f8c390e2000 r-xp 00000000 103:02 2626278                   /opt/jdk-10.0.1/lib/libmanagement_ext.so
7f8c390e2000-7f8c392e1000 ---p 00005000 103:02 2626278                   /opt/jdk-10.0.1/lib/libmanagement_ext.so
7f8c392e1000-7f8c392e2000 rw-p 00004000 103:02 2626278                   /opt/jdk-10.0.1/lib/libmanagement_ext.so
7f8c392e2000-7f8c392eb000 r--s 00000000 103:03 2752980                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5q.cfs
7f8c392f4000-7f8c392f7000 r--s 00000000 103:03 2752875                   /data/nodes/0/indices/6UUqrS5_RUSS5Ozy2tSYTg/0/index/_4.cfs
7f8c39304000-7f8c39308000 r--s 00000000 103:03 2753055                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5u.cfs
7f8c39308000-7f8c3930d000 r--s 00000000 103:03 2752895                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dh.cfs
7f8c3930d000-7f8c39313000 r--s 00000000 103:03 2752949                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5p.cfs
7f8c39313000-7f8c39319000 r--s 00000000 103:03 2752951                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5r.cfs
7f8c39319000-7f8c3931d000 ---p 00000000 00:00 0
7f8c3931d000-7f8c3941a000 rw-p 00000000 00:00 0
7f8c3941a000-7f8c3941e000 r-xp 00000000 103:02 2626276                   /opt/jdk-10.0.1/lib/libmanagement.so
7f8c3941e000-7f8c3961d000 ---p 00004000 103:02 2626276                   /opt/jdk-10.0.1/lib/libmanagement.so
7f8c3961d000-7f8c3961e000 rw-p 00003000 103:02 2626276                   /opt/jdk-10.0.1/lib/libmanagement.so
7f8c3961e000-7f8c3965b000 r-xp 00000000 103:02 2626287                   /opt/jdk-10.0.1/lib/libsunec.so
7f8c3965b000-7f8c3985b000 ---p 0003d000 103:02 2626287                   /opt/jdk-10.0.1/lib/libsunec.so
7f8c3985b000-7f8c39862000 rw-p 0003d000 103:02 2626287                   /opt/jdk-10.0.1/lib/libsunec.so
7f8c39862000-7f8c39a73000 rw-p 00000000 00:00 0
7f8c39a73000-7f8c39c73000 rw-p 00000000 00:00 0
7f8c39c73000-7f8c3bb27000 rw-p 00000000 00:00 0
7f8c3bb27000-7f8c3bd97000 rwxp 00000000 00:00 0
7f8c3bd97000-7f8c3c0b6000 ---p 00000000 00:00 0
7f8c3c0b6000-7f8c3d786000 rwxp 00000000 00:00 0
7f8c3d786000-7f8c435ee000 ---p 00000000 00:00 0
7f8c435ee000-7f8c43dee000 rwxp 00000000 00:00 0
7f8c43dee000-7f8c4ab27000 ---p 00000000 00:00 0
7f8c4ab27000-7f8c54000000 r--s 00000000 103:02 2626291                   /opt/jdk-10.0.1/lib/modules
7f8c54000000-7f8c55d4e000 rw-p 00000000 00:00 0
7f8c55d4e000-7f8c58000000 ---p 00000000 00:00 0
7f8c58009000-7f8c5800e000 r--s 00000000 103:03 2752916                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5r.cfs
7f8c58015000-7f8c5801a000 r--s 00000000 103:03 2752915                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5o.cfs
7f8c5801b000-7f8c58022000 r--s 00000000 103:03 2752926                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5v.cfs
7f8c58022000-7f8c58027000 r--s 00000000 103:03 2752920                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5t.cfs
7f8c58027000-7f8c5802c000 r--s 00000000 103:03 2752918                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5q.cfs
7f8c58031000-7f8c58035000 r--s 00000000 103:03 2752922                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dl.cfs
7f8c58035000-7f8c58039000 ---p 00000000 00:00 0
7f8c58039000-7f8c58136000 rw-p 00000000 00:00 0
7f8c58136000-7f8c5813a000 ---p 00000000 00:00 0
7f8c5813a000-7f8c58158000 rw-p 00000000 00:00 0
7f8c58158000-7f8c5815c000 ---p 00000000 00:00 0
7f8c5815c000-7f8c5817a000 rw-p 00000000 00:00 0
7f8c5817a000-7f8c58190000 r-xp 00000000 103:02 2626280                   /opt/jdk-10.0.1/lib/libnet.so
7f8c58190000-7f8c58390000 ---p 00016000 103:02 2626280                   /opt/jdk-10.0.1/lib/libnet.so
7f8c58390000-7f8c58391000 rw-p 00016000 103:02 2626280                   /opt/jdk-10.0.1/lib/libnet.so
7f8c58391000-7f8c583a1000 r-xp 00000000 103:02 2626281                   /opt/jdk-10.0.1/lib/libnio.so
7f8c583a1000-7f8c585a1000 ---p 00010000 103:02 2626281                   /opt/jdk-10.0.1/lib/libnio.so
7f8c585a1000-7f8c585a2000 rw-p 00010000 103:02 2626281                   /opt/jdk-10.0.1/lib/libnio.so
7f8c585a2000-7f8c585a6000 ---p 00000000 00:00 0
7f8c585a6000-7f8c586a3000 rw-p 00000000 00:00 0
7f8c586a3000-7f8c586a4000 ---p 00000000 00:00 0
7f8c586a4000-7f8c587a5000 rw-p 00000000 00:00 0
7f8c587a5000-7f8c587a9000 ---p 00000000 00:00 0
7f8c587a9000-7f8c588a6000 rw-p 00000000 00:00 0
7f8c588a6000-7f8c588aa000 ---p 00000000 00:00 0
7f8c588aa000-7f8c589a7000 rw-p 00000000 00:00 0
7f8c589a7000-7f8c589ab000 ---p 00000000 00:00 0
7f8c589ab000-7f8c58aa8000 rw-p 00000000 00:00 0
7f8c58aa8000-7f8c58aac000 ---p 00000000 00:00 0
7f8c58aac000-7f8c58ba9000 rw-p 00000000 00:00 0
7f8c58ba9000-7f8c58bad000 ---p 00000000 00:00 0
7f8c58bad000-7f8c58caa000 rw-p 00000000 00:00 0
7f8c58caa000-7f8c58cae000 ---p 00000000 00:00 0
7f8c58cae000-7f8c58dab000 rw-p 00000000 00:00 0
7f8c58dab000-7f8c58daf000 ---p 00000000 00:00 0
7f8c58daf000-7f8c58eac000 rw-p 00000000 00:00 0
7f8c58eac000-7f8c58ead000 ---p 00000000 00:00 0
7f8c58ead000-7f8c5907f000 rw-p 00000000 00:00 0
7f8c5907f000-7f8c5987f000 rw-p 00000000 00:00 0
7f8c5987f000-7f8c59880000 ---p 00000000 00:00 0
7f8c59880000-7f8c59a82000 rw-p 00000000 00:00 0
7f8c59a82000-7f8c59e59000 rw-p 00000000 00:00 0
7f8c59e59000-7f8c59f5a000 rw-p 00000000 00:00 0
7f8c59f5a000-7f8c5a35a000 rw-p 00000000 00:00 0
7f8c5a35a000-7f8c5a35b000 rw-p 00000000 00:00 0
7f8c5a35b000-7f8c5a35c000 ---p 00000000 00:00 0
7f8c5a35c000-7f8c5a45d000 rw-p 00000000 00:00 0
7f8c5a45d000-7f8c5a46d000 rw-p 00000000 00:00 0
7f8c5a46d000-7f8c5a548000 ---p 00000000 00:00 0
7f8c5a548000-7f8c5a576000 rw-p 00000000 00:00 0
7f8c5a576000-7f8c5a633000 ---p 00000000 00:00 0
7f8c5a633000-7f8c5a64d000 r-xp 00000000 103:02 2626271                   /opt/jdk-10.0.1/lib/libjimage.so
7f8c5a64d000-7f8c5a84c000 ---p 0001a000 103:02 2626271                   /opt/jdk-10.0.1/lib/libjimage.so
7f8c5a84c000-7f8c5a84e000 rw-p 00019000 103:02 2626271                   /opt/jdk-10.0.1/lib/libjimage.so
7f8c5a84e000-7f8c5a860000 rw-p 00000000 00:00 0
7f8c5a860000-7f8c5a867000 r-xp 00000000 103:02 2626290                   /opt/jdk-10.0.1/lib/libzip.so
7f8c5a867000-7f8c5aa66000 ---p 00007000 103:02 2626290                   /opt/jdk-10.0.1/lib/libzip.so
7f8c5aa66000-7f8c5aa67000 rw-p 00006000 103:02 2626290                   /opt/jdk-10.0.1/lib/libzip.so
7f8c5aa67000-7f8c5aa73000 r-xp 00000000 103:02 2107327                   /usr/lib64/libnss_files-2.17.so
7f8c5aa73000-7f8c5ac72000 ---p 0000c000 103:02 2107327                   /usr/lib64/libnss_files-2.17.so
7f8c5ac72000-7f8c5ac73000 r--p 0000b000 103:02 2107327                   /usr/lib64/libnss_files-2.17.so
7f8c5ac73000-7f8c5ac74000 rw-p 0000c000 103:02 2107327                   /usr/lib64/libnss_files-2.17.so
7f8c5ac74000-7f8c5ac7a000 rw-p 00000000 00:00 0
7f8c5ac7a000-7f8c5aca4000 r-xp 00000000 103:02 2626267                   /opt/jdk-10.0.1/lib/libjava.so
7f8c5aca4000-7f8c5aea4000 ---p 0002a000 103:02 2626267                   /opt/jdk-10.0.1/lib/libjava.so
7f8c5aea4000-7f8c5aea6000 rw-p 0002a000 103:02 2626267                   /opt/jdk-10.0.1/lib/libjava.so
7f8c5aea6000-7f8c5aeb3000 r-xp 00000000 103:02 2626289                   /opt/jdk-10.0.1/lib/libverify.so
7f8c5aeb3000-7f8c5b0b2000 ---p 0000d000 103:02 2626289                   /opt/jdk-10.0.1/lib/libverify.so
7f8c5b0b2000-7f8c5b0b4000 rw-p 0000c000 103:02 2626289                   /opt/jdk-10.0.1/lib/libverify.so
7f8c5b0b4000-7f8c5b0bb000 r-xp 00000000 103:02 2107394                   /usr/lib64/librt-2.17.so
7f8c5b0bb000-7f8c5b2ba000 ---p 00007000 103:02 2107394                   /usr/lib64/librt-2.17.so
7f8c5b2ba000-7f8c5b2bb000 r--p 00006000 103:02 2107394                   /usr/lib64/librt-2.17.so
7f8c5b2bb000-7f8c5b2bc000 rw-p 00007000 103:02 2107394                   /usr/lib64/librt-2.17.so
7f8c5b2bc000-7f8c5b3bd000 r-xp 00000000 103:02 2107298                   /usr/lib64/libm-2.17.so
7f8c5b3bd000-7f8c5b5bc000 ---p 00101000 103:02 2107298                   /usr/lib64/libm-2.17.so
7f8c5b5bc000-7f8c5b5bd000 r--p 00100000 103:02 2107298                   /usr/lib64/libm-2.17.so
7f8c5b5bd000-7f8c5b5be000 rw-p 00101000 103:02 2107298                   /usr/lib64/libm-2.17.so
7f8c5b5be000-7f8c5c5ce000 r-xp 00000000 103:02 2626301                   /opt/jdk-10.0.1/lib/server/libjvm.so
7f8c5c5ce000-7f8c5c7cd000 ---p 01010000 103:02 2626301                   /opt/jdk-10.0.1/lib/server/libjvm.so
7f8c5c7cd000-7f8c5c8aa000 r--p 0100f000 103:02 2626301                   /opt/jdk-10.0.1/lib/server/libjvm.so
7f8c5c8aa000-7f8c5c8e3000 rw-p 010ec000 103:02 2626301                   /opt/jdk-10.0.1/lib/server/libjvm.so
7f8c5c8e3000-7f8c5c940000 rw-p 00000000 00:00 0
7f8c5c940000-7f8c5cb03000 r-xp 00000000 103:02 2107190                   /usr/lib64/libc-2.17.so
7f8c5cb03000-7f8c5cd02000 ---p 001c3000 103:02 2107190                   /usr/lib64/libc-2.17.so
7f8c5cd02000-7f8c5cd06000 r--p 001c2000 103:02 2107190                   /usr/lib64/libc-2.17.so
7f8c5cd06000-7f8c5cd08000 rw-p 001c6000 103:02 2107190                   /usr/lib64/libc-2.17.so
7f8c5cd08000-7f8c5cd0d000 rw-p 00000000 00:00 0
7f8c5cd0d000-7f8c5cd0f000 r-xp 00000000 103:02 2107217                   /usr/lib64/libdl-2.17.so
7f8c5cd0f000-7f8c5cf0f000 ---p 00002000 103:02 2107217                   /usr/lib64/libdl-2.17.so
7f8c5cf0f000-7f8c5cf10000 r--p 00002000 103:02 2107217                   /usr/lib64/libdl-2.17.so
7f8c5cf10000-7f8c5cf11000 rw-p 00003000 103:02 2107217                   /usr/lib64/libdl-2.17.so
7f8c5cf11000-7f8c5cf1f000 r-xp 00000000 103:02 2626252                   /opt/jdk-10.0.1/lib/jli/libjli.so
7f8c5cf1f000-7f8c5d11f000 ---p 0000e000 103:02 2626252                   /opt/jdk-10.0.1/lib/jli/libjli.so
7f8c5d11f000-7f8c5d120000 rw-p 0000e000 103:02 2626252                   /opt/jdk-10.0.1/lib/jli/libjli.so
7f8c5d120000-7f8c5d137000 r-xp 00000000 103:02 2107375                   /usr/lib64/libpthread-2.17.so
7f8c5d137000-7f8c5d336000 ---p 00017000 103:02 2107375                   /usr/lib64/libpthread-2.17.so
7f8c5d336000-7f8c5d337000 r--p 00016000 103:02 2107375                   /usr/lib64/libpthread-2.17.so
7f8c5d337000-7f8c5d338000 rw-p 00017000 103:02 2107375                   /usr/lib64/libpthread-2.17.so
7f8c5d338000-7f8c5d33c000 rw-p 00000000 00:00 0
7f8c5d33c000-7f8c5d351000 r-xp 00000000 103:02 2107454                   /usr/lib64/libz.so.1.2.7
7f8c5d351000-7f8c5d550000 ---p 00015000 103:02 2107454                   /usr/lib64/libz.so.1.2.7
7f8c5d550000-7f8c5d551000 r--p 00014000 103:02 2107454                   /usr/lib64/libz.so.1.2.7
7f8c5d551000-7f8c5d552000 rw-p 00015000 103:02 2107454                   /usr/lib64/libz.so.1.2.7
7f8c5d552000-7f8c5d574000 r-xp 00000000 103:02 2107166                   /usr/lib64/ld-2.17.so
7f8c5d575000-7f8c5d577000 r--s 00000000 103:03 2752869                   /data/nodes/0/indices/6UUqrS5_RUSS5Ozy2tSYTg/0/index/_3.cfs
7f8c5d57a000-7f8c5d57f000 r--s 00000000 103:03 2752934                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5x.cfs
7f8c5d57f000-7f8c5d580000 rwxp 00000000 00:00 0
7f8c5d580000-7f8c5d581000 r--p 00000000 00:00 0
7f8c5d581000-7f8c5d5da000 rw-p 00000000 00:00 0
7f8c5d5da000-7f8c5d65a000 rw-p 00000000 00:00 0
7f8c5d65a000-7f8c5d661000 ---p 00000000 00:00 0
7f8c5d661000-7f8c5d669000 rw-s 00000000 103:02 6292730                   /tmp/hsperfdata_elasticsearch/1
7f8c5d669000-7f8c5d66d000 ---p 00000000 00:00 0
7f8c5d66d000-7f8c5d770000 rw-p 00000000 00:00 0
7f8c5d770000-7f8c5d771000 ---p 00000000 00:00 0
7f8c5d771000-7f8c5d772000 r--p 00000000 00:00 0
7f8c5d772000-7f8c5d773000 rw-p 00000000 00:00 0
7f8c5d773000-7f8c5d774000 r--p 00021000 103:02 2107166                   /usr/lib64/ld-2.17.so
7f8c5d774000-7f8c5d775000 rw-p 00022000 103:02 2107166                   /usr/lib64/ld-2.17.so
7f8c5d775000-7f8c5d776000 rw-p 00000000 00:00 0
7ffc0a637000-7ffc0a659000 rw-p 00000000 00:00 0                          [stack]
7ffc0a7b1000-7ffc0a7b3000 r--p 00000000 00:00 0                          [vvar]
7ffc0a7b3000-7ffc0a7b5000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


VM Arguments:
jvm_args: -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.9Ts2lgAF -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Des.cgroups.hierarchy.override=/ -Xms2g -Xmx2g -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=tar
java_command: org.elasticsearch.bootstrap.Elasticsearch -Ecluster.name=tds-es-db -Expack.watcher.enabled=false -Expack.ml.enabled=false -Expack.logstash.enabled=false -Ediscovery.zen.ping.unicast.hosts=elasticsearch-discovery -Ehttp.enabled=false -Epath.data=/data -Enode.data=true -Enode.master=false -Enode.name=es-data-us-east-1c-0 -Enetwork.host=_eth0:ipv4_ -Expack.graph.enabled=false -Ediscovery.zen.minimum_master_nodes=2 -Expack.security.enabled=false -Enode.ingest=false
java_class_path (initial): /usr/share/elasticsearch/lib/spatial4j-0.7.jar:/usr/share/elasticsearch/lib/lucene-backward-codecs-7.3.1.jar:/usr/share/elasticsearch/lib/elasticsearch-core-6.3.0.jar:/usr/share/elasticsearch/lib/jackson-core-2.8.10.jar:/usr/share/elasticsearch/lib/lucene-analyzers-common-7.3.1.jar:/usr/share/elasticsearch/lib/log4j-1.2-api-2.9.1.jar:/usr/share/elasticsearch/lib/HdrHistogram-2.1.9.jar:/usr/share/elasticsearch/lib/elasticsearch-x-content-6.3.0.jar:/usr/share/elasticsearch/lib/lucene-highlighter-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-memory-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-queryparser-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-misc-7.3.1.jar:/usr/share/elasticsearch/lib/snakeyaml-1.17.jar:/usr/share/elasticsearch/lib/lucene-spatial3d-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-spatial-extras-7.3.1.jar:/usr/share/elasticsearch/lib/plugin-classloader-6.3.0.jar:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.8.10.jar:/usr/share/elasticsearch/lib/jts-core-1.15.0.jar:/usr/share/elasticsearch/lib/lucene-grouping-7.3.1.jar:/usr/share/elasticsearch/lib/log4j-core-2.9.1.jar:/usr/share/elasticsearch/lib/lucene-sandbox-7.3.1.jar:/usr/share/elasticsearch/lib/jna-4.5.1.jar:/usr/share/elasticsearch/lib/elasticsearch-6.3.0.jar:/usr/share/elasticsearch/lib/lucene-queries-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-core-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-spatial-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-join-7.3.1.jar:/usr/share/elasticsearch/lib/plugin-cli-6.3.0.jar:/usr/share/elasticsearch/lib/log4j-api-2.9.1.jar:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.8.10.jar:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.8.10.jar:/usr/share/elasticsearch/lib/hppc-0.7.1.jar:/usr/share/elasticsearch/lib/t-digest-3.2.jar:/usr/share/elasticsearch/lib/joda-time-2.9.9.jar:/usr/share/elasticsearch/lib/elasticsearch-cli-6.3.0.jar:/usr/share/elasticsearch/lib/jopt-simple-5.0.2.jar:/usr/share/elasticsearch
Launcher Type: SUN_STANDARD

Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags
#2: file=logs/gc.log gc+ref=info,gc+ref+start=info,gc+compaction=info,safepoint=info,gc+metaspace+alloc=info,gc+metaspace=info,gc+metaspace+freelist=info,gc+metaspace+freelist+blocks=info,gc+heap+exit=info,gc+cpu=info,gc+task+time=info,gc+task+thread=info,gc+jni=info,gc+stringdedup=info,gc+marking+start=info,gc+remset+exit=info,gc+remset=info,gc+mmu=info,gc+ihop=info,gc+phases+task=info,gc+phases+ref=info,gc+verify+start=info,gc+ergo+ihop=info,gc+refine=info,gc+ergo+refine=info,gc+stats=info,gc+ergo+cset=info,gc+humongous=info,gc+stringtable=info,gc+task+stats=info,gc+heap+region=info,gc+ergo+heap=info,gc+marking=info,gc+phases+verify=info,gc+phases+verify+start=info,gc+phases=info,gc+phases+start=info,gc+survivor=info,gc+state=info,gc+promotion=info,gc+plab=info,gc+verify=info,gc+alloc=info,gc+liveness=info,gc+classhisto+start=info,gc+start=info,gc+classhisto=info,gc+barrier=info,gc+bot=info,gc+freelist+census=info,gc+freelist+stats=info,gc+region=info,gc+heap+coops=info,gc+heap=info,gc+age=trace,gc+ergo=info,gc=info,gc+tlab=info,gc+task=info,gc+freelist=info utctime,pid,tags filecount=32,filesize=64M

Environment Variables:
JAVA_HOME=/opt/jdk-10.0.1
PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Signal Handlers:
SIGSEGV: [libjvm.so+0xd3d000], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xd3d000], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xd3d000], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xb511c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xb511c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xd3d000], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xb51550], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0xb514e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0xb514e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0xb514e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0xb514e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:CentOS Linux release 7.5.1804 (Core)
uname:Linux 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE infinity, NPROC 1048576, NOFILE 1048576, AS infinity, DATA infinity, FSIZE infinity
load average:0.48 1.16 1.00

/proc/meminfo:
MemTotal:        7876552 kB
MemFree:         1727588 kB
MemAvailable:    3184348 kB
Buffers:           73012 kB
Cached:          1497432 kB
SwapCached:            0 kB
Active:          4694876 kB
Inactive:        1150576 kB
Active(anon):    4275856 kB
Inactive(anon):     1952 kB
Active(file):     419020 kB
Inactive(file):  1148624 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             11684 kB
Writeback:             0 kB
AnonPages:       4273520 kB
Mapped:           320080 kB
Shmem:              2720 kB
Slab:             235824 kB
SReclaimable:     193912 kB
SUnreclaim:        41912 kB
KernelStack:        8592 kB
PageTables:        15824 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3938276 kB
Committed_AS:    6391232 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      141288 kB
DirectMap2M:     6891520 kB
DirectMap1G:     1048576 kB

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-1
cpu_memory_nodes: 0
active_processor_count: 1
cpu_quota: 50000
cpu_period: 100000
cpu_shares: 512
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -2
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 2799579136
memory_max_usage_in_bytes: 2803335168


CPU:total 2 (initial active 1) (2 cores per cpu, 2 threads per core) family 6 model 85 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx, evex, fma
CPU Model and flags from /proc/cpuinfo:
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

Memory: 4k page, physical 7876552k(1727588k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (10.0.1+10) for linux-amd64 JRE (10.0.1+10), built on Mar 26 2018 18:10:26 by "mach5one" with gcc 4.9.2

END.

jpountz · 2018-06-20T11:49:26Z

Thanks. For the record, I asked on the Lucene lists as well in case someone already saw this segfault: https://markmail.org/message/n7rjurxzinyrhrwb.

WUMUXIAN · 2018-06-22T04:55:46Z

@jpountz Thanks for the info.

lachlancooper · 2018-06-29T06:44:26Z

I'm also encountering this error in a similar environment: official docker image 6.3.0, m5.large AWS EC2 instance, Ubuntu 16.04. The previous version I was running, 6.2.4, was not affected. I'm happy to provide more info if required.

xrl · 2018-06-29T23:16:44Z

I am also seeing this error on a m5.4xlarge in AWS. Linux 4.9. Debian stretch on the host. Using the docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.0 image.

SMAPPER · 2018-07-02T01:59:07Z

I am also experiencing this issue with the elasticsearch:6.3.0 docker image.

xose · 2018-07-04T14:09:55Z

I started experiencing this crash today:

I've been running the 6.3.0 Docker image on AWS i3 instances for weeks without any crashes. Today I launched some c5d instances to help with some hot indexes, and every single one of those has been crashing every couple minutes when I started indexing on them. I have since moved all indexes back to the i3s but I'll try to fetch some logs, although they were nearly identical to the ones already provided here.

Based on the comments above, this might be an issue with 6.3.0 and the M5/C5 instance family on AWS. Please let me know if I can help debug this in some way.

OS: Amazon Linux ECS 2018.03.a

xose · 2018-07-04T16:29:46Z

I've been doing some tests, and I believe I found the issue.

The 6.3.0 Docker image ships with Java 10.0.1, more specifically the generic binary targz from Oracle. I modified the Docker image to use the OpenJDK 8 binary provided by the base CentOS image package manager (1.8.0.171). It's now been running for ~1h without a single crash.

So the issue seems to be with Java 10 on AWS C5/M5 instances.

jpountz · 2018-07-11T07:04:55Z

Thanks for digging @xose ! @dliappis and I tried to reproduce this one on a c5d.large instance with Amazon Linux ECS 2018.03.a, Elasticsearch 6.3.0 and OpenJDK 10.0.1 by running various rally tracks, but this didn't reproduce. There might be other conditions that are needed for this bug to reproduce. Given that the error occurs when merging the list of field names that are indexed, it might depend on the actual list of field names that exist in the index.

shaunstone · 2018-07-13T03:03:09Z

sharing my experience.

using the 6.3 docker image on a m5.large.
I have no indexes other than the .monitoring and .kibana indexes.
i am attempting to create an index and get repeated failures due to synonym issues (which is a seperate issue) eventually I have gotten the segfault so far 4 different times in a matter of a few hours

HitkoDev · 2018-07-16T09:15:53Z

I've been experiencing a similar problem, running Elastic on m5 instance. However, unlike most people here, I am only experiencing it with 6.3.1 docker image; 6.3.0 is running fine, even though both images use JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/10.0.1/10.0.1+10].

jasontedor · 2018-07-16T21:13:38Z

For the users that are reporting this issue, would you please try the following as a workaround? Add the options -XX:CompileCommand=exclude,org/apache/lucene/util/MergedIterator.pullTop and -XX:CompileCommand=exclude,org/apache/lucene/util/MergedIterator.pushTop to either jvm.options or via ES_JAVA_OPTS and let us know if the issue continues to reproduce?

When Elasticsearch starts you should see the log lines

CompileCommand: exclude org/apache/lucene/util/MergedIterator.pullTop
CompileCommand: exclude org/apache/lucene/util/MergedIterator.pushTop

as an indication that the options were successfully applied. If this eliminates the issue, this would point to a compilation bug in JDK 10. Meanwhile, I am taking this issue upstream.

jasontedor · 2018-07-16T22:14:16Z

Also, another option to try as this appears to be manifesting on server-grade Skylake chips: start with the JVM option -XX:UseAVX=2; this will disable AVX-512. This is separate from the above, so if you can try this without the compile command options from the previous comment and report whether or not the issue reproduces that would be useful to know.

jasontedor · 2018-07-17T01:22:25Z

I note that the disassembly for the instructions given above from the hserr log

0x00007f8c43de5351:   98 ff 0f 83 14 01 00 00 48 83 c0 40 83 ea 40 75
0x00007f8c43de5361:   de 62 a1 7d a1 ef c0 eb 64 62 e1 7e a9 6f 04 07
0x00007f8c43de5371:   62 e1 7d a1 ef 04 06 48 83 ec 40 62 f1 7e 49 7f
0x00007f8c43de5381:   04 24 48 83 ec 40 62 f1 7e 49 7f 0c 24 62 b1 7e

is

0:  98                      cwde
1:  ff 0f                   dec    DWORD PTR [rdi]
3:  83 14 01 00             adc    DWORD PTR [rcx+rax*1],0x0
7:  00 48 83                add    BYTE PTR [rax-0x7d],cl
a:  c0 40 83 ea             rol    BYTE PTR [rax-0x7d],0xea
e:  40 75 de                rex jne 0xffffffffffffffef
11: 62 a1 7d a1 ef c0       vpxord ymm16{k1}{z},ymm16,ymm16
17: eb 64                   jmp    0x7d
19: 62 e1 7e a9 6f 04 07    vmovdqu32 ymm16{k1}{z},YMMWORD PTR [rdi+rax*1]
20: 62 e1 7d a1 ef 04 06    vpxord ymm16{k1}{z},ymm16,YMMWORD PTR [rsi+rax*1]
27: 48 83 ec 40             sub    rsp,0x40
2b: 62 f1 7e 49 7f 04 24    vmovdqu32 ZMMWORD PTR [rsp]{k1},zmm0
32: 48 83 ec 40             sub    rsp,0x40
36: 62 f1 7e 49 7f 0c 24    vmovdqu32 ZMMWORD PTR [rsp]{k1},zmm1
3d: 62                      .byte 0x62
3e: b1 7e                   mov    cl,0x7e

which shows that AVX-512 instructions are indeed in play here. In particular, the instruction that we crashed on is:

62 e1 7d a1 ef 04 06    vpxord ymm16{k1}{z},ymm16,YMMWORD PTR [rsi+rax*1]

I really hope that someone that can reliably reproduce this issue can report whether or not the issue goes away if AVX-512 is disabled (via -XX:UseAVX=2)? 🙏

dsmitty166 · 2018-07-17T01:35:21Z

Currently setting -XX:CompileCommand=exclude,org/apache/lucene/util/MergedIterator.pullTop and -XX:CompileCommand=exclude,org/apache/lucene/util/MergedIterator.pushTop seems to help.

I can test AVX-512 if this remains stable. I'm running on a Intel(R) Xeon(R) Silver 4114 Proc

dsmitty166 · 2018-07-17T01:48:27Z

Update: tested with -XX:UseAVX=2 in ES_JAVA_OPTS and it seems stable. Been up for 10 mins now, typically would crash in less than 2. I will continue to monitor.

jasontedor · 2018-07-17T13:35:19Z

@dsmitty166 Thanks for checking and this is welcome news. I feel fairly comfortable asserting that the issue is due to a bug in the C2 compiler on JDK 10 when using AVX-512. Here are some more data points:

we know the bug disappears on JDK 8 <-- no support for AVX-512 until JDK 9
we do not know of any instance of the bug appearing on Elasticsearch 6.2 on JDK 9 <-- AVX-512 is disabled by default
we know the bug disappears on hardware that does not support AVX-512
it appears the bug disappears if we disable compilation for the methods in question, or if we disable AVX-512
we know the bug presents on Elasticsearch 6.3 on JDK 10 on Skylake X <-- AVX-512 is enabled by default
we know the offending instruction at the time of the segmentation fault is an AVX-512 instruction

I have taken this upstream and will link to the mail thread once it receives moderator approval. For now, I think we can say that a workaround is to use -XX:UseAVX=2.

jasontedor · 2018-07-17T19:51:17Z

We have integrated a change (#32138) into 6.3.2 that will disable AVX-512 on JDK 10. We will keep this disabled until there is an upstream patch to the JDK to address this issue.

jasontedor · 2018-07-17T20:30:05Z

This is now tracked upstream via JDK-8207746.

iwanowww · 2018-07-18T14:21:51Z

Is there a way to reproduce the crash? Unfortunately, the log doesn't contain enough info to spot the root cause. I'll be grateful for instructions how to trigger it. Thanks in advance!

strawgate · 2018-07-19T00:45:51Z

I'm running on a Intel(R) Xeon(R) Silver 4116 Proc (in docker for Windows) and was having this issue.

Appears to stop happening when -XX:UseAVX=2 is specified.

jasontedor · 2018-07-24T03:42:17Z

@iwanowww Thanks for your support. You can reproduce this issue as follows:

on an Ubuntu machine that support AVX-512
install docker-ce 18.06.0ce3-0~ubuntu
install docker-compose 1.22: curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s_-$(uname -m) -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose
clone elastic/apm-integration-testing
run sudo ./scripts/compose.py start 6.3.1 --release --no-kibana --with-opbeans-node

The Elasticsearch container localtesting_6.3.1_elasticsearch will die after approximately ten minutes with the reported segmentation fault. Sorry that the steps are so involved and we do not have a truly minimal reproducer. We have tried to reduce this to simpler cases yet every attempt has led to the issue not reproducing so we are not sure the exact circumstances under which the failure occurs, it seems dependent on the data being pushed to Elasticsearch. If there is information that we can obtain that would make this easier for you, please let us know. We would be happy to do the legwork here.

jasontedor · 2018-07-27T18:14:46Z

@iwanowww I have also tested this on JDK 11 (11-ea+24) and the crash continues to reproduce when AVX-512 is enabled and does not reproduce when AVX-512 is disabled (-XX:UseAVX=2).

iwanowww · 2018-08-03T19:26:56Z

Thanks a lot for the info, @jasontedor!

vivdesh · 2018-08-08T19:46:08Z

@jasontedor Hi Jason, I am getting stuck at step 3/7 for npm install pm2 -g. Is there any way we can bypass that?

jasontedor · 2018-08-08T20:13:56Z

@vivdesh What is the issue you’re having?

vivdesh · 2018-08-08T20:51:15Z

@jasontedor
npm ERR! network request to https://registry.npmjs.org/pm2 failed, reason: connect ETIMEDOUT 104.16.21.35:443
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.

I have set the right proxies and registry, but still having the issue

vivdesh · 2018-08-09T23:33:11Z

@jasontedor Hi Jason, I have it running on skylake server after setting the proxies inside the server.
Now I would like to change the jdk to jdk11 from inside the docker container. Is there any easy way to set JAVA_HOME inside the container on startup?

jasontedor · 2018-08-15T18:06:14Z

@vivdesh It can be tricky. The best option would be to create a new image based on JDK 11. I have done this in my own testing when I reported above that the issue still persists on JDK 11. Is this something that you would repeatedly need to do, or are you looking for a single build off of the latest JDK 11-ea build? If the latter, I think that the most straightforward approach would be that I build a new image for you and point you how to use it in the reproduction. If the former, we can come up with something.

vivdesh · 2018-08-15T18:35:24Z

@jasontedor I was able to reproduce the problem with jdk11 by mounting the local jdk image and have the fix. You can check the progress: https://bugs.openjdk.java.net/browse/JDK-8207746. Thanks.

jasontedor · 2018-08-15T18:46:44Z

@vivdesh That is great news, thank you! If I understand correctly, this will be fixed in JDK 11 but remain a problem in JDK 10?

vivdesh · 2018-08-17T23:04:45Z

@jasontedor Yes. this is fixed in the jdk11. http://hg.openjdk.java.net/jdk/jdk11/rev/7339b9e38182
Could be backported to jdk10u, but not sure about it.

jasontedor · 2018-08-17T23:48:34Z

Thanks for your support @vivdesh. It’s no problem for us if this problem remains in JDK 10, we have disabled by default the use of AVX-512 there, and will re-enable it for JDK 11.

elastic/elasticsearch#31425

Suggesting update to a higher version of elasticsearch, (one thats not running on JDK10) due to an issue related to JVM inside JDK10. ref elastic/elasticsearch#31425 (comment)

LiamKarlMitchell · 2020-07-07T00:36:24Z

So where do I put -XX:UseAVX=2 to make this work?
Using elastic search 6.8.3 in a docker container.
Works fine first after build but after a docker-compose down and up it does not work anymore.

Do I put it in the jvm options config file?
/usr/share/elasticsearch/config/jvm.options

jasontedor · 2020-07-07T00:51:13Z

@LiamKarlMitchell Are you using the official 6.8.3 Docker image? If so, I don't think you can be experiencing this issue because that container is based on JDK 12 where this issue is mitigated. If not, please use the official image and we will support you. Either way, please use the forums for additional help.

LiamKarlMitchell · 2020-07-07T02:57:05Z

I did a rebuild and deleted volumes.
I extend with own Dockerfile from the official image just to add the icu and phonetic plugins.
The docker-compose file has a local volume for the esdata.

FROM elasticsearch:6.8.3
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu && \
    /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-phonetic

After destroying the volume and rebuilding it looks like it works :).
Thanks

vladimirdolzhenko added the jvm bug label Jun 19, 2018

colings86 added the :Core/Infra/Core Core issues without another label label Jun 20, 2018

xose mentioned this issue Jul 4, 2018

Java 10 segfaults on latest generation AWS instances elastic/elasticsearch-docker#179

Closed

This was referenced Jul 17, 2018

Add known issue with AVX-512 and JDK 10 #32135

Merged

Disable C2 from using AVX-512 on JDK 10 #32138

Merged

jasontedor closed this as completed Aug 17, 2018

QingsongYao pushed a commit to QingsongYao/charts that referenced this issue Aug 20, 2018

fix ES issue related to java 10 by using -XX:UseAVX=2

5cdfe50

elastic/elasticsearch#31425

QingsongYao mentioned this issue Aug 20, 2018

fix ES issue related to java 10 by using -XX:UseAVX=2 helm/charts#7253

Closed

Inzagio mentioned this issue Jun 26, 2019

Update docker-compose.yml martinjt/logging-ndcoslo#1

Open

org.apache.lucene.util.MergedIterator.pullTop() Crash #31425

org.apache.lucene.util.MergedIterator.pullTop() Crash #31425

Comments

WUMUXIAN commented Jun 19, 2018

vladimirdolzhenko commented Jun 19, 2018

ywelsch commented Jun 19, 2018

WUMUXIAN commented Jun 20, 2018

jpountz commented Jun 20, 2018

WUMUXIAN commented Jun 20, 2018

jpountz commented Jun 20, 2018

WUMUXIAN commented Jun 22, 2018

lachlancooper commented Jun 29, 2018

xrl commented Jun 29, 2018

SMAPPER commented Jul 2, 2018

xose commented Jul 4, 2018

xose commented Jul 4, 2018

jpountz commented Jul 11, 2018

shaunstone commented Jul 13, 2018

HitkoDev commented Jul 16, 2018

jasontedor commented Jul 16, 2018

jasontedor commented Jul 16, 2018

jasontedor commented Jul 17, 2018 • edited

dsmitty166 commented Jul 17, 2018

dsmitty166 commented Jul 17, 2018

jasontedor commented Jul 17, 2018

jasontedor commented Jul 17, 2018

jasontedor commented Jul 17, 2018

iwanowww commented Jul 18, 2018

strawgate commented Jul 19, 2018 • edited

jasontedor commented Jul 24, 2018

jasontedor commented Jul 27, 2018

iwanowww commented Aug 3, 2018

vivdesh commented Aug 8, 2018

jasontedor commented Aug 8, 2018

vivdesh commented Aug 8, 2018

vivdesh commented Aug 9, 2018

jasontedor commented Aug 15, 2018

vivdesh commented Aug 15, 2018

jasontedor commented Aug 15, 2018

vivdesh commented Aug 17, 2018

jasontedor commented Aug 17, 2018

LiamKarlMitchell commented Jul 7, 2020 • edited

jasontedor commented Jul 7, 2020

LiamKarlMitchell commented Jul 7, 2020

jasontedor commented Jul 17, 2018 •

edited

strawgate commented Jul 19, 2018 •

edited

LiamKarlMitchell commented Jul 7, 2020 •

edited