[CI] UpdateNumberOfReplicasIT.testAutoExpandNumberOfReplicas0ToData fails #25335
Description
The test UpdateNumberOfReplicasIT.testAutoExpandNumberOfReplicas0ToData failed on CI:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/2965/consoleFull
It doesn't reproduce locally with the same JVM versions:
gradle :core:integTest -Dtests.seed=A93BDC488D12ACBB -Dtests.class=org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT -Dtests.method="testAutoExpandNumberOfReplicas0ToData" -Dtests.security.manager=true -Dtests.jvm.argline="--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED" -Dtests.locale=rof-TZ -Dtests.timezone=US/Michigan
and it seems to be related to a missing file:
Caused by: java.nio.file.NoSuchFileException: /var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/0/indices/s8cWmM-JQRO3FIjBctPQYg/1/_state/state-1.st
Suite: org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT
1> [2017-06-21T14:07:31,877][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
1> [2017-06-21T14:07:31,879][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
1> [2017-06-21T14:07:31,879][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
1> [2017-06-21T14:07:31,879][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
1> # allow user 'jenkins' mlockall
1> jenkins soft memlock unlimited
1> jenkins hard memlock unlimited
1> [2017-06-21T14:07:31,880][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
1> [2017-06-21T08:07:33,245][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testAutoExpandNumberOfReplicas0ToData]: before test
1> [2017-06-21T08:07:33,261][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberOfReplicas0ToData]: setting up test
1> [2017-06-21T08:07:33,286][INFO ][o.e.t.InternalTestCluster] Setup InternalTestCluster [SUITE-CHILD_VM=[1]-CLUSTER_SEED=[1620327433514336453]-HASH=[A282E4C4284B]-cluster] with seed [167C8F37467140C5] using [0] dedicated masters, [2] (data) nodes and [0] coord only nodes (min_master_nodes are [auto-managed])
1> [2017-06-21T08:07:33,604][INFO ][o.e.n.Node ] [node_s0] initializing ...
1> [2017-06-21T08:07:33,673][INFO ][o.e.e.NodeEnvironment ] [node_s0] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:33,673][INFO ][o.e.e.NodeEnvironment ] [node_s0] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:33,674][INFO ][o.e.n.Node ] [node_s0] node name [node_s0], node ID [E84dTZseTwCU-8paQEqsww]
1> [2017-06-21T08:07:33,675][INFO ][o.e.n.Node ] [node_s0] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:33,675][INFO ][o.e.n.Node ] [node_s0] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:33,676][WARN ][o.e.n.Node ] [node_s0] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:33,683][INFO ][o.e.p.PluginsService ] [node_s0] no modules loaded
1> [2017-06-21T08:07:33,684][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
1> [2017-06-21T08:07:33,684][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:33,684][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
1> [2017-06-21T08:07:33,684][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:33,684][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:35,564][INFO ][o.e.d.DiscoveryModule ] [node_s0] using discovery type [test-zen]
1> [2017-06-21T08:07:35,938][INFO ][o.e.n.Node ] [node_s0] initialized
1> [2017-06-21T08:07:35,950][INFO ][o.e.n.Node ] [node_s1] initializing ...
1> [2017-06-21T08:07:35,955][INFO ][o.e.e.NodeEnvironment ] [node_s1] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:35,955][INFO ][o.e.e.NodeEnvironment ] [node_s1] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:35,956][INFO ][o.e.n.Node ] [node_s1] node name [node_s1], node ID [ShA9dW0NTwOOv-YrA1jAmA]
1> [2017-06-21T08:07:35,956][INFO ][o.e.n.Node ] [node_s1] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:35,956][INFO ][o.e.n.Node ] [node_s1] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:35,957][WARN ][o.e.n.Node ] [node_s1] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:35,958][INFO ][o.e.p.PluginsService ] [node_s1] no modules loaded
1> [2017-06-21T08:07:35,958][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
1> [2017-06-21T08:07:35,958][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:35,958][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
1> [2017-06-21T08:07:35,959][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:35,959][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
2> M6 21, 2017 8:07:38 KANG’AMA com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
2> WARNING: Uncaught exception in thread: Thread[elasticsearch[node_s0][clusterApplierService#updateTask][T#1],5,TGRP-UpdateNumberOfReplicasIT]
1> [2017-06-21T08:07:36,009][INFO ][o.e.d.DiscoveryModule ] [node_s1] using discovery type [test-zen]
2> java.lang.AssertionError
2> at __randomizedtesting.SeedInfo.seed([A93BDC488D12ACBB]:0)
2> at org.elasticsearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:460)
2> at org.elasticsearch.env.NodeEnvironment.deleteShardDirectorySafe(NodeEnvironment.java:400)
2> at org.elasticsearch.indices.IndicesService.deleteShardStore(IndicesService.java:700)
2> at org.elasticsearch.indices.store.IndicesStore$ShardActiveResponseHandler.lambda$allNodesResponded$2(IndicesStore.java:289)
1> [2017-06-21T08:07:36,154][INFO ][o.e.n.Node ] [node_s1] initialized
2> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$runOnApplierThread$0(ClusterApplierService.java:301)
2> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.apply(ClusterApplierService.java:150)
1> [2017-06-21T08:07:36,188][INFO ][o.e.n.Node ] [node_s1] starting ...
2> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:394)
1> [2017-06-21T08:07:36,188][INFO ][o.e.n.Node ] [node_s0] starting ...
2> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:155)
2> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
1> [2017-06-21T08:07:36,283][INFO ][o.e.t.t.MockTransportService] [node_s0] publish_address {127.0.0.1:9500}, bound_addresses {[::1]:9500}, {127.0.0.1:9500}
1> [2017-06-21T08:07:36,284][INFO ][o.e.t.t.MockTransportService] [node_s1] publish_address {127.0.0.1:9501}, bound_addresses {[::1]:9501}, {127.0.0.1:9501}
2> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247)
1> [2017-06-21T08:07:36,355][INFO ][o.e.t.d.MockZenPing ] [node_s0] pinging using mock zen ping
2> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210)
2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161)
1> [2017-06-21T08:07:36,373][INFO ][o.e.t.d.MockZenPing ] [node_s1] pinging using mock zen ping
2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2> at java.base/java.lang.Thread.run(Thread.java:844)
1> [2017-06-21T08:07:36,673][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-elected-as-master ([1] nodes joined)[{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501}], reason: new_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501},}
1> [2017-06-21T08:07:36,702][INFO ][o.e.c.s.ClusterApplierService] [node_s1] detected_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [1]])
2> REPRODUCE WITH: gradle :core:integTest -Dtests.seed=A93BDC488D12ACBB -Dtests.class=org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT -Dtests.method="testAutoExpandNumberOfReplicas0ToData" -Dtests.security.manager=true -Dtests.jvm.argline="--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED" -Dtests.locale=rof-TZ -Dtests.timezone=US/Michigan
1> [2017-06-21T08:07:36,706][INFO ][o.e.n.Node ] [node_s1] started
1> [2017-06-21T08:07:36,709][INFO ][o.e.c.s.ClusterApplierService] [node_s0] new_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [1] source [zen-disco-elected-as-master ([1] nodes joined)[{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501}]]])
1> [2017-06-21T08:07:36,719][INFO ][o.e.n.Node ] [node_s0] started
1> [2017-06-21T08:07:36,786][INFO ][o.e.g.GatewayService ] [node_s0] recovered [0] indices into cluster_state
1> [2017-06-21T08:07:36,791][INFO ][o.e.p.PluginsService ] [transport_client_node_s0] no modules loaded
1> [2017-06-21T08:07:36,791][INFO ][o.e.p.PluginsService ] [transport_client_node_s0] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:36,900][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] test using _default_ mappings: [{"_default_":{}}]
1> [2017-06-21T08:07:37,146][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberOfReplicas0ToData]: all set up test
1> [2017-06-21T08:07:37,147][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> creating index test with auto expand replicas
1> [2017-06-21T08:07:37,150][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] using custom data_path for index: [bHwAGehNRp]
1> [2017-06-21T08:07:37,241][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [test] creating index, cause [api], templates [random_index_template], shards [3]/[1], mappings [_default_]
1> [2017-06-21T08:07:37,647][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:37,944][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][2], [test][0]] ...]).
1> [2017-06-21T08:07:37,962][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:37,962][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> add another node, should increase the number of replicas
1> [2017-06-21T08:07:37,962][INFO ][o.e.t.InternalTestCluster] increasing cluster size from 2 to 3
1> [2017-06-21T08:07:37,965][INFO ][o.e.n.Node ] [node_s2] initializing ...
1> [2017-06-21T08:07:37,975][INFO ][o.e.e.NodeEnvironment ] [node_s2] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:37,975][INFO ][o.e.e.NodeEnvironment ] [node_s2] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:37,976][INFO ][o.e.n.Node ] [node_s2] node name [node_s2], node ID [-Hhg2w1TR8mPPySaYZc4_A]
1> [2017-06-21T08:07:37,976][INFO ][o.e.n.Node ] [node_s2] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:37,976][INFO ][o.e.n.Node ] [node_s2] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:37,976][WARN ][o.e.n.Node ] [node_s2] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:37,977][INFO ][o.e.p.PluginsService ] [node_s2] no modules loaded
1> [2017-06-21T08:07:37,977][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
1> [2017-06-21T08:07:37,978][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:37,978][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
1> [2017-06-21T08:07:37,978][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:37,978][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:38,080][INFO ][o.e.d.DiscoveryModule ] [node_s2] using discovery type [test-zen]
1> [2017-06-21T08:07:38,218][INFO ][o.e.n.Node ] [node_s2] initialized
1> [2017-06-21T08:07:38,219][INFO ][o.e.p.PluginsService ] [transport_client_node_s1] no modules loaded
1> [2017-06-21T08:07:38,219][INFO ][o.e.p.PluginsService ] [transport_client_node_s1] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:38,455][INFO ][o.e.n.Node ] [node_s2] starting ...
1> [2017-06-21T08:07:38,466][INFO ][o.e.t.t.MockTransportService] [node_s2] publish_address {127.0.0.1:9502}, bound_addresses {[::1]:9502}, {127.0.0.1:9502}
1> [2017-06-21T08:07:38,467][INFO ][o.e.t.d.MockZenPing ] [node_s2] pinging using mock zen ping
1> [2017-06-21T08:07:38,586][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-join[{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502}], reason: added {{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502},}
1> [2017-06-21T08:07:38,592][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [10]])
1> [2017-06-21T08:07:38,612][INFO ][o.e.c.s.ClusterApplierService] [node_s2] detected_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500},{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [10]])
1> [2017-06-21T08:07:38,750][INFO ][o.e.n.Node ] [node_s2] started
1> [2017-06-21T08:07:38,758][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [10] source [zen-disco-node-join[{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502}]]])
1> [2017-06-21T08:07:38,824][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [2] for indices [test]
1> [2017-06-21T08:07:38,872][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/s8cWmM-JQRO3FIjBctPQYg] auto expanded replicas to [2]
1> [2017-06-21T08:07:38,862][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [node_s0] [test][1]: failed to list shard for shard_store on node [E84dTZseTwCU-8paQEqsww]
1> org.elasticsearch.action.FailedNodeException: Failed node [E84dTZseTwCU-8paQEqsww]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1047) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1151) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1129) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:648) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:623) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[main/:?]
1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161) ~[?:?]
1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
1> at java.lang.Thread.run(Thread.java:844) ~[?:?]
1> Caused by: org.elasticsearch.transport.RemoteTransportException: [node_s0][127.0.0.1:9500][internal:cluster/nodes/indices/shard/store[n]]
1> Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:1, legacy:false, file:/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/0/indices/s8cWmM-JQRO3FIjBctPQYg/1/_state/state-1.st]
1> at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:150) ~[main/:?]
1> at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:334) ~[main/:?]
1> at org.elasticsearch.index.shard.ShardPath.loadShardPath(ShardPath.java:118) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:142) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[main/:?]
1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:637) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> ... 3 more
1> Caused by: java.io.IOException: failed to read [id:1, legacy:false, file:/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/0/indices/s8cWmM-JQRO3FIjBctPQYg/1/_state/state-1.st]
1> at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:327) ~[main/:?]
1> at org.elasticsearch.index.shard.ShardPath.loadShardPath(ShardPath.java:118) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:142) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[main/:?]
1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:637) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> ... 3 more
1> Caused by: java.nio.file.NoSuchFileException: /var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/0/indices/s8cWmM-JQRO3FIjBctPQYg/1/_state/state-1.st
1> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
1> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
1> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
1> at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215) ~[?:?]
1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) ~[lucene-test-framework-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:04]
1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) ~[lucene-test-framework-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:04]
1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) ~[lucene-test-framework-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:04]
1> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240) ~[lucene-test-framework-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:04]
1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) ~[lucene-test-framework-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:04]
1> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240) ~[lucene-test-framework-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:04]
1> at java.nio.file.Files.newByteChannel(Files.java:369) ~[?:?]
1> at java.nio.file.Files.newByteChannel(Files.java:415) ~[?:?]
1> at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) ~[lucene-core-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:02]
1> at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:187) ~[main/:?]
1> at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:322) ~[main/:?]
1> at org.elasticsearch.index.shard.ShardPath.loadShardPath(ShardPath.java:118) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:142) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[main/:?]
1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:637) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> ... 3 more
1> [2017-06-21T08:07:38,942][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:38,943][INFO ][o.e.p.PluginsService ] [transport_client_node_s2] no modules loaded
1> [2017-06-21T08:07:38,944][INFO ][o.e.p.PluginsService ] [transport_client_node_s2] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:38,955][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [2] for indices [test]
1> [2017-06-21T08:07:39,006][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/s8cWmM-JQRO3FIjBctPQYg] auto expanded replicas to [2]
1> [2017-06-21T08:07:39,190][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]] ...]).
1> [2017-06-21T08:07:39,211][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:39,211][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> closing one node
1> [2017-06-21T08:07:39,216][INFO ][o.e.t.InternalTestCluster] changing cluster size from 3 data nodes to 2
1> [2017-06-21T08:07:39,234][INFO ][o.e.n.Node ] [node_s1] stopping ...
1> [2017-06-21T08:07:39,298][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501} left]).
1> [2017-06-21T08:07:39,298][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-left({node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501}), reason(left)[{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501} left], reason: removed {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501},}
1> [2017-06-21T08:07:39,301][INFO ][o.e.n.Node ] [node_s1] stopped
1> [2017-06-21T08:07:39,302][INFO ][o.e.n.Node ] [node_s1] closing ...
1> [2017-06-21T08:07:39,306][INFO ][o.e.c.s.ClusterApplierService] [node_s2] removed {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [19]])
1> [2017-06-21T08:07:39,312][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [19] source [zen-disco-node-left({node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501}), reason(left)[{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{DdJdNTY5SZS9TIv0wiJS3Q}{127.0.0.1}{127.0.0.1:9501} left]]])
1> [2017-06-21T08:07:39,315][INFO ][o.e.n.Node ] [node_s1] closed
1> [2017-06-21T08:07:39,324][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [test]
1> [2017-06-21T08:07:39,338][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/s8cWmM-JQRO3FIjBctPQYg] auto expanded replicas to [1]
1> [2017-06-21T08:07:39,388][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:39,389][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:39,389][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> closing another node
1> [2017-06-21T08:07:39,391][INFO ][o.e.t.InternalTestCluster] changing cluster size from 2 data nodes to 1
1> [2017-06-21T08:07:39,398][INFO ][o.e.c.s.ClusterSettings ] [node_s2] updating [discovery.zen.minimum_master_nodes] from [2] to [1]
1> [2017-06-21T08:07:39,401][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [discovery.zen.minimum_master_nodes] from [2] to [1]
1> [2017-06-21T08:07:39,413][INFO ][o.e.n.Node ] [node_s2] stopping ...
1> [2017-06-21T08:07:39,427][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502} left, {node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502} transport disconnected]).
1> [2017-06-21T08:07:39,428][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-failed({node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502}), reason(transport disconnected)[{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502} transport disconnected], zen-disco-node-left({node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502}), reason(left)[{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502} left], reason: removed {{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502},}
1> [2017-06-21T08:07:39,439][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [23] source [zen-disco-node-failed({node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502}), reason(transport disconnected)[{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502} transport disconnected], zen-disco-node-left({node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502}), reason(left)[{node_s2}{-Hhg2w1TR8mPPySaYZc4_A}{ubQ0chZ4RVCjVKnL3KziDg}{127.0.0.1}{127.0.0.1:9502} left]]])
1> [2017-06-21T08:07:39,443][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [0] for indices [test]
1> [2017-06-21T08:07:39,464][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/s8cWmM-JQRO3FIjBctPQYg] auto expanded replicas to [0]
1> [2017-06-21T08:07:39,488][INFO ][o.e.n.Node ] [node_s2] stopped
1> [2017-06-21T08:07:39,489][INFO ][o.e.n.Node ] [node_s2] closing ...
1> [2017-06-21T08:07:39,491][INFO ][o.e.n.Node ] [node_s2] closed
1> [2017-06-21T08:07:39,523][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:39,524][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:39,524][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberOfReplicas0ToData]: cleaning up after test
1> [2017-06-21T08:07:39,560][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [test/s8cWmM-JQRO3FIjBctPQYg] deleting index
1> [2017-06-21T08:07:39,623][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberOfReplicas0ToData]: cleaned up after test
1> [2017-06-21T08:07:39,626][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testAutoExpandNumberOfReplicas0ToData]: after test
ERROR 6.49s J1 | UpdateNumberOfReplicasIT.testAutoExpandNumberOfReplicas0ToData <<< FAILURES!
> Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=29, name=elasticsearch[node_s0][clusterApplierService#updateTask][T#1], state=RUNNABLE, group=TGRP-UpdateNumberOfReplicasIT]
> at __randomizedtesting.SeedInfo.seed([A93BDC488D12ACBB:5A9B8BCA00001CE0]:0)
> Caused by: java.lang.AssertionError
> at __randomizedtesting.SeedInfo.seed([A93BDC488D12ACBB]:0)
> at org.elasticsearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:460)
> at org.elasticsearch.env.NodeEnvironment.deleteShardDirectorySafe(NodeEnvironment.java:400)
> at org.elasticsearch.indices.IndicesService.deleteShardStore(IndicesService.java:700)
> at org.elasticsearch.indices.store.IndicesStore$ShardActiveResponseHandler.lambda$allNodesResponded$2(IndicesStore.java:289)
> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$runOnApplierThread$0(ClusterApplierService.java:301)
> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.apply(ClusterApplierService.java:150)
> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:394)
> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:155)
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247)
> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:844)
1> [2017-06-21T08:07:39,680][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testSimpleUpdateNumberOfReplicas]: before test
1> [2017-06-21T08:07:39,680][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testSimpleUpdateNumberOfReplicas]: setting up test
1> [2017-06-21T08:07:39,683][INFO ][o.e.t.InternalTestCluster] Successfully wiped data directory for node location: /var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/2
1> [2017-06-21T08:07:39,686][INFO ][o.e.t.InternalTestCluster] Successfully wiped data directory for node location: /var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/1
1> [2017-06-21T08:07:39,700][INFO ][o.e.n.Node ] [node_s1] initializing ...
1> [2017-06-21T08:07:39,705][INFO ][o.e.e.NodeEnvironment ] [node_s1] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:39,705][INFO ][o.e.e.NodeEnvironment ] [node_s1] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:39,706][INFO ][o.e.n.Node ] [node_s1] node name [node_s1], node ID [ShA9dW0NTwOOv-YrA1jAmA]
1> [2017-06-21T08:07:39,706][INFO ][o.e.n.Node ] [node_s1] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:39,706][INFO ][o.e.n.Node ] [node_s1] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:39,707][WARN ][o.e.n.Node ] [node_s1] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:39,708][INFO ][o.e.p.PluginsService ] [node_s1] no modules loaded
1> [2017-06-21T08:07:39,708][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
1> [2017-06-21T08:07:39,708][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:39,708][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
1> [2017-06-21T08:07:39,709][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:39,710][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:39,755][INFO ][o.e.d.DiscoveryModule ] [node_s1] using discovery type [test-zen]
1> [2017-06-21T08:07:39,829][INFO ][o.e.n.Node ] [node_s1] initialized
1> [2017-06-21T08:07:39,830][INFO ][o.e.n.Node ] [node_s1] starting ...
1> [2017-06-21T08:07:39,836][INFO ][o.e.t.t.MockTransportService] [node_s1] publish_address {127.0.0.1:9501}, bound_addresses {[::1]:9501}, {127.0.0.1:9501}
1> [2017-06-21T08:07:39,838][INFO ][o.e.t.d.MockZenPing ] [node_s1] pinging using mock zen ping
1> [2017-06-21T08:07:39,880][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-join[{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501}], reason: added {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},}
1> [2017-06-21T08:07:39,891][INFO ][o.e.c.s.ClusterApplierService] [node_s1] detected_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [28]])
1> [2017-06-21T08:07:39,893][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [discovery.zen.minimum_master_nodes] from [2] to [1]
1> [2017-06-21T08:07:39,895][INFO ][o.e.n.Node ] [node_s1] started
1> [2017-06-21T08:07:39,897][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [28] source [zen-disco-node-join[{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501}]]])
1> [2017-06-21T08:07:39,898][WARN ][o.e.d.z.ElectMasterService] [node_s0] value for setting "discovery.zen.minimum_master_nodes" is too low. This can result in data loss! Please set it to at least a quorum of master-eligible nodes (current value: [1], total number of master-eligible nodes used for publishing in this round: [2])
1> [2017-06-21T08:07:39,904][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [discovery.zen.minimum_master_nodes] from [1] to [2]
1> [2017-06-21T08:07:39,905][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [discovery.zen.minimum_master_nodes] from [1] to [2]
1> [2017-06-21T08:07:39,910][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] test using _default_ mappings: [{"_default_":{}}]
1> [2017-06-21T08:07:39,924][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testSimpleUpdateNumberOfReplicas]: all set up test
1> [2017-06-21T08:07:39,924][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Creating index test
1> [2017-06-21T08:07:39,924][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] using custom data_path for index: [ElssAyjCxQ]
1> [2017-06-21T08:07:39,933][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [test] creating index, cause [api], templates [random_index_template], shards [3]/[0], mappings [_default_]
1> [2017-06-21T08:07:39,996][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]] ...]).
1> [2017-06-21T08:07:40,010][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Running Cluster Health
1> [2017-06-21T08:07:40,011][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Done Cluster Health, status GREEN
1> [2017-06-21T08:07:40,111][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [test/-zLwVm_WTFCAuSnGeSDmWA] create_mapping [type1]
1> [2017-06-21T08:07:40,684][INFO ][o.e.p.PluginsService ] [transport_client_node_s1] no modules loaded
1> [2017-06-21T08:07:40,685][INFO ][o.e.p.PluginsService ] [transport_client_node_s1] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:40,776][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Increasing the number of replicas from 1 to 2
1> [2017-06-21T08:07:40,777][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [2] for indices [test]
1> [2017-06-21T08:07:40,834][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Running Cluster Health
1> [2017-06-21T08:07:41,082][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Done Cluster Health, status YELLOW
1> [2017-06-21T08:07:41,082][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] starting another node to new replicas will be allocated to it
1> [2017-06-21T08:07:41,082][INFO ][o.e.t.InternalTestCluster] increasing cluster size from 2 to 3
1> [2017-06-21T08:07:41,084][INFO ][o.e.n.Node ] [node_s2] initializing ...
1> [2017-06-21T08:07:41,087][INFO ][o.e.e.NodeEnvironment ] [node_s2] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:41,088][INFO ][o.e.e.NodeEnvironment ] [node_s2] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:41,088][INFO ][o.e.n.Node ] [node_s2] node name [node_s2], node ID [Qu3r-51cRXCzJOC3xLPo6w]
1> [2017-06-21T08:07:41,088][INFO ][o.e.n.Node ] [node_s2] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:41,088][INFO ][o.e.n.Node ] [node_s2] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:41,088][WARN ][o.e.n.Node ] [node_s2] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:41,089][INFO ][o.e.p.PluginsService ] [node_s2] no modules loaded
1> [2017-06-21T08:07:41,089][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
2> NOTE: test params are: codec=Asserting(Lucene70): {_field_names=PostingsFormat(name=Asserting), value.keyword=PostingsFormat(name=Asserting), _id=PostingsFormat(name=Asserting), value=PostingsFormat(name=Asserting)}, docValues:{_seq_no=DocValuesFormat(name=Lucene70), value.keyword=DocValuesFormat(name=Lucene70), _primary_term=DocValuesFormat(name=Lucene70), _version=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=80, maxMBSortInHeap=6.848477618368005, sim=RandomSimilarity(queryNorm=false): {}, locale=rof-TZ, timezone=US/Michigan
1> [2017-06-21T08:07:41,089][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:41,089][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
2> NOTE: Linux 4.8.0-53-generic amd64/Oracle Corporation 9-ea (64-bit)/cpus=8,threads=1,free=386772232,total=536870912
1> [2017-06-21T08:07:41,089][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:41,089][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:41,136][INFO ][o.e.d.DiscoveryModule ] [node_s2] using discovery type [test-zen]
1> [2017-06-21T08:07:41,205][INFO ][o.e.n.Node ] [node_s2] initialized
2> NOTE: All tests run in this JVM: [UpdateNumberOfReplicasIT]
1> [2017-06-21T08:07:41,212][INFO ][o.e.n.Node ] [node_s2] starting ...
1> [2017-06-21T08:07:41,216][INFO ][o.e.t.t.MockTransportService] [node_s2] publish_address {127.0.0.1:9502}, bound_addresses {[::1]:9502}, {127.0.0.1:9502}
1> [2017-06-21T08:07:41,217][INFO ][o.e.t.d.MockZenPing ] [node_s2] pinging using mock zen ping
1> [2017-06-21T08:07:41,234][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-join[{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502}], reason: added {{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502},}
1> [2017-06-21T08:07:41,240][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [41]])
1> [2017-06-21T08:07:41,287][INFO ][o.e.c.s.ClusterApplierService] [node_s2] detected_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},{node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [41]])
1> [2017-06-21T08:07:41,306][INFO ][o.e.n.Node ] [node_s2] started
1> [2017-06-21T08:07:41,312][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [41] source [zen-disco-node-join[{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502}]]])
1> [2017-06-21T08:07:41,380][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Running Cluster Health
1> [2017-06-21T08:07:41,466][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][2]] ...]).
1> [2017-06-21T08:07:41,483][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Done Cluster Health, status GREEN
1> [2017-06-21T08:07:41,524][INFO ][o.e.p.PluginsService ] [transport_client_node_s2] no modules loaded
1> [2017-06-21T08:07:41,524][INFO ][o.e.p.PluginsService ] [transport_client_node_s2] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:41,581][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Decreasing number of replicas from 2 to 0
1> [2017-06-21T08:07:41,582][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [0] for indices [test]
1> [2017-06-21T08:07:41,662][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Running Cluster Health
1> [2017-06-21T08:07:41,732][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] Done Cluster Health, status GREEN
1> [2017-06-21T08:07:41,819][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testSimpleUpdateNumberOfReplicas]: cleaning up after test
1> [2017-06-21T08:07:41,841][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [test/-zLwVm_WTFCAuSnGeSDmWA] deleting index
1> [2017-06-21T08:07:41,888][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testSimpleUpdateNumberOfReplicas]: cleaned up after test
1> [2017-06-21T08:07:41,888][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testSimpleUpdateNumberOfReplicas]: after test
1> [2017-06-21T08:07:41,908][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testAutoExpandNumberReplicas2]: before test
1> [2017-06-21T08:07:41,908][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberReplicas2]: setting up test
1> [2017-06-21T08:07:41,924][INFO ][o.e.n.Node ] [node_s2] stopping ...
1> [2017-06-21T08:07:41,925][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-left({node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502}), reason(left)[{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502} left], reason: removed {{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502},}
1> [2017-06-21T08:07:41,928][INFO ][o.e.n.Node ] [node_s2] stopped
1> [2017-06-21T08:07:41,928][INFO ][o.e.n.Node ] [node_s2] closing ...
1> [2017-06-21T08:07:41,931][INFO ][o.e.n.Node ] [node_s2] closed
1> [2017-06-21T08:07:41,932][INFO ][o.e.t.InternalTestCluster] Successfully wiped data directory for node location: /var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/2
1> [2017-06-21T08:07:41,932][INFO ][o.e.c.s.ClusterApplierService] [node_s1] removed {{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [52]])
1> [2017-06-21T08:07:41,933][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [52] source [zen-disco-node-left({node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502}), reason(left)[{node_s2}{Qu3r-51cRXCzJOC3xLPo6w}{BqcbCAtARKuX6Kb47acGAA}{127.0.0.1}{127.0.0.1:9502} left]]])
1> [2017-06-21T08:07:41,942][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] test using _default_ mappings: [{"_default_":{}}]
1> [2017-06-21T08:07:41,968][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberReplicas2]: all set up test
1> [2017-06-21T08:07:41,968][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> creating index test with auto expand replicas set to 0-2
1> [2017-06-21T08:07:41,968][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] using custom data_path for index: [sufYyhvrob]
1> [2017-06-21T08:07:41,968][INFO ][o.e.t.InternalTestCluster] increasing cluster size from 2 to 3
1> [2017-06-21T08:07:41,970][INFO ][o.e.n.Node ] [node_s2] initializing ...
1> [2017-06-21T08:07:41,974][INFO ][o.e.e.NodeEnvironment ] [node_s2] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:41,974][INFO ][o.e.e.NodeEnvironment ] [node_s2] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:41,974][INFO ][o.e.n.Node ] [node_s2] node name [node_s2], node ID [AxNGFLjdQmyv5UPrUxLM2w]
1> [2017-06-21T08:07:41,975][INFO ][o.e.n.Node ] [node_s2] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:41,975][INFO ][o.e.n.Node ] [node_s2] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:41,975][WARN ][o.e.n.Node ] [node_s2] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:41,976][INFO ][o.e.p.PluginsService ] [node_s2] no modules loaded
1> [2017-06-21T08:07:41,976][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
1> [2017-06-21T08:07:41,976][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:41,976][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
1> [2017-06-21T08:07:41,976][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:41,976][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:42,024][INFO ][o.e.d.DiscoveryModule ] [node_s2] using discovery type [test-zen]
1> [2017-06-21T08:07:42,069][INFO ][o.e.n.Node ] [node_s2] initialized
1> [2017-06-21T08:07:42,095][INFO ][o.e.n.Node ] [node_s2] starting ...
1> [2017-06-21T08:07:42,102][INFO ][o.e.t.t.MockTransportService] [node_s2] publish_address {127.0.0.1:9502}, bound_addresses {[::1]:9502}, {127.0.0.1:9502}
1> [2017-06-21T08:07:42,114][INFO ][o.e.t.d.MockZenPing ] [node_s2] pinging using mock zen ping
1> [2017-06-21T08:07:42,129][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-join[{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502}], reason: added {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},}
1> [2017-06-21T08:07:42,130][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [55]])
1> [2017-06-21T08:07:42,134][INFO ][o.e.c.s.ClusterApplierService] [node_s2] detected_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},{node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [55]])
1> [2017-06-21T08:07:42,142][INFO ][o.e.n.Node ] [node_s2] started
1> [2017-06-21T08:07:42,143][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [55] source [zen-disco-node-join[{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502}]]])
1> [2017-06-21T08:07:42,151][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [test] creating index, cause [api], templates [random_index_template], shards [5]/[1], mappings [_default_]
1> [2017-06-21T08:07:42,212][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [2] for indices [test]
1> [2017-06-21T08:07:42,220][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/gnCPCG0dS4SZrXJrHJee4w] auto expanded replicas to [2]
1> [2017-06-21T08:07:42,320][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:42,325][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [2] for indices [test]
1> [2017-06-21T08:07:42,342][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/gnCPCG0dS4SZrXJrHJee4w] auto expanded replicas to [2]
1> [2017-06-21T08:07:42,447][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [node_s0] [test][3]: failed to list shard for shard_store on node [ShA9dW0NTwOOv-YrA1jAmA]
1> org.elasticsearch.action.FailedNodeException: Failed node [ShA9dW0NTwOOv-YrA1jAmA]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1047) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport.lambda$handleException$15(TcpTransport.java:1448) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:139) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1446) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1438) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1368) ~[main/:?]
1> at org.elasticsearch.transport.MockTcpTransport.readMessage(MockTcpTransport.java:171) ~[framework-6.0.0-alpha3-SNAPSHOT.jar:?]
1> at org.elasticsearch.transport.MockTcpTransport.access$900(MockTcpTransport.java:72) ~[framework-6.0.0-alpha3-SNAPSHOT.jar:?]
1> at org.elasticsearch.transport.MockTcpTransport$MockChannel$1.lambda$doRun$0(MockTcpTransport.java:349) ~[framework-6.0.0-alpha3-SNAPSHOT.jar:?]
1> at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[main/:?]
1> at org.elasticsearch.transport.MockTcpTransport$MockChannel$1.doRun(MockTcpTransport.java:349) ~[framework-6.0.0-alpha3-SNAPSHOT.jar:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161) ~[?:?]
1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
1> at java.lang.Thread.run(Thread.java:844) ~[?:?]
1> Caused by: org.elasticsearch.transport.RemoteTransportException: [node_s1][127.0.0.1:37346][internal:cluster/nodes/indices/shard/store[n]]
1> Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][3]]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[main/:?]
1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1525) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161) ~[?:?]
1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
1> at java.lang.Thread.run(Thread.java:844) ~[?:?]
1> Caused by: java.io.FileNotFoundException: no segments* file found in store(MMapDirectory@/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/custom/sufYyhvrob/1/gnCPCG0dS4SZrXJrHJee4w/3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2a6f8610): files: [write.lock]
1> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:687) ~[lucene-core-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:02]
1> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) ~[lucene-core-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:02]
1> at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) ~[lucene-core-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:02]
1> at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:127) ~[main/:?]
1> at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:200) ~[main/:?]
1> at org.elasticsearch.index.store.Store.access$200(Store.java:128) ~[main/:?]
1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:831) ~[main/:?]
1> at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:764) ~[main/:?]
1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:288) ~[main/:?]
1> at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:978) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:125) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[main/:?]
1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1525) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161) ~[?:?]
1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
1> at java.lang.Thread.run(Thread.java:844) ~[?:?]
1> [2017-06-21T08:07:42,458][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [node_s0] [test][0]: failed to list shard for shard_store on node [AxNGFLjdQmyv5UPrUxLM2w]
1> org.elasticsearch.action.FailedNodeException: Failed node [AxNGFLjdQmyv5UPrUxLM2w]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[main/:?]
1> at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1047) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport.lambda$handleException$15(TcpTransport.java:1448) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:139) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1446) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1438) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1368) ~[main/:?]
1> at org.elasticsearch.transport.MockTcpTransport.readMessage(MockTcpTransport.java:171) ~[framework-6.0.0-alpha3-SNAPSHOT.jar:?]
1> at org.elasticsearch.transport.MockTcpTransport.access$900(MockTcpTransport.java:72) ~[framework-6.0.0-alpha3-SNAPSHOT.jar:?]
1> at org.elasticsearch.transport.MockTcpTransport$MockChannel$1.lambda$doRun$0(MockTcpTransport.java:349) ~[framework-6.0.0-alpha3-SNAPSHOT.jar:?]
1> at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[main/:?]
1> at org.elasticsearch.transport.MockTcpTransport$MockChannel$1.doRun(MockTcpTransport.java:349) ~[framework-6.0.0-alpha3-SNAPSHOT.jar:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161) ~[?:?]
1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
1> at java.lang.Thread.run(Thread.java:844) ~[?:?]
1> Caused by: org.elasticsearch.transport.RemoteTransportException: [node_s2][127.0.0.1:45006][internal:cluster/nodes/indices/shard/store[n]]
1> Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[test][0]]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[main/:?]
1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1525) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161) ~[?:?]
1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
1> at java.lang.Thread.run(Thread.java:844) ~[?:?]
1> Caused by: java.io.FileNotFoundException: no segments* file found in store(MMapDirectory@/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/custom/sufYyhvrob/2/gnCPCG0dS4SZrXJrHJee4w/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2a6f8610): files: [write.lock]
1> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:687) ~[lucene-core-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:02]
1> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) ~[lucene-core-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:02]
1> at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) ~[lucene-core-7.0.0-snapshot-92b1783.jar:7.0.0-snapshot-92b1783 92b17838a346ad55a6a4ab796b8ab8cbbe4ffea2 - jpountz - 2017-06-14 11:01:02]
1> at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:127) ~[main/:?]
1> at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:200) ~[main/:?]
1> at org.elasticsearch.index.store.Store.access$200(Store.java:128) ~[main/:?]
1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:831) ~[main/:?]
1> at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:764) ~[main/:?]
1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:288) ~[main/:?]
1> at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:978) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:125) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) ~[main/:?]
1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[main/:?]
1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[main/:?]
1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?]
1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1525) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1161) ~[?:?]
1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
1> at java.lang.Thread.run(Thread.java:844) ~[?:?]
1> [2017-06-21T08:07:42,542][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]] ...]).
1> [2017-06-21T08:07:42,551][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:42,551][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> add two more nodes
1> [2017-06-21T08:07:42,551][INFO ][o.e.t.InternalTestCluster] increasing cluster size from 3 to 4
1> [2017-06-21T08:07:42,553][INFO ][o.e.n.Node ] [node_s3] initializing ...
1> [2017-06-21T08:07:42,556][INFO ][o.e.e.NodeEnvironment ] [node_s3] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:42,557][INFO ][o.e.e.NodeEnvironment ] [node_s3] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:42,557][INFO ][o.e.n.Node ] [node_s3] node name [node_s3], node ID [QoBs5pVuSdiCFmOPuQw0gg]
1> [2017-06-21T08:07:42,557][INFO ][o.e.n.Node ] [node_s3] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:42,557][INFO ][o.e.n.Node ] [node_s3] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:42,557][WARN ][o.e.n.Node ] [node_s3] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:42,558][INFO ][o.e.p.PluginsService ] [node_s3] no modules loaded
1> [2017-06-21T08:07:42,558][INFO ][o.e.p.PluginsService ] [node_s3] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
1> [2017-06-21T08:07:42,558][INFO ][o.e.p.PluginsService ] [node_s3] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:42,558][INFO ][o.e.p.PluginsService ] [node_s3] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
1> [2017-06-21T08:07:42,558][INFO ][o.e.p.PluginsService ] [node_s3] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:42,558][INFO ][o.e.p.PluginsService ] [node_s3] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:42,570][INFO ][o.e.d.DiscoveryModule ] [node_s3] using discovery type [test-zen]
1> [2017-06-21T08:07:42,630][INFO ][o.e.n.Node ] [node_s3] initialized
1> [2017-06-21T08:07:42,633][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [discovery.zen.minimum_master_nodes] from [2] to [3]
1> [2017-06-21T08:07:42,633][INFO ][o.e.c.s.ClusterSettings ] [node_s2] updating [discovery.zen.minimum_master_nodes] from [2] to [3]
1> [2017-06-21T08:07:42,635][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [discovery.zen.minimum_master_nodes] from [2] to [3]
1> [2017-06-21T08:07:42,637][INFO ][o.e.n.Node ] [node_s3] starting ...
1> [2017-06-21T08:07:42,640][INFO ][o.e.t.t.MockTransportService] [node_s3] publish_address {127.0.0.1:9503}, bound_addresses {[::1]:9503}, {127.0.0.1:9503}
1> [2017-06-21T08:07:42,640][INFO ][o.e.t.d.MockZenPing ] [node_s3] pinging using mock zen ping
1> [2017-06-21T08:07:42,660][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-join[{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503}], reason: added {{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},}
1> [2017-06-21T08:07:42,663][INFO ][o.e.c.s.ClusterApplierService] [node_s2] added {{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [67]])
1> [2017-06-21T08:07:42,663][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [67]])
1> [2017-06-21T08:07:42,664][INFO ][o.e.c.s.ClusterApplierService] [node_s3] detected_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},{node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [67]])
1> [2017-06-21T08:07:42,696][INFO ][o.e.n.Node ] [node_s3] started
1> [2017-06-21T08:07:42,698][INFO ][o.e.p.PluginsService ] [transport_client_node_s2] no modules loaded
1> [2017-06-21T08:07:42,698][INFO ][o.e.p.PluginsService ] [transport_client_node_s2] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:42,703][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [67] source [zen-disco-node-join[{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503}]]])
1> [2017-06-21T08:07:42,809][INFO ][o.e.t.InternalTestCluster] increasing cluster size from 4 to 5
1> [2017-06-21T08:07:42,810][INFO ][o.e.n.Node ] [node_s4] initializing ...
1> [2017-06-21T08:07:42,817][INFO ][o.e.e.NodeEnvironment ] [node_s4] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:42,817][INFO ][o.e.e.NodeEnvironment ] [node_s4] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:42,817][INFO ][o.e.n.Node ] [node_s4] node name [node_s4], node ID [zXdIgndET8qTrxmrX3c-EQ]
1> [2017-06-21T08:07:42,817][INFO ][o.e.n.Node ] [node_s4] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:42,817][INFO ][o.e.n.Node ] [node_s4] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:42,818][WARN ][o.e.n.Node ] [node_s4] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:42,818][INFO ][o.e.p.PluginsService ] [node_s4] no modules loaded
1> [2017-06-21T08:07:42,818][INFO ][o.e.p.PluginsService ] [node_s4] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
1> [2017-06-21T08:07:42,818][INFO ][o.e.p.PluginsService ] [node_s4] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:42,818][INFO ][o.e.p.PluginsService ] [node_s4] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
1> [2017-06-21T08:07:42,818][INFO ][o.e.p.PluginsService ] [node_s4] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:42,818][INFO ][o.e.p.PluginsService ] [node_s4] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:42,831][INFO ][o.e.d.DiscoveryModule ] [node_s4] using discovery type [test-zen]
1> [2017-06-21T08:07:42,858][INFO ][o.e.n.Node ] [node_s4] initialized
1> [2017-06-21T08:07:42,864][INFO ][o.e.n.Node ] [node_s4] starting ...
1> [2017-06-21T08:07:42,869][INFO ][o.e.t.t.MockTransportService] [node_s4] publish_address {127.0.0.1:9504}, bound_addresses {[::1]:9504}, {127.0.0.1:9504}
1> [2017-06-21T08:07:42,870][INFO ][o.e.t.d.MockZenPing ] [node_s4] pinging using mock zen ping
1> [2017-06-21T08:07:42,877][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-join[{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504}], reason: added {{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504},}
1> [2017-06-21T08:07:42,880][INFO ][o.e.c.s.ClusterApplierService] [node_s3] added {{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [73]])
1> [2017-06-21T08:07:42,881][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [73]])
1> [2017-06-21T08:07:42,882][INFO ][o.e.c.s.ClusterApplierService] [node_s2] added {{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [73]])
1> [2017-06-21T08:07:42,884][INFO ][o.e.c.s.ClusterApplierService] [node_s4] detected_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},{node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [73]])
1> [2017-06-21T08:07:42,906][INFO ][o.e.n.Node ] [node_s4] started
1> [2017-06-21T08:07:42,908][INFO ][o.e.p.PluginsService ] [transport_client_node_s3] no modules loaded
1> [2017-06-21T08:07:42,908][INFO ][o.e.p.PluginsService ] [transport_client_node_s3] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:42,910][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [73] source [zen-disco-node-join[{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504}]]])
1> [2017-06-21T08:07:42,972][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> update the auto expand replicas to 0-3
1> [2017-06-21T08:07:43,006][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:43,015][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [3] for indices [test]
1> [2017-06-21T08:07:43,027][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/gnCPCG0dS4SZrXJrHJee4w] auto expanded replicas to [3]
1> [2017-06-21T08:07:43,039][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [3] for indices [test]
1> [2017-06-21T08:07:43,045][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/gnCPCG0dS4SZrXJrHJee4w] auto expanded replicas to [3]
1> [2017-06-21T08:07:43,102][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0], [test][1]] ...]).
1> [2017-06-21T08:07:43,109][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:43,109][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberReplicas2]: cleaning up after test
1> [2017-06-21T08:07:43,138][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [test/gnCPCG0dS4SZrXJrHJee4w] deleting index
1> [2017-06-21T08:07:43,246][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberReplicas2]: cleaned up after test
1> [2017-06-21T08:07:43,246][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testAutoExpandNumberReplicas2]: after test
1> [2017-06-21T08:07:43,324][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testUpdateWithInvalidNumberOfReplicas]: before test
1> [2017-06-21T08:07:43,324][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testUpdateWithInvalidNumberOfReplicas]: setting up test
1> [2017-06-21T08:07:43,328][INFO ][o.e.c.s.ClusterSettings ] [node_s2] updating [discovery.zen.minimum_master_nodes] from [3] to [2]
1> [2017-06-21T08:07:43,328][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [discovery.zen.minimum_master_nodes] from [3] to [2]
1> [2017-06-21T08:07:43,328][INFO ][o.e.c.s.ClusterSettings ] [node_s3] updating [discovery.zen.minimum_master_nodes] from [3] to [2]
1> [2017-06-21T08:07:43,329][INFO ][o.e.c.s.ClusterSettings ] [node_s4] updating [discovery.zen.minimum_master_nodes] from [3] to [2]
1> [2017-06-21T08:07:43,330][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [discovery.zen.minimum_master_nodes] from [3] to [2]
1> [2017-06-21T08:07:43,335][INFO ][o.e.n.Node ] [node_s2] stopping ...
1> [2017-06-21T08:07:43,336][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-left({node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502}), reason(left)[{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502} left], reason: removed {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},}
1> [2017-06-21T08:07:43,338][INFO ][o.e.c.s.ClusterApplierService] [node_s4] removed {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [87]])
1> [2017-06-21T08:07:43,338][INFO ][o.e.c.s.ClusterApplierService] [node_s3] removed {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [87]])
1> [2017-06-21T08:07:43,339][INFO ][o.e.n.Node ] [node_s2] stopped
1> [2017-06-21T08:07:43,339][INFO ][o.e.n.Node ] [node_s2] closing ...
1> [2017-06-21T08:07:43,340][INFO ][o.e.c.s.ClusterApplierService] [node_s1] removed {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [87]])
1> [2017-06-21T08:07:43,340][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [87] source [zen-disco-node-left({node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502}), reason(left)[{node_s2}{AxNGFLjdQmyv5UPrUxLM2w}{eIztE-NXTUeVsCFh7uPH2w}{127.0.0.1}{127.0.0.1:9502} left]]])
1> [2017-06-21T08:07:43,343][INFO ][o.e.n.Node ] [node_s2] closed
1> [2017-06-21T08:07:43,345][INFO ][o.e.n.Node ] [node_s3] stopping ...
1> [2017-06-21T08:07:43,347][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-left({node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503}), reason(left)[{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503} left], reason: removed {{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},}
1> [2017-06-21T08:07:43,348][INFO ][o.e.c.s.ClusterApplierService] [node_s1] removed {{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [88]])
1> [2017-06-21T08:07:43,349][INFO ][o.e.c.s.ClusterApplierService] [node_s4] removed {{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [88]])
1> [2017-06-21T08:07:43,349][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [88] source [zen-disco-node-left({node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503}), reason(left)[{node_s3}{QoBs5pVuSdiCFmOPuQw0gg}{frmQsw5PRPeKTBlXmaqTng}{127.0.0.1}{127.0.0.1:9503} left]]])
1> [2017-06-21T08:07:43,351][INFO ][o.e.n.Node ] [node_s3] stopped
1> [2017-06-21T08:07:43,351][INFO ][o.e.n.Node ] [node_s3] closing ...
1> [2017-06-21T08:07:43,353][INFO ][o.e.n.Node ] [node_s3] closed
1> [2017-06-21T08:07:43,353][INFO ][o.e.n.Node ] [node_s4] stopping ...
1> [2017-06-21T08:07:43,354][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-left({node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504}), reason(left)[{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504} left], reason: removed {{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504},}
1> [2017-06-21T08:07:43,355][INFO ][o.e.c.s.ClusterApplierService] [node_s1] removed {{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [89]])
1> [2017-06-21T08:07:43,355][INFO ][o.e.n.Node ] [node_s4] stopped
1> [2017-06-21T08:07:43,356][INFO ][o.e.n.Node ] [node_s4] closing ...
1> [2017-06-21T08:07:43,356][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [89] source [zen-disco-node-left({node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504}), reason(left)[{node_s4}{zXdIgndET8qTrxmrX3c-EQ}{07sGN8cXTzyFgfZyNwbojg}{127.0.0.1}{127.0.0.1:9504} left]]])
1> [2017-06-21T08:07:43,357][INFO ][o.e.n.Node ] [node_s4] closed
1> [2017-06-21T08:07:43,358][INFO ][o.e.t.InternalTestCluster] Successfully wiped data directory for node location: /var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/4
1> [2017-06-21T08:07:43,358][INFO ][o.e.t.InternalTestCluster] Successfully wiped data directory for node location: /var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/3
1> [2017-06-21T08:07:43,359][INFO ][o.e.t.InternalTestCluster] Successfully wiped data directory for node location: /var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT_A93BDC488D12ACBB-001/tempDir-002/data/nodes/2
1> [2017-06-21T08:07:43,362][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] test using _default_ mappings: [{"_default_":{}}]
1> [2017-06-21T08:07:43,370][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testUpdateWithInvalidNumberOfReplicas]: all set up test
1> [2017-06-21T08:07:43,370][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] using custom data_path for index: [ZynxQvlFqY]
1> [2017-06-21T08:07:43,374][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [test] creating index, cause [api], templates [random_index_template], shards [9]/[1], mappings [_default_]
1> [2017-06-21T08:07:43,514][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testUpdateWithInvalidNumberOfReplicas]: cleaning up after test
1> [2017-06-21T08:07:43,582][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [test/8-BLcCWaQTqfJQ5nByXkCg] deleting index
1> [2017-06-21T08:07:43,635][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testUpdateWithInvalidNumberOfReplicas]: cleaned up after test
1> [2017-06-21T08:07:43,635][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testUpdateWithInvalidNumberOfReplicas]: after test
1> [2017-06-21T08:07:43,650][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testAutoExpandNumberReplicas1ToData]: before test
1> [2017-06-21T08:07:43,650][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberReplicas1ToData]: setting up test
1> [2017-06-21T08:07:43,656][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] test using _default_ mappings: [{"_default_":{}}]
1> [2017-06-21T08:07:43,664][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberReplicas1ToData]: all set up test
1> [2017-06-21T08:07:43,664][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> creating index test with auto expand replicas
1> [2017-06-21T08:07:43,665][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] using custom data_path for index: [cAgMzdJGal]
1> [2017-06-21T08:07:43,671][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [test] creating index, cause [api], templates [random_index_template], shards [8]/[0], mappings [_default_]
1> [2017-06-21T08:07:43,731][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [test]
1> [2017-06-21T08:07:43,744][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/tPGIsRXkTy6XUh2sg7H0aA] auto expanded replicas to [1]
1> [2017-06-21T08:07:43,767][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [test]
1> [2017-06-21T08:07:43,768][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:43,774][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/tPGIsRXkTy6XUh2sg7H0aA] auto expanded replicas to [1]
1> [2017-06-21T08:07:43,835][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][6]] ...]).
1> [2017-06-21T08:07:43,839][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:43,839][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> add another node, should increase the number of replicas
1> [2017-06-21T08:07:43,839][INFO ][o.e.t.InternalTestCluster] increasing cluster size from 2 to 3
1> [2017-06-21T08:07:43,840][INFO ][o.e.n.Node ] [node_s2] initializing ...
1> [2017-06-21T08:07:43,842][INFO ][o.e.e.NodeEnvironment ] [node_s2] using [1] data paths, mounts [[/ (/dev/mapper/vg00-root)]], net usable_space [197.9gb], net total_space [429.9gb], types [ext4]
1> [2017-06-21T08:07:43,842][INFO ][o.e.e.NodeEnvironment ] [node_s2] heap size [512mb], compressed ordinary object pointers [true]
1> [2017-06-21T08:07:43,843][INFO ][o.e.n.Node ] [node_s2] node name [node_s2], node ID [wXP_PtucR4yz8xKN-Y5baQ]
1> [2017-06-21T08:07:43,843][INFO ][o.e.n.Node ] [node_s2] version[6.0.0-alpha3-SNAPSHOT], pid[13401], build[Unknown/Unknown], OS[Linux/4.8.0-53-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/9-ea/9-ea+174]
1> [2017-06-21T08:07:43,843][INFO ][o.e.n.Node ] [node_s2] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/heapdump, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.nio.file=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent=ALL-UNNAMED, --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Dtests.prefix=tests, -Dtests.seed=A93BDC488D12ACBB, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=core, -Dtests.jvm.argline=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, -Djava.awt.headless=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dtests.ifNoTests=fail, -Dtests.task=:core:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+master+java9-periodic/core/build/testrun/integTest/J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=4]
1> [2017-06-21T08:07:43,843][WARN ][o.e.n.Node ] [node_s2] version [6.0.0-alpha3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
1> [2017-06-21T08:07:43,844][INFO ][o.e.p.PluginsService ] [node_s2] no modules loaded
1> [2017-06-21T08:07:43,844][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin]
1> [2017-06-21T08:07:43,844][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
1> [2017-06-21T08:07:43,844][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
1> [2017-06-21T08:07:43,844][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin]
1> [2017-06-21T08:07:43,844][INFO ][o.e.p.PluginsService ] [node_s2] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:43,857][INFO ][o.e.d.DiscoveryModule ] [node_s2] using discovery type [test-zen]
1> [2017-06-21T08:07:43,893][INFO ][o.e.n.Node ] [node_s2] initialized
1> [2017-06-21T08:07:43,897][INFO ][o.e.n.Node ] [node_s2] starting ...
1> [2017-06-21T08:07:43,902][INFO ][o.e.t.t.MockTransportService] [node_s2] publish_address {127.0.0.1:9502}, bound_addresses {[::1]:9502}, {127.0.0.1:9502}
1> [2017-06-21T08:07:43,902][INFO ][o.e.t.d.MockZenPing ] [node_s2] pinging using mock zen ping
1> [2017-06-21T08:07:43,912][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-join[{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502}], reason: added {{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502},}
1> [2017-06-21T08:07:43,913][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [114]])
1> [2017-06-21T08:07:43,913][INFO ][o.e.c.s.ClusterApplierService] [node_s2] detected_master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500}, added {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},{node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [114]])
1> [2017-06-21T08:07:43,934][INFO ][o.e.n.Node ] [node_s2] started
1> [2017-06-21T08:07:43,943][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [114] source [zen-disco-node-join[{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502}]]])
1> [2017-06-21T08:07:43,957][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [2] for indices [test]
1> [2017-06-21T08:07:43,964][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/tPGIsRXkTy6XUh2sg7H0aA] auto expanded replicas to [2]
1> [2017-06-21T08:07:43,998][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [2] for indices [test]
1> [2017-06-21T08:07:43,998][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:44,003][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/tPGIsRXkTy6XUh2sg7H0aA] auto expanded replicas to [2]
1> [2017-06-21T08:07:44,132][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][7]] ...]).
1> [2017-06-21T08:07:44,137][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:44,137][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> closing one node
1> [2017-06-21T08:07:44,137][INFO ][o.e.p.PluginsService ] [transport_client_node_s2] no modules loaded
1> [2017-06-21T08:07:44,137][INFO ][o.e.p.PluginsService ] [transport_client_node_s2] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
1> [2017-06-21T08:07:44,162][INFO ][o.e.t.InternalTestCluster] changing cluster size from 3 data nodes to 2
1> [2017-06-21T08:07:44,172][INFO ][o.e.n.Node ] [node_s1] stopping ...
1> [2017-06-21T08:07:44,174][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501} left]).
1> [2017-06-21T08:07:44,174][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-left({node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501}), reason(left)[{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501} left], reason: removed {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},}
1> [2017-06-21T08:07:44,176][INFO ][o.e.c.s.ClusterApplierService] [node_s2] removed {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [127]])
1> [2017-06-21T08:07:44,181][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [127] source [zen-disco-node-left({node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501}), reason(left)[{node_s1}{ShA9dW0NTwOOv-YrA1jAmA}{vCmLuikYTmW0Wz6xUyZF2g}{127.0.0.1}{127.0.0.1:9501} left]]])
1> [2017-06-21T08:07:44,186][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [test]
1> [2017-06-21T08:07:44,198][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] [test/tPGIsRXkTy6XUh2sg7H0aA] auto expanded replicas to [1]
1> [2017-06-21T08:07:44,201][INFO ][o.e.n.Node ] [node_s1] stopped
1> [2017-06-21T08:07:44,201][INFO ][o.e.n.Node ] [node_s1] closing ...
1> [2017-06-21T08:07:44,203][INFO ][o.e.n.Node ] [node_s1] closed
1> [2017-06-21T08:07:44,215][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:44,216][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status GREEN
1> [2017-06-21T08:07:44,216][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> closing another node
1> [2017-06-21T08:07:44,217][INFO ][o.e.t.InternalTestCluster] changing cluster size from 2 data nodes to 1
1> [2017-06-21T08:07:44,220][INFO ][o.e.c.s.ClusterSettings ] [node_s2] updating [discovery.zen.minimum_master_nodes] from [2] to [1]
1> [2017-06-21T08:07:44,225][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [discovery.zen.minimum_master_nodes] from [2] to [1]
1> [2017-06-21T08:07:44,228][INFO ][o.e.n.Node ] [node_s2] stopping ...
1> [2017-06-21T08:07:44,236][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502} left]).
1> [2017-06-21T08:07:44,237][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-left({node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502}), reason(left)[{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502} left], reason: removed {{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502},}
1> [2017-06-21T08:07:44,237][INFO ][o.e.c.s.ClusterApplierService] [node_s0] removed {{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502},}, reason: apply cluster state (from master [master {node_s0}{E84dTZseTwCU-8paQEqsww}{996vXPKCQ7KV1aVJab2iag}{127.0.0.1}{127.0.0.1:9500} committed version [131] source [zen-disco-node-left({node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502}), reason(left)[{node_s2}{wXP_PtucR4yz8xKN-Y5baQ}{oSLeOtK4QNG1_uBXMw5uDg}{127.0.0.1}{127.0.0.1:9502} left]]])
1> [2017-06-21T08:07:44,242][INFO ][o.e.n.Node ] [node_s2] stopped
1> [2017-06-21T08:07:44,243][INFO ][o.e.n.Node ] [node_s2] closing ...
1> [2017-06-21T08:07:44,247][INFO ][o.e.n.Node ] [node_s2] closed
1> [2017-06-21T08:07:44,257][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> running cluster health
1> [2017-06-21T08:07:44,258][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] --> done cluster health, status YELLOW
1> [2017-06-21T08:07:44,258][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberReplicas1ToData]: cleaning up after test
1> [2017-06-21T08:07:44,266][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [test/tPGIsRXkTy6XUh2sg7H0aA] deleting index
1> [2017-06-21T08:07:44,301][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [UpdateNumberOfReplicasIT#testAutoExpandNumberReplicas1ToData]: cleaned up after test
1> [2017-06-21T08:07:44,301][INFO ][o.e.i.s.UpdateNumberOfReplicasIT] [testAutoExpandNumberReplicas1ToData]: after test
1> [2017-06-21T08:07:44,303][INFO ][o.e.n.Node ] [node_s0] stopping ...
1> [2017-06-21T08:07:44,305][INFO ][o.e.n.Node ] [node_s0] stopped
1> [2017-06-21T08:07:44,305][INFO ][o.e.n.Node ] [node_s0] closing ...
1> [2017-06-21T08:07:44,308][INFO ][o.e.n.Node ] [node_s0] closed
Completed [6/294] on J1 in 13.27s, 5 tests, 1 error <<< FAILURES!