Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] NodeDisconnectIT testNotifyOnDisconnectInSniffer failure #52215

Closed
davidkyle opened this issue Feb 11, 2020 · 3 comments
Closed

[CI] NodeDisconnectIT testNotifyOnDisconnectInSniffer failure #52215

davidkyle opened this issue Feb 11, 2020 · 3 comments

Comments

@davidkyle
Copy link
Member

@davidkyle davidkyle commented Feb 11, 2020

java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([A37722C9900F7688:2A4BE9BB222164B6]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.elasticsearch.client.transport.NodeDisconnectIT.testNotifyOnDisconnectInSniffer(NodeDisconnectIT.java:78)

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob+fast+part1/3319/console
https://gradle-enterprise.elastic.co/s/vf7pp6fchoh5w

Reproduce

./gradlew ':server:integTest' --tests "org.elasticsearch.client.transport.NodeDisconnectIT.testNotifyOnDisconnectInSniffer" \
  -Dtests.seed=A37722C9900F7688 \
  -Dtests.security.manager=true \
  -Dtests.locale=bg \
  -Dtests.timezone=Etc/GMT+0 \
  -Dcompiler.java=13

This is the only failure I could find in build stats, initially I thought it was the same as #37567 (comment) but the odd error in the log caught my eye

No handler for action [cluster:monitor/nodes/liveness]
full log 14:56:56 1> [2020-02-11T14:56:52,868][INFO ][o.e.c.t.NodeDisconnectIT ] [testNotifyOnDisconnectInSniffer] before test 14:56:56 1> [2020-02-11T14:56:52,869][INFO ][o.e.c.t.NodeDisconnectIT ] [testNotifyOnDisconnectInSniffer] [NodeDisconnectIT#testNotifyOnDisconnectInSniffer]: setting up test 14:56:56 1> [2020-02-11T14:56:52,869][INFO ][o.e.t.InternalTestCluster] [testNotifyOnDisconnectInSniffer] Setup InternalTestCluster [TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster] with seed [42F50465CEEE904A] using [0] dedicated masters, [3] (data) nodes and [1] coord only nodes (min_master_nodes are [auto-managed]) 14:56:56 1> [2020-02-11T14:56:52,873][INFO ][o.e.e.NodeEnvironment ] [testNotifyOnDisconnectInSniffer] using [1] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [34.2gb], net total_space [47.1gb], types [tmpfs] 14:56:56 1> [2020-02-11T14:56:52,874][INFO ][o.e.e.NodeEnvironment ] [testNotifyOnDisconnectInSniffer] heap size [491mb], compressed ordinary object pointers [true] 14:56:56 1> [2020-02-11T14:56:52,875][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] node name [node_t0], node ID [TUWuLZ62QQO-StG8hmnAEw], cluster name [TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster] 14:56:56 1> [2020-02-11T14:56:52,875][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] version[7.7.0], pid[196317], build[unknown/unknown/unknown/unknown], OS[Linux/3.10.0-1062.12.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_241/25.241-b07] 14:56:56 1> [2020-02-11T14:56:52,875][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] JVM home [/var/lib/jenkins/.java/oracle-8u241-linux/jre] 14:56:56 1> [2020-02-11T14:56:52,875][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=13, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.1.1-all/cfmwm155h49vnt3hynmlrsdst/gradle-6.1.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.1.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,JRE, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=A37722C9900F7688, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+7.x+multijob+fast+part1/server/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] no modules loaded 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,876][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 14:56:56 1> [2020-02-11T14:56:52,889][INFO ][o.e.d.DiscoveryModule ] [testNotifyOnDisconnectInSniffer] using discovery type [zen] and seed hosts providers [settings, file] 14:56:56 1> [2020-02-11T14:56:52,903][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] initialized 14:56:56 1> [2020-02-11T14:56:52,906][INFO ][o.e.e.NodeEnvironment ] [testNotifyOnDisconnectInSniffer] using [1] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [34.2gb], net total_space [47.1gb], types [tmpfs] 14:56:56 1> [2020-02-11T14:56:52,906][INFO ][o.e.e.NodeEnvironment ] [testNotifyOnDisconnectInSniffer] heap size [491mb], compressed ordinary object pointers [true] 14:56:56 1> [2020-02-11T14:56:52,907][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] node name [node_t1], node ID [o4DHOlyATZ6wFAHBzIl_Aw], cluster name [TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster] 14:56:56 1> [2020-02-11T14:56:52,908][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] version[7.7.0], pid[196317], build[unknown/unknown/unknown/unknown], OS[Linux/3.10.0-1062.12.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_241/25.241-b07] 14:56:56 1> [2020-02-11T14:56:52,908][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] JVM home [/var/lib/jenkins/.java/oracle-8u241-linux/jre] 14:56:56 1> [2020-02-11T14:56:52,908][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=13, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.1.1-all/cfmwm155h49vnt3hynmlrsdst/gradle-6.1.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.1.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,JRE, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=A37722C9900F7688, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+7.x+multijob+fast+part1/server/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] 14:56:56 1> [2020-02-11T14:56:52,908][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] no modules loaded 14:56:56 1> [2020-02-11T14:56:52,909][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] 14:56:56 1> [2020-02-11T14:56:52,909][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] 14:56:56 1> [2020-02-11T14:56:52,909][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] 14:56:56 1> [2020-02-11T14:56:52,909][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 14:56:56 1> [2020-02-11T14:56:52,909][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,909][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,909][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,909][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 14:56:56 1> [2020-02-11T14:56:52,922][INFO ][o.e.d.DiscoveryModule ] [testNotifyOnDisconnectInSniffer] using discovery type [zen] and seed hosts providers [settings, file] 14:56:56 1> [2020-02-11T14:56:52,939][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] initialized 14:56:56 1> [2020-02-11T14:56:52,943][INFO ][o.e.e.NodeEnvironment ] [testNotifyOnDisconnectInSniffer] using [1] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [34.2gb], net total_space [47.1gb], types [tmpfs] 14:56:56 1> [2020-02-11T14:56:52,943][INFO ][o.e.e.NodeEnvironment ] [testNotifyOnDisconnectInSniffer] heap size [491mb], compressed ordinary object pointers [true] 14:56:56 1> [2020-02-11T14:56:52,944][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] node name [node_t2], node ID [vmEv1dSZSj-J8DgpOuKgXg], cluster name [TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster] 14:56:56 1> [2020-02-11T14:56:52,944][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] version[7.7.0], pid[196317], build[unknown/unknown/unknown/unknown], OS[Linux/3.10.0-1062.12.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_241/25.241-b07] 14:56:56 1> [2020-02-11T14:56:52,944][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] JVM home [/var/lib/jenkins/.java/oracle-8u241-linux/jre] 14:56:56 1> [2020-02-11T14:56:52,944][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=13, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.1.1-all/cfmwm155h49vnt3hynmlrsdst/gradle-6.1.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.1.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,JRE, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=A37722C9900F7688, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+7.x+multijob+fast+part1/server/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] no modules loaded 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,945][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 14:56:56 1> [2020-02-11T14:56:52,960][INFO ][o.e.d.DiscoveryModule ] [testNotifyOnDisconnectInSniffer] using discovery type [zen] and seed hosts providers [settings, file] 14:56:56 1> [2020-02-11T14:56:52,981][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] initialized 14:56:56 1> [2020-02-11T14:56:52,985][INFO ][o.e.e.NodeEnvironment ] [testNotifyOnDisconnectInSniffer] using [1] data paths, mounts [[/dev/shm (tmpfs)]], net usable_space [34.2gb], net total_space [47.1gb], types [tmpfs] 14:56:56 1> [2020-02-11T14:56:52,985][INFO ][o.e.e.NodeEnvironment ] [testNotifyOnDisconnectInSniffer] heap size [491mb], compressed ordinary object pointers [true] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] node name [node_tc3], node ID [sDbp9A8RQgyyYZAFg9MsTA], cluster name [TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] version[7.7.0], pid[196317], build[unknown/unknown/unknown/unknown], OS[Linux/3.10.0-1062.12.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_241/25.241-b07] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] JVM home [/var/lib/jenkins/.java/oracle-8u241-linux/jre] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] JVM arguments [-Dfile.encoding=UTF8, -Dcompiler.java=13, -Des.scripting.update.ctx_in_params=false, -Des.search.rewrite_sort=true, -Des.transport.cname_in_publish_address=true, -Dgradle.dist.lib=/var/lib/jenkins/.gradle/wrapper/dists/gradle-6.1.1-all/cfmwm155h49vnt3hynmlrsdst/gradle-6.1.1/lib, -Dgradle.user.home=/var/lib/jenkins/.gradle, -Dgradle.worker.jar=/var/lib/jenkins/.gradle/caches/6.1.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.locale.providers=SPI,JRE, -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager, -Djna.nosys=true, -Dorg.gradle.native=false, -Dtests.artifact=server, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=true, -Dtests.seed=A37722C9900F7688, -Dtests.task=:server:integTest, -XX:+HeapDumpOnOutOfMemoryError, -esa, -XX:HeapDumpPath=/dev/shm/elastic+elasticsearch+7.x+multijob+fast+part1/server/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=UTF-8, -Djava.io.tmpdir=./temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] no modules loaded 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.index.MockEngineFactoryPlugin] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.node.NodeMocksPlugin] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.test.transport.MockTransportService$TestPlugin] 14:56:56 1> [2020-02-11T14:56:52,987][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 14:56:56 1> [2020-02-11T14:56:53,004][INFO ][o.e.d.DiscoveryModule ] [testNotifyOnDisconnectInSniffer] using discovery type [zen] and seed hosts providers [settings, file] 14:56:56 1> [2020-02-11T14:56:53,020][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] initialized 14:56:56 1> [2020-02-11T14:56:53,021][INFO ][o.e.n.Node ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#1]]] starting ... 14:56:56 1> [2020-02-11T14:56:53,021][INFO ][o.e.n.Node ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#2]]] starting ... 14:56:56 1> [2020-02-11T14:56:53,022][INFO ][o.e.n.Node ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#3]]] starting ... 14:56:56 1> [2020-02-11T14:56:53,023][INFO ][o.e.n.Node ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#4]]] starting ... 14:56:56 1> [2020-02-11T14:56:53,030][INFO ][o.e.t.TransportService ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#3]]] publish_address {127.0.0.1:39718}, bound_addresses {[::1]:43120}, {127.0.0.1:39718} 14:56:56 1> [2020-02-11T14:56:53,030][INFO ][o.e.t.TransportService ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#1]]] publish_address {127.0.0.1:43096}, bound_addresses {[::1]:45393}, {127.0.0.1:43096} 14:56:56 1> [2020-02-11T14:56:53,030][INFO ][o.e.t.TransportService ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#4]]] publish_address {127.0.0.1:40990}, bound_addresses {[::1]:35485}, {127.0.0.1:40990} 14:56:56 1> [2020-02-11T14:56:53,031][INFO ][o.e.t.TransportService ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#2]]] publish_address {127.0.0.1:36209}, bound_addresses {[::1]:45802}, {127.0.0.1:36209} 14:56:56 1> [2020-02-11T14:56:53,055][INFO ][o.e.c.c.Coordinator ] [node_t1] setting initial configuration to VotingConfiguration{o4DHOlyATZ6wFAHBzIl_Aw,vmEv1dSZSj-J8DgpOuKgXg,{bootstrap-placeholder}-node_t0} 14:56:56 1> [2020-02-11T14:56:53,100][INFO ][o.e.c.s.MasterService ] [node_t1] elected-as-master ([2] nodes joined)[{node_t2}{vmEv1dSZSj-J8DgpOuKgXg}{cYHlayD-RhaxyggCRaN4rA}{127.0.0.1}{127.0.0.1:39718}{dim} elect leader, {node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}]}, added {{node_t2}{vmEv1dSZSj-J8DgpOuKgXg}{cYHlayD-RhaxyggCRaN4rA}{127.0.0.1}{127.0.0.1:39718}{dim}} 14:56:56 1> [2020-02-11T14:56:53,109][INFO ][o.e.c.c.CoordinationState] [node_t1] cluster UUID set to [dEMi3uDwR62QWzblxni_2A] 14:56:56 1> [2020-02-11T14:56:53,110][INFO ][o.e.c.c.CoordinationState] [node_t2] cluster UUID set to [dEMi3uDwR62QWzblxni_2A] 14:56:56 1> [2020-02-11T14:56:53,119][INFO ][o.e.c.s.ClusterApplierService] [node_t2] master node changed {previous [], current [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}]}, added {{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}} 14:56:56 1> [2020-02-11T14:56:53,120][INFO ][o.e.n.Node ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#3]]] started 14:56:56 1> [2020-02-11T14:56:53,121][INFO ][o.e.c.s.ClusterApplierService] [node_t1] master node changed {previous [], current [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}]}, added {{node_t2}{vmEv1dSZSj-J8DgpOuKgXg}{cYHlayD-RhaxyggCRaN4rA}{127.0.0.1}{127.0.0.1:39718}{dim}}, term: 1, version: 1, reason: Publication{term=1, version=1} 14:56:56 1> [2020-02-11T14:56:53,121][INFO ][o.e.n.Node ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#2]]] started 14:56:56 1> [2020-02-11T14:56:53,122][INFO ][o.e.c.s.MasterService ] [node_t1] node-join[{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim} join existing leader], term: 1, version: 2, delta: added {{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim}} 14:56:56 1> [2020-02-11T14:56:53,125][INFO ][o.e.c.s.ClusterApplierService] [node_t2] added {{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}} 14:56:56 1> [2020-02-11T14:56:53,133][INFO ][o.e.c.s.ClusterApplierService] [node_t0] master node changed {previous [], current [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}]}, added {{node_t2}{vmEv1dSZSj-J8DgpOuKgXg}{cYHlayD-RhaxyggCRaN4rA}{127.0.0.1}{127.0.0.1:39718}{dim},{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}} 14:56:56 1> [2020-02-11T14:56:53,134][INFO ][o.e.n.Node ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#1]]] started 14:56:56 1> [2020-02-11T14:56:53,134][INFO ][o.e.c.s.ClusterApplierService] [node_t1] added {{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim}}, term: 1, version: 2, reason: Publication{term=1, version=2} 14:56:56 1> [2020-02-11T14:56:53,174][INFO ][o.e.g.GatewayService ] [node_t1] recovered [0] indices into cluster_state 14:56:56 1> [2020-02-11T14:56:54,056][INFO ][o.e.c.s.MasterService ] [node_t1] node-join[{node_tc3}{sDbp9A8RQgyyYZAFg9MsTA}{2trEfiHkQYKhkWx4DyvVsQ}{127.0.0.1}{127.0.0.1:40990} join existing leader], term: 1, version: 5, delta: added {{node_tc3}{sDbp9A8RQgyyYZAFg9MsTA}{2trEfiHkQYKhkWx4DyvVsQ}{127.0.0.1}{127.0.0.1:40990}} 14:56:56 1> [2020-02-11T14:56:54,059][INFO ][o.e.c.s.ClusterApplierService] [node_t0] added {{node_tc3}{sDbp9A8RQgyyYZAFg9MsTA}{2trEfiHkQYKhkWx4DyvVsQ}{127.0.0.1}{127.0.0.1:40990}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}} 14:56:56 1> [2020-02-11T14:56:54,059][INFO ][o.e.c.s.ClusterApplierService] [node_t2] added {{node_tc3}{sDbp9A8RQgyyYZAFg9MsTA}{2trEfiHkQYKhkWx4DyvVsQ}{127.0.0.1}{127.0.0.1:40990}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}} 14:56:56 1> [2020-02-11T14:56:54,059][INFO ][o.e.c.s.ClusterApplierService] [node_tc3] master node changed {previous [], current [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}]}, added {{node_t2}{vmEv1dSZSj-J8DgpOuKgXg}{cYHlayD-RhaxyggCRaN4rA}{127.0.0.1}{127.0.0.1:39718}{dim},{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim},{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}} 14:56:56 1> [2020-02-11T14:56:54,061][INFO ][o.e.n.Node ] [[test_TEST-TEST_WORKER_VM=[465]-CLUSTER_SEED=[4824767411107303498]-HASH=[15162D2B868]-cluster[T#4]]] started 14:56:56 1> [2020-02-11T14:56:54,064][INFO ][o.e.c.s.ClusterApplierService] [node_t1] added {{node_tc3}{sDbp9A8RQgyyYZAFg9MsTA}{2trEfiHkQYKhkWx4DyvVsQ}{127.0.0.1}{127.0.0.1:40990}}, term: 1, version: 5, reason: Publication{term=1, version=5} 14:56:56 1> [2020-02-11T14:56:54,075][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] no modules loaded 14:56:56 1> [2020-02-11T14:56:54,075][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 14:56:56 1> [2020-02-11T14:56:54,092][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] no modules loaded 14:56:56 1> [2020-02-11T14:56:54,092][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 14:56:56 1> [2020-02-11T14:56:54,129][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_t1] adding template [random_index_template] for index patterns [*] 14:56:56 1> [2020-02-11T14:56:54,144][INFO ][o.e.c.t.NodeDisconnectIT ] [testNotifyOnDisconnectInSniffer] [NodeDisconnectIT#testNotifyOnDisconnectInSniffer]: all set up test 14:56:56 1> [2020-02-11T14:56:54,150][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] no modules loaded 14:56:56 1> [2020-02-11T14:56:54,151][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 14:56:56 1> [2020-02-11T14:56:54,189][INFO ][o.e.t.InternalTestCluster] [testNotifyOnDisconnectInSniffer] Closing random node [node_t0] 14:56:56 1> [2020-02-11T14:56:54,191][INFO ][o.e.t.InternalTestCluster] [testNotifyOnDisconnectInSniffer] adding voting config exclusions [node_t0] prior to restart/shutdown 14:56:56 1> [2020-02-11T14:56:54,249][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] stopping ... 14:56:56 1> [2020-02-11T14:56:54,251][INFO ][o.e.c.c.Coordinator ] [node_t0] master node [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}] failed, restarting discovery 14:56:56 1> org.elasticsearch.transport.NodeDisconnectedException: [node_t1][127.0.0.1:36209][disconnected] disconnected 14:56:56 1> [2020-02-11T14:56:54,252][INFO ][o.e.c.s.MasterService ] [node_t1] node-left[{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim} reason: disconnected], term: 1, version: 10, delta: removed {{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim}} 14:56:56 1> [2020-02-11T14:56:54,252][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] stopped 14:56:56 1> [2020-02-11T14:56:54,252][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] closing ... 14:56:56 1> [2020-02-11T14:56:54,254][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] closed 14:56:56 1> [2020-02-11T14:56:54,255][INFO ][o.e.t.InternalTestCluster] [testNotifyOnDisconnectInSniffer] removing voting config exclusions for [node_t0] after restart/shutdown 14:56:56 1> [2020-02-11T14:56:54,256][INFO ][o.e.c.s.ClusterApplierService] [node_tc3] removed {{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim}}, term: 1, version: 10, reason: ApplyCommitRequest{term=1, version=10, sourceNode={node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}} 14:56:56 1> [2020-02-11T14:56:54,259][INFO ][o.e.c.s.ClusterApplierService] [node_t2] removed {{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim}}, term: 1, version: 10, reason: ApplyCommitRequest{term=1, version=10, sourceNode={node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}} 14:56:56 1> [2020-02-11T14:56:54,260][INFO ][o.e.c.s.ClusterApplierService] [node_t1] removed {{node_t0}{TUWuLZ62QQO-StG8hmnAEw}{202HRPTnT-O6K6DYk_I97w}{127.0.0.1}{127.0.0.1:43096}{dim}}, term: 1, version: 10, reason: Publication{term=1, version=10} 14:56:56 1> [2020-02-11T14:56:54,281][INFO ][o.e.c.t.TransportClientNodesService] [testNotifyOnDisconnectInSniffer] failed to get node info for {#transport#-1}{mCenUNDbQSe0Xtd6HXghVQ}{127.0.0.1}{127.0.0.1:43096}, disconnecting... 14:56:56 1> org.elasticsearch.transport.RemoteTransportException: [_client_][127.0.0.1:43096][cluster:monitor/nodes/liveness] 14:56:56 1> Caused by: org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [cluster:monitor/nodes/liveness] 14:56:56 1> at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:167) ~[main/:?] 14:56:56 1> at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:119) ~[main/:?] 14:56:56 1> at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103) ~[main/:?] 14:56:56 1> at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676) ~[main/:?] 14:56:56 1> at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:700) ~[main/:?] 14:56:56 1> at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:276) ~[framework-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:228) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:40) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:139) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:151) ~[framework-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:420) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:174) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:131) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241] 14:56:56 1> [2020-02-11T14:56:54,289][INFO ][o.e.c.t.NodeDisconnectIT ] [testNotifyOnDisconnectInSniffer] [NodeDisconnectIT#testNotifyOnDisconnectInSniffer]: cleaning up after test 14:56:56 1> [2020-02-11T14:56:54,297][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] no modules loaded 14:56:56 1> [2020-02-11T14:56:54,297][INFO ][o.e.p.PluginsService ] [testNotifyOnDisconnectInSniffer] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 14:56:56 1> [2020-02-11T14:56:54,320][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_t1] removing template [random_index_template] 14:56:56 1> [2020-02-11T14:56:54,335][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] stopping ... 14:56:56 1> [2020-02-11T14:56:54,338][INFO ][o.e.c.c.Coordinator ] [node_t2] master node [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}] failed, restarting discovery 14:56:56 1> org.elasticsearch.transport.NodeDisconnectedException: [node_t1][127.0.0.1:36209][disconnected] disconnected 14:56:56 1> [2020-02-11T14:56:54,338][INFO ][o.e.c.c.Coordinator ] [node_tc3] master node [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}] failed, restarting discovery 14:56:56 1> org.elasticsearch.transport.NodeDisconnectedException: [node_t1][127.0.0.1:36209][disconnected] disconnected 14:56:56 1> [2020-02-11T14:56:54,339][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] stopped 14:56:56 1> [2020-02-11T14:56:54,340][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] closing ... 14:56:56 1> [2020-02-11T14:56:54,340][INFO ][o.e.c.s.ClusterApplierService] [node_t2] master node changed {previous [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}], current []}, term: 1, version: 12, reason: becoming candidate: onLeaderFailure 14:56:56 1> [2020-02-11T14:56:54,348][INFO ][o.e.c.s.ClusterApplierService] [node_tc3] master node changed {previous [{node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim}], current []}, term: 1, version: 12, reason: becoming candidate: onLeaderFailure 14:56:56 1> [2020-02-11T14:56:54,348][WARN ][o.e.c.NodeConnectionsService] [node_t2] failed to connect to {node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim} (tried [1] times) 14:56:56 1> org.elasticsearch.transport.ConnectTransportException: [node_t1][127.0.0.1:36209] connect_exception 14:56:56 1> at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:998) ~[main/:?] 14:56:56 1> at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:198) ~[main/:?] 14:56:56 1> at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_241] 14:56:56 1> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_241] 14:56:56 1> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_241] 14:56:56 1> at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_241] 14:56:56 1> at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:142) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:117) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:123) ~[framework-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:428) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:451) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:440) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:261) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:155) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:131) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241] 14:56:56 1> Caused by: java.net.ConnectException: Connection refused 14:56:56 1> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] 14:56:56 1> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] 14:56:56 1> at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:139) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> ... 9 more 14:56:56 1> [2020-02-11T14:56:54,352][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] closed 14:56:56 1> [2020-02-11T14:56:54,350][WARN ][o.e.c.NodeConnectionsService] [node_tc3] failed to connect to {node_t1}{o4DHOlyATZ6wFAHBzIl_Aw}{4u7acTq_SZirSlF0NpZsoA}{127.0.0.1}{127.0.0.1:36209}{dim} (tried [1] times) 14:56:56 1> org.elasticsearch.transport.ConnectTransportException: [node_t1][127.0.0.1:36209] connect_exception 14:56:56 1> at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:998) ~[main/:?] 14:56:56 1> at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:198) ~[main/:?] 14:56:56 1> at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_241] 14:56:56 1> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_241] 14:56:56 1> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_241] 14:56:56 1> at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_241] 14:56:56 1> at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:142) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:117) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:123) ~[framework-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:428) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.registerChannel(NioSelector.java:451) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.setUpNewChannels(NioSelector.java:440) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.preSelect(NioSelector.java:261) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:155) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:131) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241] 14:56:56 1> Caused by: java.net.ConnectException: Connection refused 14:56:56 1> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] 14:56:56 1> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] 14:56:56 1> at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:139) ~[elasticsearch-nio-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT] 14:56:56 1> ... 9 more 14:56:56 1> [2020-02-11T14:56:54,356][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] stopping ... 14:56:56 1> [2020-02-11T14:56:54,362][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] stopped 14:56:56 1> [2020-02-11T14:56:54,362][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] closing ... 14:56:56 1> [2020-02-11T14:56:54,364][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] closed 14:56:56 1> [2020-02-11T14:56:54,366][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] stopping ... 14:56:56 1> [2020-02-11T14:56:54,369][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] stopped 14:56:56 1> [2020-02-11T14:56:54,369][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] closing ... 14:56:56 1> [2020-02-11T14:56:54,370][INFO ][o.e.n.Node ] [testNotifyOnDisconnectInSniffer] closed 14:56:56 1> [2020-02-11T14:56:54,370][INFO ][o.e.c.t.NodeDisconnectIT ] [testNotifyOnDisconnectInSniffer] [NodeDisconnectIT#testNotifyOnDisconnectInSniffer]: cleaned up after test 14:56:56 1> [2020-02-11T14:56:54,371][INFO ][o.e.c.t.NodeDisconnectIT ] [testNotifyOnDisconnectInSniffer] after test
@elasticmachine

This comment has been minimized.

Copy link
Collaborator

@elasticmachine elasticmachine commented Feb 11, 2020

Pinging @elastic/es-distributed (:Distributed/Network)

@davidkyle

This comment has been minimized.

Copy link
Member Author

@davidkyle davidkyle commented Feb 11, 2020

See also #52213

@original-brownbear

This comment has been minimized.

Copy link
Member

@original-brownbear original-brownbear commented Feb 14, 2020

I'm closing this. It's the same as #37567 effectively. We're not handling exceptions properly in the sniffer but as far as I can tell this cannot have any functional impact outside of notifying the node disconnected handler. With the transport client deprecated and going away I agree with Yannick -> no point in putting time into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.