AutoFollowIT#testCleanFollowedLeaderIndexUUIDs failures #41071

cbuescher · 2019-04-10T11:30:25Z

On 7.x: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-darwin-compatibility/99/console

Could not reproduce locally on 7.x:

./gradlew :x-pack:plugin:ccr:internalClusterTest --tests "org.elasticsearch.xpack.ccr.AutoFollowIT.testCleanFollowedLeaderIndexUUIDs" \
  -Dtests.seed=CB4054056BD16251 \
  -Dtests.security.manager=true \
  -Dtests.locale=el-CY \
  -Dtests.timezone=America/Santo_Domingo \
  -Dcompiler.java=12 \
  -Druntime.java=8

Stacktraces:

java.lang.RuntimeException: failed to start nodes
	at org.elasticsearch.test.InternalTestCluster.startAndPublishNodesAndClients(InternalTestCluster.java:1672)
	at org.elasticsearch.test.InternalTestCluster.reset(InternalTestCluster.java:1214)
	at org.elasticsearch.test.InternalTestCluster.beforeTest(InternalTestCluster.java:1110)
	at org.elasticsearch.xpack.CcrIntegTestCase.startClusters(CcrIntegTestCase.java:171)
	[...]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: failed to connect to remote clusters
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.elasticsearch.test.InternalTestCluster.startAndPublishNodesAndClients(InternalTestCluster.java:1667)
	... 39 more
Caused by: java.lang.IllegalStateException: failed to connect to remote clusters
	at org.elasticsearch.transport.RemoteClusterService.initializeRemoteClusters(RemoteClusterService.java:431)
	at org.elasticsearch.transport.TransportService.doStart(TransportService.java:241)
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:61)
	at org.elasticsearch.node.Node.start(Node.java:662)
	at org.elasticsearch.test.InternalTestCluster$NodeAndClient.startNode(InternalTestCluster.java:961)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:677)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: java.util.concurrent.ExecutionException: ConnectTransportException[[leader1][127.0.0.1:53965] general node connection failure]; nested: IllegalStateException[handshake failed with {leader1}{C0InN5g5Q1ywlJOdPNNOwg}{nCYQ3OCCQc23dtHZNr8p9w}{127.0.0.1}{127.0.0.1:53965}{xpack.installed=true}]; nested: ReceiveTimeoutTransportException[[leader1][127.0.0.1:53965][internal:transport/handshake] request_id [4] timed out after [30001ms]];

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-04-10T11:30:27Z

Pinging @elastic/es-distributed

ywelsch · 2019-05-07T07:46:30Z

This looks to be a transport issue. Can you have a look @tbrooks8 ?

Recent failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.1+multijob-darwin-compatibility/10/consoleFull

Relevant log line:

1> [2019-05-06T23:06:23,228][ERROR][o.e.t.n.MockNioTransport ] [follower1] exception from server channel caught on transport layer [sun.nio.ch.ServerSocketChannelImpl[localhost/127.0.0.1:57988]]
  1> java.net.SocketException: Invalid argument
  1> 	at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:?]
  1> 	at sun.nio.ch.Net.setSocketOption(Net.java:334) ~[?:?]
  1> 	at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:190) ~[?:?]
  1> 	at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:271) ~[?:?]
  1> 	at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:306) ~[?:?]
  1> 	at org.elasticsearch.nio.ChannelFactory$RawChannelFactory.configureSocketChannel(ChannelFactory.java:210) ~[elasticsearch-nio-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.ChannelFactory$RawChannelFactory.acceptNioChannel(ChannelFactory.java:185) ~[elasticsearch-nio-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.ChannelFactory.acceptNioChannel(ChannelFactory.java:55) ~[elasticsearch-nio-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.ServerChannelContext.acceptChannels(ServerChannelContext.java:47) ~[elasticsearch-nio-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.EventHandler.acceptChannel(EventHandler.java:45) ~[elasticsearch-nio-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:227) ~[elasticsearch-nio-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[elasticsearch-nio-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129) ~[elasticsearch-nio-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

dnhatn · 2019-05-26T14:59:27Z

Similar issue #41071.

dnhatn · 2019-06-11T22:11:53Z

Another instance https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.1+multijob-darwin-compatibility/15/consoleText.

 1> [2019-06-11T10:53:49,883][ERROR][o.e.t.n.MockNioTransport ] [followerm1] exception from server channel caught on transport layer [sun.nio.ch.ServerSocketChannelImpl[localhost/127.0.0.1:52943]]
  1> java.net.SocketException: Invalid argument
  1> 	at sun.nio.ch.Net.setIntOption0(Native Method) ~[?:?]
  1> 	at sun.nio.ch.Net.setSocketOption(Net.java:334) ~[?:?]
  1> 	at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:190) ~[?:?]
  1> 	at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:271) ~[?:?]
  1> 	at sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:306) ~[?:?]
  1> 	at org.elasticsearch.nio.ChannelFactory$RawChannelFactory.configureSocketChannel(ChannelFactory.java:210) ~[elasticsearch-nio-7.1.2-SNAPSHOT.jar:7.1.2-SNAPSHOT]

org.elasticsearch.xpack.ccr.AutoFollowIT > testAutoFollowSoftDeletesDisabled FAILED
    java.lang.RuntimeException: failed to start nodes

        Caused by:
        java.util.concurrent.ExecutionException: java.lang.IllegalStateException: failed to connect to remote clusters

            Caused by:
            java.lang.IllegalStateException: failed to connect to remote clusters

                Caused by:
                java.util.concurrent.ExecutionException: ConnectTransportException[[][127.0.0.1:52858] connect_timeout[30s]]

                    Caused by:
                    ConnectTransportException[[][127.0.0.1:52858] connect_timeout[30s]]

ywelsch · 2019-06-12T10:23:40Z

@tbrooks8 can you have a look?

Tim-Brooks · 2019-06-13T15:33:31Z

My best guess is that this is similar to this: envoyproxy/envoy#1446

ywelsch · 2019-06-17T09:23:56Z

@tbrooks8 interesting find. How do you think we should fix this? Should we just log a warning on OS X and move on? Could we possibly delay setting the option until we're fully connected?

Tim-Brooks · 2019-06-18T16:48:23Z

We could delay setting the socket options until connection is complete. That would require a little reworking how how things are now. Possibly just add a connection complete future to set them.

I'll think about it submit a PR sometime this week.

Brings some temporary relief for test failures until #41071 is addressed.

ebadyano · 2019-07-22T15:34:31Z

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-darwin-compatibility/102/consoleFull

jdconrad · 2019-07-26T21:40:15Z

Another:
https://scans.gradle.com/s/fk4yve5kmpzcy/console-log

Currently in the transport-nio work we connect and bind channels on the a thread before the channel is registered with a selector. Additionally, it is at this point that we set all the socket options. This commit moves these operations onto the event-loop after the channel has been registered with a selector. It attempts to set the socket options for a non-server channel at registration time. If that fails, it will attempt to set the options after the channel is connected. This should fix #41071.

Currently in the transport-nio work we connect and bind channels on the a thread before the channel is registered with a selector. Additionally, it is at this point that we set all the socket options. This commit moves these operations onto the event-loop after the channel has been registered with a selector. It attempts to set the socket options for a non-server channel at registration time. If that fails, it will attempt to set the options after the channel is connected. This should fix elastic#41071.

Currently in the transport-nio work we connect and bind channels on the a thread before the channel is registered with a selector. Additionally, it is at this point that we set all the socket options. This commit moves these operations onto the event-loop after the channel has been registered with a selector. It attempts to set the socket options for a non-server channel at registration time. If that fails, it will attempt to set the options after the channel is connected. This should fix #41071.

cbuescher added >test-failure Triaged test failures from CI :Distributed/CCR Issues around the Cross Cluster State Replication features labels Apr 10, 2019

ywelsch assigned Tim-Brooks May 7, 2019

Tim-Brooks mentioned this issue Jun 14, 2019

[CI] CcrRepositoryIT.testIndividualActionsTimeout failing #42499

Closed

Tim-Brooks mentioned this issue Jun 28, 2019

Move nio channel initialization to event loop #43780

Merged

This was referenced Jul 15, 2019

Multipe org.elasticsearch.xpack.ccr.CcrAliasesIT failing on Mac #44353

Closed

Ignore failures to set socket options on Mac #44355

Merged

ywelsch added a commit that referenced this issue Jul 17, 2019

Ignore failures to set socket options on Mac (#44355)

855d27e

Brings some temporary relief for test failures until #41071 is addressed.

ywelsch added a commit that referenced this issue Jul 17, 2019

Ignore failures to set socket options on Mac (#44355)

c8b66c5

Brings some temporary relief for test failures until #41071 is addressed.

ywelsch mentioned this issue Jul 29, 2019

Do not block transport thread on startup #44939

Merged

Tim-Brooks closed this as completed in f39e8e5 Jul 30, 2019

Tim-Brooks mentioned this issue Aug 2, 2019

Move nio channel initialization to event loop #45155

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AutoFollowIT#testCleanFollowedLeaderIndexUUIDs failures #41071

AutoFollowIT#testCleanFollowedLeaderIndexUUIDs failures #41071

cbuescher commented Apr 10, 2019

elasticmachine commented Apr 10, 2019

ywelsch commented May 7, 2019

dnhatn commented May 26, 2019

dnhatn commented Jun 11, 2019

ywelsch commented Jun 12, 2019

Tim-Brooks commented Jun 13, 2019

ywelsch commented Jun 17, 2019

Tim-Brooks commented Jun 18, 2019

ebadyano commented Jul 22, 2019

jdconrad commented Jul 26, 2019

AutoFollowIT#testCleanFollowedLeaderIndexUUIDs failures #41071

AutoFollowIT#testCleanFollowedLeaderIndexUUIDs failures #41071

Comments

cbuescher commented Apr 10, 2019

elasticmachine commented Apr 10, 2019

ywelsch commented May 7, 2019

dnhatn commented May 26, 2019

dnhatn commented Jun 11, 2019

ywelsch commented Jun 12, 2019

Tim-Brooks commented Jun 13, 2019

ywelsch commented Jun 17, 2019

Tim-Brooks commented Jun 18, 2019

ebadyano commented Jul 22, 2019

jdconrad commented Jul 26, 2019