[CI] AutoFollowIT.testCleanFollowedLeaderIndexUUIDs failure to get remote license

[tvernum]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/127/

Doesn't reproduce

./gradlew :x-pack:plugin:ccr:internalClusterTest -Dtests.seed=8B7CD44168C1888C -Dtests.class=org.elasticsearch.xpack.ccr.AutoFollowIT -Dtests.method="testCleanFollowedLeaderIndexUUIDs" -Dtests.security.manager=true -Dtests.locale=sr-Latn-RS -Dtests.timezone=Pacific/Guam -Dcompiler.java=11 -Druntime.java=8

03:15:41   1> [2018-12-19T13:14:50,195][INFO ][o.e.x.c.AutoFollowIT     ] [testCleanFollowedLeaderIndexUUIDs] after test
03:15:41 ERROR   16.6s J0 | AutoFollowIT.testCleanFollowedLeaderIndexUUIDs <<< FAILURES!
03:15:41    > Throwable #1: ElasticsearchStatusException[can not fetch remote cluster state as the license state of the remote cluster [leader_cluster] could not be determined]; nested: ElasticsearchException[could not determine the license type for cluster [leader_cluster]]; nested: RemoteTransportException[[java.lang.NullPointerException]]; nested: ResponseHandlerFailureTransportException[java.lang.NullPointerException]; nested: NullPointerException;
03:15:41    > 	at __randomizedtesting.SeedInfo.seed([8B7CD44168C1888C:C2E1D68121E99BF7]:0)
03:15:41    > 	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.clusterStateUnknownRemoteLicense(CcrLicenseChecker.java:433)
03:15:41    > 	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.lambda$checkRemoteClusterLicenseAndFetchClusterState$5(CcrLicenseChecker.java:174)
03:15:41    > 	at org.elasticsearch.xpack.ccr.CcrLicenseChecker$1.onFailure(CcrLicenseChecker.java:225)
03:15:41    > 	at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onFailure(RemoteClusterLicenseChecker.java:180)
03:15:41    > 	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50)
03:15:41    > 	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)
03:15:41    > 	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1104)
03:15:41    > 	at org.elasticsearch.transport.TcpTransport.lambda$handleException$22(TcpTransport.java:1208)
03:15:41    > 	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:140)
03:15:41    > 	at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1206)
03:15:41    > 	at org.elasticsearch.transport.TcpTransport.access$900(TcpTransport.java:111)
03:15:41    > 	at org.elasticsearch.transport.TcpTransport$2.onFailure(TcpTransport.java:1177)
03:15:41    > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
03:15:41    > 	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:140)
03:15:41    > 	at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1174)
03:15:41    > 	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1129)
03:15:41    > 	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:913)
03:15:41    > 	at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:940)
03:15:41    > 	at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:236)
03:15:41    > 	at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213)
03:15:41    > 	at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:47)
03:15:41    > 	at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119)
03:15:41    > 	at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:369)
03:15:41    > 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:230)
03:15:41    > 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:158)
03:15:41    > 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:123)
03:15:41    > 	at java.lang.Thread.run(Thread.java:748)
03:15:41    > Caused by: ElasticsearchException[could not determine the license type for cluster [leader_cluster]]; nested: RemoteTransportException[[java.lang.NullPointerException]]; nested: ResponseHandlerFailureTransportException[java.lang.NullPointerException]; nested: NullPointerException;
03:15:41    > 	... 24 more
03:15:41    > Caused by: RemoteTransportException[[java.lang.NullPointerException]]; nested: ResponseHandlerFailureTransportException[java.lang.NullPointerException]; nested: NullPointerException;
03:15:41    > Caused by: ResponseHandlerFailureTransportException[java.lang.NullPointerException]; nested: NullPointerException;
03:15:41    > Caused by: java.lang.NullPointerException
03:15:41    > 	at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:162)
03:15:41    > 	at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:157)
03:15:41    > 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
03:15:41    > 	at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:48)
03:15:41    > 	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1094)

I wonder if this could be the same root cause as #36764?

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

This failure is different, a NPE in a response callback.

[martijnvg]

I suspect that the trail license isn't generated yet in the remote cluster and that is causing this failure.
I will change the test to wait for trail license being generated in both clusters. Also I will change RemoteClusterLicenseChecker to deal with the fact that no LicenseInfo can be returned from xpack info api.

[martijnvg]

I've adjusted the ccr tests to wait for license metadata to be available:

• 3cc0cf0 (master)
• eea2345 (6.x)
• 2410af5 (6.5)

[martijnvg]

This specific test failure hasn't occurred since the above commits have been pushed.