Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] FollowerFailOverIT testReadRequestsReturnsLatestMappingVersion failed #37807

Closed
droberts195 opened this issue Jan 24, 2019 · 8 comments · Fixed by #37886 or #38521
Closed

[CI] FollowerFailOverIT testReadRequestsReturnsLatestMappingVersion failed #37807

droberts195 opened this issue Jan 24, 2019 · 8 comments · Fixed by #37886 or #38521
Assignees
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI

Comments

@droberts195
Copy link
Contributor

org.elasticsearch.xpack.ccr.FollowerFailOverIT testReadRequestsReturnsLatestMappingVersion failed in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1474/console

The test got an unexpected InterruptedException:

Uncaught exception in thread: Thread[elasticsearch[leaderd2][clusterApplierService#updateTask][T#1],5,TGRP-FollowerFailOverIT]
java.lang.AssertionError: java.lang.InterruptedException
	at __randomizedtesting.SeedInfo.seed([D0B1EF2765FCDBBC]:0)
	at org.elasticsearch.xpack.ccr.FollowerFailOverIT.lambda$testReadRequestsReturnsLatestMappingVersion$4(FollowerFailOverIT.java:261)
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:487)
	at java.lang.Iterable.forEach(Iterable.java:75)
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:484)
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:471)
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421)
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:662)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
	at org.elasticsearch.xpack.ccr.FollowerFailOverIT.lambda$testReadRequestsReturnsLatestMappingVersion$4(FollowerFailOverIT.java:259)
	... 12 more

The repro command is:

./gradlew :x-pack:plugin:ccr:internalClusterTest \
  -Dtests.seed=97C386C4128137C7 \
  -Dtests.class=org.elasticsearch.xpack.ccr.FollowerFailOverIT \
  -Dtests.method="testReadRequestsReturnsLatestMappingVersion" \
  -Dtests.security.manager=true \
  -Dtests.locale=fi \
  -Dtests.timezone=Europe/Mariehamn \
  -Dcompiler.java=11 \
  -Druntime.java=8

This did not reproduce locally for me.

@droberts195 droberts195 added >test-failure Triaged test failures from CI :Distributed/CCR Issues around the Cross Cluster State Replication features labels Jan 24, 2019
@droberts195
Copy link
Contributor Author

I muted the test on master in f12bfb4

@alpar-t
Copy link
Contributor

alpar-t commented Jan 24, 2019

This reproduces for me locally with:

 ./gradlew :x-pack:plugin:ccr:internalClusterTest -Dtests.seed=FE7DE10DFB70778B -Dtests.class=org.elasticsearch.xpack.ccr.FollowerFailOverIT 

@droberts195
Copy link
Contributor Author

The actual test failure in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1474/consoleText was:

ERROR   60.8s J5 | FollowerFailOverIT.testReadRequestsReturnsLatestMappingVersion <<< FAILURES!
   > Throwable #1: ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping) within 23s]
   > 	at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:134)
   > 	at java.util.ArrayList.forEach(ArrayList.java:1257)
   > 	at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:133)
   > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:662)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   > 	at java.lang.Thread.run(Thread.java:748)Throwable #2: java.lang.AssertionError: AcknowledgedResponse failed - not acked
   > Expected: <true>
   >      but: was <false>
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:112)
   > 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:116)
   > 	at org.elasticsearch.test.TestCluster.wipeIndices(TestCluster.java:142)
   > 	at org.elasticsearch.test.TestCluster.wipe(TestCluster.java:79)
   > 	at org.elasticsearch.xpack.CcrIntegTestCase.afterTest(CcrIntegTestCase.java:178)
   > 	at java.lang.Thread.run(Thread.java:748)

@droberts195
Copy link
Contributor Author

The same problem occurred on 6.x too, in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/1145/consoleText

The test assertion error is:

ERROR   63.8s J6 | FollowerFailOverIT.testReadRequestsReturnsLatestMappingVersion <<< FAILURES!
   > Throwable #1: ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping) within 14s]
   > 	at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:128)
   > 	at java.util.ArrayList.forEach(ArrayList.java:1257)
   > 	at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:127)
   > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:662)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   > 	at java.lang.Thread.run(Thread.java:748)Throwable #2: java.lang.AssertionError: AcknowledgedResponse failed - not acked
   > Expected: <true>
   >      but: was <false>
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:109)
   > 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:113)
   > 	at org.elasticsearch.test.TestCluster.wipeIndices(TestCluster.java:142)
   > 	at org.elasticsearch.test.TestCluster.wipe(TestCluster.java:79)
   > 	at org.elasticsearch.xpack.CcrIntegTestCase.afterTest(CcrIntegTestCase.java:182)
   > 	at java.lang.Thread.run(Thread.java:748)

And the InterruptedException is:

ERROR   14.8s J6 | FollowerFailOverIT.testAddNewReplicasOnFollower <<< FAILURES!
   > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=602, name=elasticsearch[leaderd2][clusterApplierService#updateTask][T#1], state=RUNNABLE, group=TGRP-FollowerFailOverIT]
   > 	at __randomizedtesting.SeedInfo.seed([F6E560D2FC81DB13:4A344E072C3C7893]:0)
   > Caused by: java.lang.AssertionError: java.lang.InterruptedException
   > 	at __randomizedtesting.SeedInfo.seed([F6E560D2FC81DB13]:0)
   > 	at org.elasticsearch.xpack.ccr.FollowerFailOverIT.lambda$testReadRequestsReturnsLatestMappingVersion$4(FollowerFailOverIT.java:261)
   > 	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:485)
   > 	at java.lang.Iterable.forEach(Iterable.java:75)
   > 	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:482)
   > 	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:469)
   > 	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:420)
   > 	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:164)
   > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:662)
   > 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
   > 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.lang.InterruptedException
   > 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
   > 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
   > 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
   > 	at org.elasticsearch.xpack.ccr.FollowerFailOverIT.lambda$testReadRequestsReturnsLatestMappingVersion$4(FollowerFailOverIT.java:259)
   > 	... 12 more

The repro command from this build was:

./gradlew :x-pack:plugin:ccr:internalClusterTest \
  -Dtests.seed=F6E560D2FC81DB13 \
  -Dtests.class=org.elasticsearch.xpack.ccr.FollowerFailOverIT \
  -Dtests.method="testReadRequestsReturnsLatestMappingVersion" \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-MA \
  -Dtests.timezone=Africa/Asmera \
  -Dcompiler.java=11 \
  -Druntime.java=8

@droberts195
Copy link
Contributor Author

I muted the test on 6.x in c162771

@dnhatn
Copy link
Member

dnhatn commented Jan 24, 2019

I am on it today.

dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Jan 25, 2019
If the index request is executed before the mapping update is applied on
the IndexShard, the index request will perform a dynamic mapping update.
This mapping update will be timeout (i.e, ProcessClusterEventTimeoutException)
because the latch is not open. This leads to the failure of the index
request and the test. This commit makes sure the mapping is ready
before we execute the index request.

Closes elastic#37807
dnhatn added a commit that referenced this issue Jan 28, 2019
If the index request is executed before the mapping update is applied on
the IndexShard, the index request will perform a dynamic mapping update.
This mapping update will be timeout (i.e, ProcessClusterEventTimeoutException)
because the latch is not open. This leads to the failure of the index
request and the test. This commit makes sure the mapping is ready
before we execute the index request.

Closes #37807
dnhatn added a commit that referenced this issue Jan 31, 2019
If the index request is executed before the mapping update is applied on
the IndexShard, the index request will perform a dynamic mapping update.
This mapping update will be timeout (i.e, ProcessClusterEventTimeoutException)
because the latch is not open. This leads to the failure of the index
request and the test. This commit makes sure the mapping is ready
before we execute the index request.

Closes #37807
@original-brownbear
Copy link
Member

Seems this happened again in master today in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/549/console

REPRODUCE WITH: ./gradlew :x-pack:plugin:ccr:internalClusterTest \
  -Dtests.seed=A97E0712E7C05875 \
  -Dtests.class=org.elasticsearch.xpack.ccr.FollowerFailOverIT \
  -Dtests.method="testReadRequestsReturnLatestMappingVersion" \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-AE \
  -Dtests.timezone=America/Miquelon \
  -Dcompiler.java=11 \
  -Druntime.java=8
FAILURE  147s J6 | FollowerFailOverIT.testReadRequestsReturnLatestMappingVersion <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 	at __randomizedtesting.SeedInfo.seed([A97E0712E7C05875:AB46D32F2D14C3EE]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:858)
   > 	at org.elasticsearch.xpack.CcrIntegTestCase.assertIndexFullyReplicatedToFollower(CcrIntegTestCase.java:448)
   > 	at org.elasticsearch.xpack.ccr.FollowerFailOverIT.testReadRequestsReturnLatestMappingVersion(FollowerFailOverIT.java:294)
   > 	at java.lang.Thread.run(Thread.java:748)
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: 
   > Expected: <{0=[DocIdSeqNoAndTerm{id='1 seqNo=0 primaryTerm=1}]}>
   >      but: was <{0=[]}>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.CcrIntegTestCase.lambda$assertIndexFullyReplicatedToFollower$4(CcrIntegTestCase.java:451)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 		... 39 more

@original-brownbear
Copy link
Member

Muted this one in master 887fa2c

dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Feb 9, 2019
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Feb 9, 2019
dnhatn added a commit that referenced this issue Feb 10, 2019
dnhatn added a commit that referenced this issue Feb 10, 2019
dnhatn added a commit that referenced this issue Feb 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI
Projects
None yet
4 participants