Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] CloseIndexIT.testCloseIndex failed #40024

Closed
bizybot opened this issue Mar 14, 2019 · 3 comments · Fixed by #41660
Closed

[CI] CloseIndexIT.testCloseIndex failed #40024

bizybot opened this issue Mar 14, 2019 · 3 comments · Fixed by #41660
Assignees
@dnhatn
Labels
:Data Management/Indices APIs APIs to create and manage indices and templates :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. >test-failure Triaged test failures from CI

Comments

@bizybot
Copy link
Contributor

bizybot commented Mar 14, 2019

CloseIndexIT.testCloseIndex failed on
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+internalClusterTest/2283/console

Unable to reproduce locally with the following command:

./gradlew :server:integTest \
-Dtests.seed=82A3B95002BA7D9B \
-Dtests.class=org.elasticsearch.indices.state.CloseIndexIT \
-Dtests.method="testCloseIndex" \
-Dtests.security.manager=true \
-Dtests.locale=fr-CA \
-Dtests.timezone=America/Tortola \
-Dcompiler.java=11 \
-Druntime.java=8
@bizybot bizybot added >test-failure Triaged test failures from CI :Data Management/Indices APIs APIs to create and manage indices and templates labels Mar 14, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@gwbrown
Copy link
Contributor

gwbrown commented Mar 15, 2019 •
edited
Loading

This build appears to have been rotated out from Jenkins. From the build stats kibana, the stack trace is:

java.lang.AssertionError: max seq. no. [-1] does not match [2]
	at __randomizedtesting.SeedInfo.seed([82A3B95002BA7D9B]:0)
	at org.elasticsearch.index.engine.ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:142)
	at org.elasticsearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:116)
	at org.elasticsearch.index.engine.NoOpEngine.<init>(NoOpEngine.java:40)
	at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1442)
	at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1395)
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:424)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95)
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302)
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1681)
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2330)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:677)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Looks like this might be a failure related to sequence numbers and the global checkpoint - adding the distributed team, could someone take a look?

@gwbrown gwbrown added the :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. label Mar 15, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn dnhatn self-assigned this Mar 15, 2019
@dnhatn dnhatn mentioned this issue Apr 30, 2019
Merged
dnhatn added a commit that referenced this issue May 22, 2019
@dnhatn
Peer recovery should flush at the end (#41660)
75be2a6 
Flushing at the end of a peer recovery (if needed) can bring these
benefits:

1. Closing an index won't end up with the red state for a recovering
replica should always be ready for closing whether it performs the
verifying-before-close step or not.

2. Good opportunities to compact store (i.e., flushing and merging
Lucene, and trimming translog)

Closes #40024
Closes #39588
dnhatn added a commit that referenced this issue May 22, 2019
@dnhatn
Peer recovery should flush at the end (#41660)
bcbf1af 
Flushing at the end of a peer recovery (if needed) can bring these
benefits:

1. Closing an index won't end up with the red state for a recovering
replica should always be ready for closing whether it performs the
verifying-before-close step or not.

2. Good opportunities to compact store (i.e., flushing and merging
Lucene, and trimming translog)

Closes #40024
Closes #39588
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this issue May 27, 2019
@dnhatn
Peer recovery should flush at the end (elastic#41660)
ba74214 
Flushing at the end of a peer recovery (if needed) can bring these
benefits:

1. Closing an index won't end up with the red state for a recovering
replica should always be ready for closing whether it performs the
verifying-before-close step or not.

2. Good opportunities to compact store (i.e., flushing and merging
Lucene, and trimming translog)

Closes elastic#40024
Closes elastic#39588
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Data Management/Indices APIs APIs to create and manage indices and templates :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Peer recovery should flush at the end dnhatn/elasticsearch
4 participants
@bizybot @gwbrown @dnhatn @elasticmachine