Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SplitIndexIT fails with AlreadyClosedException #40944

Closed
davidkyle opened this issue Apr 8, 2019 · 4 comments · Fixed by #41006
Closed

SplitIndexIT fails with AlreadyClosedException #40944

davidkyle opened this issue Apr 8, 2019 · 4 comments · Fixed by #41006
Assignees
@original-brownbear
Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >test-failure Triaged test failures from CI

Comments

@davidkyle
Copy link
Member

Numerous failures in the test suite failed with the error engine is closed

java.util.concurrent.ExecutionException: org.apache.lucene.store.AlreadyClosedException: [source][0] engine is closed
	at __randomizedtesting.SeedInfo.seed([5A4A0A922CF76DCF:A8D0613C4A83C12B]:0)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:266)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:253)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:87)
	at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:326)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.lucene.store.AlreadyClosedException: [source][0] engine is closed
	at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:789)
	at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:798)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:829)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:790)
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:763)
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:720)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:263)
	at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:159)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:197)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:116)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:77)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:927)
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:364)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:311)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:269)
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:236)
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2513)
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:865)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:310)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:273)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:747)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	... 1 more
Caused by: java.lang.Exception: test
	at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:316)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	... 1 more

./gradlew :server:integTest \
  -Dtests.seed=5A4A0A922CF76DCF \
  -Dtests.class=org.elasticsearch.action.admin.indices.create.SplitIndexIT \
  -Dtests.method="testSplitIndexPrimaryTerm" \
  -Dtests.security.manager=true \
  -Dtests.locale=uz-Arab-AF \
  -Dtests.timezone=EET \
  -Dcompiler.java=12 \
  -Druntime.java=11

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu-14.04&&immutable/320/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu-16.04&&immutable/320/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/4700/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/4696/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/4670/console
@davidkyle davidkyle added >test-failure Triaged test failures from CI :Data Management/Indices APIs APIs to create and manage indices and templates labels Apr 8, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@davidkyle davidkyle mentioned this issue Apr 8, 2019
Merged
@davidkyle
Copy link
Member Author

The test suite is muted on the master branch in #40945

davidkyle added a commit that referenced this issue Apr 8, 2019
@davidkyle
Mute SplitIndexIT (#40945)
a158d91 
Relates to #40944
@davidkyle davidkyle mentioned this issue Apr 8, 2019
Closed
@original-brownbear original-brownbear self-assigned this Apr 8, 2019
@original-brownbear
Copy link
Member

Looks like this could be a mistake made in #39793. I'm looking into this.

@original-brownbear
Copy link
Member

I think the thing causing this is, is this line https://github.com/elastic/elasticsearch/pull/39793/files#diff-720a796f6beda1dfa6af60b45ffe1010R303.

Previously we let exceptions from primary.applyIndexOperationOnPrimary (like the one we're seeing here) bubble up and be handled upstream but with #39793 we are marking the bulk item as failed and keep going with the request. This seems to be what's causing the issue here (and in #40948 probably).

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Apr 9, 2019
@original-brownbear
Fix Exception Handling for TransportShardBulkAction
c71e36b 
* Prior to elastic#39793 exceptions for the primary write and delete actions
were bubbled up to the caller so that closed shards would be handled accordingly upstream.
 elastic#39793 accidentally changed the behaviour here and simply marked those exceptions as bulk item failures on the request and kept processing bulk request items on closed shards.
* This fix returns to that behaviour and adjusts the listeners passed in `TransportReplicationAction`
such that they behave like the previous synchronous `catch`.
   * Dried up the exception handling slightly for that and inlined all the listeners to make the logic a little
easier to follow
* Reenable SplitIndexIT now that clsoed shards are properly handled again
* Closes elastic#40944
@original-brownbear original-brownbear mentioned this issue Apr 9, 2019
Merged
original-brownbear added a commit that referenced this issue Apr 9, 2019
@original-brownbear
Fix Exception Handling for TransportShardBulkAction (#41006)
1830a49 
* Prior to #39793 exceptions for the primary write and delete actions
were bubbled up to the caller so that closed shards would be handled accordingly upstream.
 #39793 accidentally changed the behaviour here and simply marked those exceptions as bulk item failures on the request and kept processing bulk request items on closed shards.
* This fix returns to that behaviour and adjusts the listeners passed in `TransportReplicationAction`
such that they behave like the previous synchronous `catch`.
   * Dried up the exception handling slightly for that and inlined all the listeners to make the logic a little
easier to follow
* Reenable SplitIndexIT now that clsoed shards are properly handled again
* Closes #40944
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this issue May 27, 2019
@davidkyle
Mute SplitIndexIT (elastic#40945)
807dc83 
Relates to elastic#40944
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this issue May 27, 2019
@original-brownbear
Fix Exception Handling for TransportShardBulkAction (elastic#41006)
b0e27c7 
* Prior to elastic#39793 exceptions for the primary write and delete actions
were bubbled up to the caller so that closed shards would be handled accordingly upstream.
 elastic#39793 accidentally changed the behaviour here and simply marked those exceptions as bulk item failures on the request and kept processing bulk request items on closed shards.
* This fix returns to that behaviour and adjusts the listeners passed in `TransportReplicationAction`
such that they behave like the previous synchronous `catch`.
   * Dried up the exception handling slightly for that and inlined all the listeners to make the logic a little
easier to follow
* Reenable SplitIndexIT now that clsoed shards are properly handled again
* Closes elastic#40944
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Exception Handling for TransportShardBulkAction original-brownbear/elasticsearch
3 participants
@davidkyle @original-brownbear @elasticmachine