Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IndexService - synchronize close to prevent race condition with shard creation #8557

Closed
wants to merge 2 commits into from

Conversation

Projects
None yet
3 participants
@bleskes
Copy link
Member

commented Nov 19, 2014

During node shutdown we have a race condition between processing cluster state updates (creating shards) and closing down the index service. This may cause shards to leak and not be closed properly.

This commit removes the concurrency in shard closing as InternalIndexService.removeShard has been synchronized for a long time now.

On the other hand, the commit restores the parallel shutdown of indices lost in 7e1d8a6

This is the cause of http://build-us-00.elasticsearch.org/job/es_core_1x_strong/1406/testReport/junit/org.elasticsearch.gateway.local/LocalGatewayIndexStateTests/testDanglingIndicesNoAutoImport/

java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still open locks: {write.lock=java.lang.RuntimeException: lock "write.lock" was not released}

Caused by: java.lang.RuntimeException: lock "write.lock" was not released
    at org.apache.lucene.store.MockLockFactoryWrapper$MockLock.obtain(MockLockFactoryWrapper.java:74)
    at org.apache.lucene.store.Lock.obtain(Lock.java:77)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
    at org.elasticsearch.index.engine.internal.InternalEngine.createWriter(InternalEngine.java:1453)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:273)
    at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:733)
Internal: IndexService - synchronize close to prevent race condition …
…with shard creation

During node shutdown we have a race condition between processing cluster state updates (creating shards) and closing down the index service. This may cause shards to leak and not be closed properly.

 This commit removes the concurrency in shard closing as InternalIndexService.removeShard has been synchronized for a long time now.

 On the other hand, the commit restores the parallel shutdown of indices lost in 7e1d8a6
@s1monw

This comment has been minimized.

indexInjector and indexservice can be final here I think

@s1monw

This comment has been minimized.

Copy link
Contributor

commented Nov 19, 2014

left a tiny comment LGTM otherwise

bleskes added a commit that referenced this pull request Nov 19, 2014

Internal: IndexService - synchronize close to prevent race condition …
…with shard creation

During node shutdown we have a race condition between processing cluster state updates (creating shards) and closing down the index service. This may cause shards to leak and not be closed properly.

This commit removes the concurrency in shard closing as InternalIndexService.removeShard has been synchronized for a long time now.

On the other hand, the commit restores the parallel shutdown of indices lost in 7e1d8a6

Closes #8557

@bleskes bleskes closed this in fb81a32 Nov 19, 2014

@bleskes bleskes deleted the bleskes:index_service_sync_close branch Nov 19, 2014

@bleskes

This comment has been minimized.

Copy link
Member Author

commented Nov 19, 2014

Tiny comment applied. Thx!

@clintongormley clintongormley added :Internal and removed review labels Mar 19, 2015

@clintongormley clintongormley changed the title Internal: IndexService - synchronize close to prevent race condition with shard creation IndexService - synchronize close to prevent race condition with shard creation Jun 7, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.