Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip global checkpoint sync for closed indices #41874

Merged
merged 7 commits into from May 21, 2019

Conversation

Projects
None yet
4 participants
@dnhatn
Copy link
Contributor

commented May 7, 2019

The verifying-before-close step ensures the global checkpoints on all shard copies are in sync; thus, we don' t need to sync global checkpoints for closed indices.

Relate #33888 (comment)

@elasticmachine

This comment has been minimized.

Copy link

commented May 7, 2019

@dnhatn dnhatn referenced this pull request May 7, 2019

Closed

Replicate closed indices #33888

50 of 50 tasks complete

@dnhatn dnhatn changed the title Closed indices should not sync global checkpoint Do not fail global checkpoint sync on closed indices May 7, 2019

@henningandersen
Copy link
Contributor

left a comment

Would it be an option to skip resync for closed indices? I think we rely on either having a perfect commit to use or doing a file based recovery anyway.


@Override
protected boolean allowClosedIndices() {
return true; // it's okay to sync global checkpoints on closed indices

This comment has been minimized.

Copy link
@henningandersen

henningandersen May 8, 2019

Contributor

This looks dangerous, since ReadOnlyEngine.syncTransLog() does nothing. So we will accept the global checkpoint on replica, but it will not be persisted.

Looks like RecoveryTarget.finalizeRecovery also relies on that method, so maybe we should implement it. I think this means that the global checkpoint is not persisted, but the replica is marked in sync. If this replica then later becomes primary after for instance a full restart, it could have an old global checkpoint and would fail the shard? Probably a different issue/PR (if I am right).

@dnhatn

This comment has been minimized.

Copy link
Contributor Author

commented May 8, 2019

@henningandersen Thanks for looking.

Would it be an option to skip resync for closed indices?

Yes, we can skip as resync is a noop for closed indices. I explored this option then prefer other option to avoid adding one more exception for closed indices. I can make this change again - it's pretty contained.

This looks dangerous, since ReadOnlyEngine.syncTransLog() does nothing. So we will accept the global checkpoint on replica, but it will not be persisted.

We can add an assertion to make sure that the persisted global checkpoint equals to the in-memory value.

@dnhatn dnhatn changed the title Do not fail global checkpoint sync on closed indices Skip global checkpoint sync for closed indices May 20, 2019

@dnhatn

This comment has been minimized.

Copy link
Contributor Author

commented May 20, 2019

@henningandersen I discussed this with Tanguy and Yannick during GAH. We need to keep resync; otherwise, replicas won't see the new primary term. I updated this PR to skip global checkpoint sync for closed indices. Can you please take another look? Thank you!

@dnhatn dnhatn requested a review from henningandersen May 20, 2019

@ywelsch
Copy link
Contributor

left a comment

LGTM

@dnhatn

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2019

@dnhatn dnhatn merged commit 5785941 into elastic:master May 21, 2019

8 checks passed

CLA All commits in pull request signed
Details
elasticsearch-ci/1 Build finished.
Details
elasticsearch-ci/2 Build finished.
Details
elasticsearch-ci/bwc Build finished.
Details
elasticsearch-ci/default-distro Build finished.
Details
elasticsearch-ci/docbldesx Build finished.
Details
elasticsearch-ci/oss-distro-docs Build finished.
Details
elasticsearch-ci/packaging-sample Build finished.
Details

@dnhatn dnhatn deleted the dnhatn:donot_sync_checkpoint branch May 21, 2019

dnhatn added a commit that referenced this pull request May 22, 2019

Skip global checkpoint sync for closed indices (#41874)
The verifying-before-close step ensures the global checkpoints on all
shard copies are in sync; thus, we don' t need to sync global
checkpoints for closed indices.

Relate #33888

gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this pull request May 27, 2019

Skip global checkpoint sync for closed indices (elastic#41874)
The verifying-before-close step ensures the global checkpoints on all
shard copies are in sync; thus, we don' t need to sync global
checkpoints for closed indices.

Relate elastic#33888
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.