Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize sequence numbers on a shrunken index #25321

Merged
merged 4 commits into from Jun 21, 2017

Conversation

@jasontedor
Copy link
Member

commented Jun 20, 2017

Bringing together shards in a shrunken index means that we need to address the start of history for the shrunken index. The problem here is that sequence numbers before the maximum of the maximum sequence numbers on the source shards can collide in the target shards in the shrunken index. To address this, we set the maximum sequence number and the local checkpoint on the target shards to this maximum of the maximum sequence numbers. This enables correct document-level semantics for documents indexed before the shrink, and history on the shrunken index will effectively start from here.

Relates #10708

Bringing together shards in a shrunken index means that we need to
address the start of history for the shrunken index. The problem here is
that sequence numbers before the maximum of the maximum sequence numbers
on the source shards can collide in the target shards in the shrunken
index. To address this, we set the maximum sequence number and the local
checkpoint on the target shards to this maximum of the maximum sequence
numbers. This enables correct document-level semantics for documents
indexed before the shrink, and history on the shrunken index will
effectively start from here.
@jasontedor jasontedor requested a review from bleskes Jun 20, 2017
@bleskes bleskes referenced this pull request Jun 20, 2017
57 of 64 tasks complete
Copy link
Member

left a comment

.collect(Collectors.toList()).toArray(new Directory[shards.size()]));

final Directory[] sources =
shards.stream().map(LocalShardSnapshot::getSnapshotDirectory).collect(Collectors.toList()).toArray(new Directory[0]);

This comment has been minimized.

Copy link
@bleskes

bleskes Jun 21, 2017

Member

Nit - there is a stream::toArray(Directory[]::new))

writer.setLiveCommitData(() -> {
final HashMap<String, String> liveCommitData = new HashMap<>(2);
liveCommitData.put(SequenceNumbers.MAX_SEQ_NO, Long.toString(maxSeqNo));
liveCommitData.put(SequenceNumbers.LOCAL_CHECKPOINT_KEY, Long.toString(maxSeqNo));

This comment has been minimized.

Copy link
@bleskes

bleskes Jun 21, 2017

Member

Is the plan to do MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID as a follow up?

This comment has been minimized.

Copy link
@jasontedor

jasontedor Jun 21, 2017

Author Member

Yes, I edited the plan on #10708 to separate these out into separate line items, I do not like mixing things. I do forgive you for not seeing this. 😛

@@ -233,7 +228,8 @@ public void testCreateShrinkIndex() {
.put("number_of_shards", randomIntBetween(2, 7))
.put("index.version.created", version)
).get();
for (int i = 0; i < 20; i++) {
final int docs = randomIntBetween(1, 128);

This comment has been minimized.

Copy link
@bleskes

bleskes Jun 21, 2017

Member

do we want to test with 0 docs too?

This comment has been minimized.

Copy link
@jasontedor

jasontedor Jun 21, 2017

Author Member

Sure!

jasontedor added 2 commits Jun 21, 2017
@jasontedor jasontedor merged commit cc67d02 into elastic:master Jun 21, 2017
2 checks passed
2 checks passed
CLA Commit author is a member of Elasticsearch
Details
elasticsearch-ci Build finished.
Details
@clintongormley clintongormley added v6.0.0-beta1 and removed v6.0.0 labels Jul 25, 2017
@colings86 colings86 added the >feature label Jul 25, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.