-
Notifications
You must be signed in to change notification settings - Fork 24.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make Snapshot Logic Write Metadata after Segments #45689
Make Snapshot Logic Write Metadata after Segments #45689
Conversation
original-brownbear
commented
Aug 18, 2019
•
edited
Loading
edited
- Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates as explained in Snapshots should write out IndexMetaData after shard data #41581
- Keep the old behavior of writing the metadata beforehand in the case of mixed version clusters for BwC reasons
- Still overwrite the metadata in the end, so even a mixed version cluster is fixed by this change if a newer version master does the finalization
- Fixes Snapshots should write out IndexMetaData after shard data #41581
* WIP, this still needs a docs fixup * Fixes elastic#41581 * Fixes elastic#25281
Pinging @elastic/es-distributed |
server/src/test/java/org/elasticsearch/snapshots/mockstore/MockRepository.java
Outdated
Show resolved
Hide resolved
Jenkins run elasticsearch-ci/1 |
Jenkins run elasticsearch-ci/1 |
@@ -3856,76 +3811,6 @@ public void testParallelRestoreOperationsFromSingleSnapshot() throws Exception { | |||
assertThat(client.prepareGet(restoredIndexName2, typeName, sameSourceIndex ? docId : docId2).get().isExists(), equalTo(true)); | |||
} | |||
|
|||
public void testAbortedSnapshotDuringInitDoesNotStart() throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test doesn't really apply anymore with snapshot initialization gone.
@@ -276,9 +276,11 @@ public void writeBlob(String blobName, InputStream inputStream, long blobSize, b | |||
// We do some checks in case there is a consistent state for a blob to prevent turning it inconsistent. | |||
final boolean hasConsistentContent = | |||
relevantActions.size() == 1 && relevantActions.get(0).operation == Operation.PUT; | |||
if (BlobStoreRepository.INDEX_LATEST_BLOB.equals(blobName)) { | |||
if (BlobStoreRepository.INDEX_LATEST_BLOB.equals(blobName) | |||
|| blobName.startsWith(BlobStoreRepository.METADATA_PREFIX)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On master fail-over, we can now have overwriting for meta- blobs here. Since any metadata will be at least as fresh as all the uploaded segments overwrites here should be.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@original-brownbear I left a couple of comments.
server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreIT.java
Outdated
Show resolved
Hide resolved
@@ -2538,28 +2536,15 @@ public void testCloseOrDeleteIndexDuringSnapshot() throws Exception { | |||
|
|||
Client client = client(); | |||
|
|||
boolean allowPartial = randomBoolean(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is that changed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can't block on the initialize
part of the snapshot anymore, so I removed the code paths in this test that required blocking on initialize
. allowPartial == true
required blocking on init => all the paths that had it set to true
were removed.
server/src/test/java/org/elasticsearch/snapshots/SnapshotResiliencyTests.java
Outdated
Show resolved
Hide resolved
Jenkins test this |
Thanks @andrershov ! All fixed I think. |
Jenkins run elasticsearch-ci/2 |
As a result of elastic#45689 snapshot finalization started to take significantly longer than before. This may be a little unfortunate since it increases the likelihood of failing to finalize after having written out all the segment blobs. This change parallelizes all the metadata writes that can safely run in parallel in the finalization step to speed the finalization step up again. Also, this will generally speed up the snapshot process overall in case of large number of indices. This is also a nice to have for elastic#46250 since we add yet another step (deleting of old index- blobs in the shards to the finalization.
As a result of #45689 snapshot finalization started to take significantly longer than before. This may be a little unfortunate since it increases the likelihood of failing to finalize after having written out all the segment blobs. This change parallelizes all the metadata writes that can safely run in parallel in the finalization step to speed the finalization step up again. Also, this will generally speed up the snapshot process overall in case of large number of indices. This is also a nice to have for #46250 since we add yet another step (deleting of old index- blobs in the shards to the finalization.
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates.
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates.
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates.
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates.
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates. (cherry picked from commit 60e1600) # Conflicts: # docs/appendices/release-notes/unreleased.rst
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates. (cherry picked from commit 60e1600)
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates. (cherry picked from commit 60e1600)
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates. (cherry picked from commit 60e1600)
Backport of elastic/elasticsearch#45689 Write metadata during snapshot finalization after segment files to prevent outdated metadata in case of dynamic mapping updates. (cherry picked from commit 60e1600)
With elastic#45689 making it so that index metadata is written after all shards have been snapshotted we can't delete indices that are part of the upcoming snapshot finalization any longer and it is not sufficient to check if all shards of an index have been snapshotted before deciding that it is safe to delete it. This change forbids deleting any index that is in the process of being snapshot to avoid issues during snapshot finalization. Closes elastic#50200
With #45689 making it so that index metadata is written after all shards have been snapshotted we can't delete indices that are part of the upcoming snapshot finalization any longer and it is not sufficient to check if all shards of an index have been snapshotted before deciding that it is safe to delete it. This change forbids deleting any index that is in the process of being snapshot to avoid issues during snapshot finalization. Relates #50200 (doesn't fully fix yet because we're not fixing the `partial=true` snapshot case here
With elastic#45689 making it so that index metadata is written after all shards have been snapshotted we can't delete indices that are part of the upcoming snapshot finalization any longer and it is not sufficient to check if all shards of an index have been snapshotted before deciding that it is safe to delete it. This change forbids deleting any index that is in the process of being snapshot to avoid issues during snapshot finalization. Relates elastic#50200 (doesn't fully fix yet because we're not fixing the `partial=true` snapshot case here
With elastic#45689 making it so that index metadata is written after all shards have been snapshotted we can't delete indices that are part of the upcoming snapshot finalization any longer and it is not sufficient to check if all shards of an index have been snapshotted before deciding that it is safe to delete it. This change forbids deleting any index that is in the process of being snapshot to avoid issues during snapshot finalization. Relates elastic#50200 (doesn't fully fix yet because we're not fixing the `partial=true` snapshot case here
With #45689 making it so that index metadata is written after all shards have been snapshotted we can't delete indices that are part of the upcoming snapshot finalization any longer and it is not sufficient to check if all shards of an index have been snapshotted before deciding that it is safe to delete it. This change forbids deleting any index that is in the process of being snapshot to avoid issues during snapshot finalization. Relates #50200 (doesn't fully fix yet because we're not fixing the `partial=true` snapshot case here
* Fix Index Deletion during Snapshot Finalization (#50202) With #45689 making it so that index metadata is written after all shards have been snapshotted we can't delete indices that are part of the upcoming snapshot finalization any longer and it is not sufficient to check if all shards of an index have been snapshotted before deciding that it is safe to delete it. This change forbids deleting any index that is in the process of being snapshot to avoid issues during snapshot finalization. Relates #50200 (doesn't fully fix yet because we're not fixing the `partial=true` snapshot case here
With elastic#45689 making it so that index metadata is written after all shards have been snapshotted we can't delete indices that are part of the upcoming snapshot finalization any longer and it is not sufficient to check if all shards of an index have been snapshotted before deciding that it is safe to delete it. This change forbids deleting any index that is in the process of being snapshot to avoid issues during snapshot finalization. Relates elastic#50200 (doesn't fully fix yet because we're not fixing the `partial=true` snapshot case here