Make peer recovery work with archive data #81522
Conversation
server/src/main/java/org/elasticsearch/indices/recovery/PeerRecoveryTargetService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java
Outdated
Show resolved
Hide resolved
| @@ -169,6 +170,10 @@ public void write(Directory directory, SegmentInfo segmentInfo, String segmentSu | |||
| private static FieldInfos filterFields(FieldInfos fieldInfos) { | |||
| List<FieldInfo> fieldInfoCopy = new ArrayList<>(fieldInfos.size()); | |||
| for (FieldInfo fieldInfo : fieldInfos) { | |||
| // omit sequence number field so that it doesn't interfere with peer recovery | |||
| if (fieldInfo.name.equals(SeqNoFieldMapper.NAME)) { | |||
There was a problem hiding this comment.
With this, LuceneChangesSnapshot will just return empty set of documents instead of failing because the field has docvalues mapped to none now.
In a follow-up, I want to explore exposing doc-values of older indices (in particular _seq_no field and soft-deletes)
| @@ -93,26 +86,6 @@ private static void ensureSnapshotIsLoaded(IndexShard indexShard) { | |||
| : "loading snapshot must not be called twice unless we are retrying a peer recovery"; | |||
| } | |||
|
|
|||
| private static void associateNewEmptyTranslogWithIndex(IndexShard indexShard) { | |||
There was a problem hiding this comment.
this is now gone. Woop woop
ywelsch
added
:Distributed/Recovery
|
Pinging @elastic/es-distributed (Team:Distributed) |
server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java
Outdated
Show resolved
Hide resolved
|
Unfortunately BWC tests were disabled on master when I branched my PR off, so these only ran now when I merged in latest master - AND FAILED. I had to take a different approach to hooking into the recovery logic (as I have to cover the case where the peer recovery source is on an old version). The good news is that the hook is simpler than the one we had before. Sorry for the inconvenience of having to review this again. |
| final Store store = indexShard.store(); | ||
| store.incRef(); | ||
| try { | ||
| StoreRecovery.bootstrap(indexShard, store); |
There was a problem hiding this comment.
With this new approach, we now have sequence-number based recoveries for searchable snapshots, which means that there's no need any longer to send over any file (i.e. the exception of ^recovery\..*\.segments_.*$ in InMemoryNoOpCommitDirectory.ensureMutable) except for the BWC case (when primary was on older node and created a different history uuid).
server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
Outdated
Show resolved
Hide resolved
Adapts peer recovery so that it properly integrates with the hook to convert old indices. Relates elastic#81210
Adapts peer recovery so that it properly integrates with the hook to convert old indices.
Relates #81210