Integrate translog recovery into Engine / InternalEngine #10452
Conversation
s1monw
force-pushed
the
recover_from_translog_in_engine
branch
from
7a33de8
to
4e6e2ed
Compare
|
@bleskes if you have a day or two to review ;) |
| /** | ||
| * Returns <code>true</code> if initial translog recovery should be skipped. Otherwise <code>false</code> | ||
| */ | ||
| public boolean isSkipInitialRecovery() { |
There was a problem hiding this comment.
I know the boolean pattern uses "is" but I think this would make more sense named shouldSkipInitialRecovery?
s1monw
force-pushed
the
recover_from_translog_in_engine
branch
3 times, most recently
from
5150dbd
to
4517a95
Compare
| @@ -1051,6 +1060,69 @@ public void flushAndClose() throws IOException { | |||
| } | |||
| } | |||
|
|
|||
| protected TranslogRecoveryStats recoverFromTranslog(long translogId, TranslogRecoveryPerformer handler) throws IOException { | |||
There was a problem hiding this comment.
I this should go into the InternalEngine. The shadow engine and a future read only engine don't need this, right?
| } | ||
| final MockDirectoryWrapper directory = DirectoryUtils.getLeaf(store.directory(), MockDirectoryWrapper.class); | ||
| if(directory != null) { | ||
| directory.setPreventDoubleWrite(false); |
There was a problem hiding this comment.
wondering, why do we need this?
There was a problem hiding this comment.
because es is broken
There was a problem hiding this comment.
I thought I replied more here... sorry it buggs me a lot that we have to do that... since we rollback we are writing the same segment files again after starting indexwirter but mockDirWrapper prevents this so we have to disable...I will add a comment
|
I like the change. Left a bunch of comments. Most of them minor, with the exception of the recovery state stats handling. |
|
pushed some updates |
| @@ -620,7 +625,8 @@ protected int sendSnapshot(Translog.Snapshot snapshot) throws ElasticsearchExcep | |||
| if (ops >= recoverySettings.translogOps() || size >= recoverySettings.translogSize().bytes()) { | |||
|
|
|||
| // don't throttle translog, since we lock for phase3 indexing, | |||
| // so we need to move it as fast as possible. Note, since we | |||
| // so we need to move it as fast as possible. Note, | |||
| // ince we | |||
|
@bleskes pushed a new commit |
|
@bleskes I pushed that stuff into the performer as you requested I hope it's worth all the added complexity |
|
LGTM. Thx @s1monw |
s1monw
force-pushed
the
recover_from_translog_in_engine
branch
from
074939c
to
05ae57c
Compare
Today the engine writes the transaction log itself as well as manages all the commit / translog mapping internally. Yet, if an engine is closed and reopend it doesn't replay it's translog or does anything to be consistent with it's latest state again. This change moves the transaction log replay code into the Engine / InternalEngine and adds unittests for replaying and consistency. Closes elastic#10452
Today the engine writes the transaction log itself as well as manages all the commit / translog mapping internally. Yet, if an engine is closed and reopend it doesn't replay it's translog or does anything to be consistent with it's latest state again. This change moves the transaction log replay code into the Engine / InternalEngine and adds unittests for replaying and consistency. Closes elastic#10452
s1monw
force-pushed
the
recover_from_translog_in_engine
branch
from
05ae57c
to
b1c9dfc
Compare
clintongormley
changed the title
clintongormley
added
:Distributed/Recovery
Today the engine writes the transaction log itself as well as manages
all the commit / translog mapping internally. Yet, if an engine is closed
and reopend it doesn't replay it's translog or does anything to be consistent
with it's latest state again.
This change moves the transaction log replay code into the Engine / InternalEngine
and adds unittests for replaying and consistency.