Introduce translog size and age based retention policies #25147

bleskes · 2017-06-09T09:59:50Z

This PR extends the TranslogDeletionPolicy to allow keeping the translog files longer than what is needed for recovery from lucene. Specifically, we allow specifying the total size of the files and their maximum age (i.e., keep up to 512MB but no longer than 3 hours). This will allow making ops based recoveries more common.

To achieve the above the translog files are extended to contain a timestamp, indicating when they were created.

Note that the default size and age still set to 0, maintaining current behavior. This is needed as the other components in the system are not yet ready for a longer translog retention. I will adapt those in follow up PRs.

Relates to #10708

…on_size_age

s1monw

left a couple of comments

s1monw · 2017-06-09T15:02:17Z

core/src/main/java/org/elasticsearch/index/engine/InternalEngine.java

+        final IndexSettings indexSettings = engineConfig.getIndexSettings();
+        translogDeletionPolicy.setMaxRetentionAgeInMillis(indexSettings.getTranslogRetentionAge().getMillis());
+        translogDeletionPolicy.setRetentionSizeInBytes(indexSettings.getTranslogRetentionSize().getBytes());
+        translog.trimUnreferencedReaders();


I don't think we should do this in the settings loop. if these settings changed and the user wants them to take effect they should call _flush

s1monw · 2017-06-09T15:05:23Z

core/src/main/java/org/elasticsearch/index/translog/TranslogDeletionPolicy.java

+        return Math.min(minByAgeAndSize, Math.min(minByView, minTranslogGenerationForRecovery));
+    }
+
+    private long getMinTranslogGenBySize(List<TranslogReader> readers, TranslogWriter writer) {


do you wanna make them pkg private then you can test them in isolation. I'd love to see that

good idea. I took it a step further and made them static, makes things easier.

s1monw · 2017-06-09T15:06:51Z

core/src/main/java/org/elasticsearch/index/translog/TranslogReader.java

                        assert path.getFileName().toString().endsWith(Translog.TRANSLOG_FILE_SUFFIX) : "new file ends with old suffix: " + path;
-                        assert checkpoint.numOps >= 0 : "expected at least 0 operatin but got: " + checkpoint.numOps;
+                        assert checkpoint.numOps >= 0 : "expected at least 0 operation but got: " + checkpoint.numOps;


just an idea, should we use the last modification time on the file instead? I mean it should be ok to do that no?

tja, maybe. I tried to avoid the entire what metadata is supported where discussion. I mean we walk to great length to avoid directory traversals. I think this is simple, solid and entirely self contained?

I am not sure I agree with this. it's redundant information and no traversal needed. you have the file you just need to read the time created. I am really careful with any kind of added metadata here you know why.

I am really careful with any kind of added metadata here you know why

I totally hear you. It's clearly and edge case. One other aspect of they way I set it up now is that it's easy to test. With FS based metadata we'd probably have to mock something, which is also doable but messier.

I lean towards using the filesystem mtime here. I do not think this would be messy to mock, only a method that you can override in tests?

OK. I personally feel it might end up messier but I trust your judgement. I'll roll back the extra inlined creation date and move to using FS mtime. We can always re-add the inline creation date if we run into trouble. Do-The-Minimal-Change-Needed™️

s1monw · 2017-06-09T15:07:17Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

+            initialMinTranslogGen, minTranslogGenerationSupplier, System.currentTimeMillis());
+    }
+
+    static TranslogWriter create(


ugly, can you put multiple on the same line

…on_size_age # Conflicts: # core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

# Conflicts: # core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

s1monw

LGTM

jasontedor

I left a few comments, all minor, I skipped all the parts that had anything to do with ctime given our previous discussion.

jasontedor · 2017-06-15T02:14:20Z

core/src/main/java/org/elasticsearch/index/IndexSettings.java

+            Property.IndexScope);
+
+    public static final Setting<TimeValue> INDEX_TRANSLOG_RETENTION_AGE_SETTING =
+        Setting.timeSetting("index.translog.retention.age", TimeValue.timeValueHours(-1), TimeValue.timeValueMillis(-1), Property.Dynamic,


The default value violates the minimum value? I'm surprised that that's permitted and something doesn't break elsewhere but can you clean this up here?

jasontedor · 2017-06-15T02:14:48Z

core/src/main/java/org/elasticsearch/index/IndexSettings.java

+        Setting.byteSizeSetting("index.translog.retention.size", new ByteSizeValue(-1, ByteSizeUnit.MB), Property.Dynamic,
+            Property.IndexScope);
+
+    public static final Setting<TimeValue> INDEX_TRANSLOG_RETENTION_AGE_SETTING =


Would you please add Javadocs explaining this setting?

jasontedor · 2017-06-15T02:14:51Z

core/src/main/java/org/elasticsearch/index/IndexSettings.java

@@ -111,6 +112,14 @@
        Setting.byteSizeSetting("index.translog.flush_threshold_size", new ByteSizeValue(512, ByteSizeUnit.MB), Property.Dynamic,
            Property.IndexScope);

+    public static final Setting<ByteSizeValue> INDEX_TRANSLOG_RETENTION_SIZE_SETTING =


Would you please add Javadocs explaining this setting?

Between size and age you have an inconsistent ordering, sometimes size then age, sometimes age then size. Can you please pick a consistent ordering for the pairing?

jasontedor · 2017-06-15T02:15:51Z

core/src/main/java/org/elasticsearch/index/IndexSettings.java

@@ -265,6 +276,9 @@ public IndexSettings(final IndexMetaData indexMetaData, final Settings nodeSetti
        syncInterval = INDEX_TRANSLOG_SYNC_INTERVAL_SETTING.get(settings);
        refreshInterval = scopedSettings.get(INDEX_REFRESH_INTERVAL_SETTING);
        flushThresholdSize = scopedSettings.get(INDEX_TRANSLOG_FLUSH_THRESHOLD_SIZE_SETTING);
+        translogRetentionAge = scopedSettings.get(INDEX_TRANSLOG_RETENTION_AGE_SETTING);
+        translogRetentionSize = scopedSettings.get(INDEX_TRANSLOG_RETENTION_SIZE_SETTING);
+        flushThresholdSize = scopedSettings.get(INDEX_TRANSLOG_FLUSH_THRESHOLD_SIZE_SETTING);


This duplicates line 278?

jasontedor · 2017-06-15T02:19:18Z

core/src/main/java/org/elasticsearch/index/translog/TranslogDeletionPolicy.java

+    private long maxRetentionAgeInMillis;
+
+    public TranslogDeletionPolicy(long retentionSizeInBytes, long maxRetentionAgeInMillis) {
+        this.retentionSizeInBytes = retentionSizeInBytes;


Why no max on this?

good question. Will add.

I will actually remove the max from the other one. Shorter.

jasontedor · 2017-06-15T02:24:05Z

core/src/main/java/org/elasticsearch/index/translog/TranslogDeletionPolicy.java

+        long minByAge = getMinTranslogGenByAge(readers, writer, maxRetentionAgeInMillis, currentTime());
+        long minBySize = getMinTranslogGenBySize(readers, writer, retentionSizeInBytes);
+        long minByAgeAndSize = Math.max(minByAge, minBySize);
+        if (minByAgeAndSize == Long.MIN_VALUE) {


It's a matter of taste but I think this is clearer:

diff --git a/core/src/main/java/org/elasticsearch/index/translog/TranslogDeletionPolicy.java b/core/src/main/java/org/elasticsearch/index/translog/TranslogDeletionPolicy.java index 9dd72526ca..216c1f243c 100644 --- a/core/src/main/java/org/elasticsearch/index/translog/TranslogDeletionPolicy.java +++ b/core/src/main/java/org/elasticsearch/index/translog/TranslogDeletionPolicy.java @@ -102,10 +102,12 @@ public class TranslogDeletionPolicy { long minByView = getMinTranslogGenRequiredByViews(); long minByAge = getMinTranslogGenByAge(readers, writer, maxRetentionAgeInMillis, currentTime()); long minBySize = getMinTranslogGenBySize(readers, writer, retentionSizeInBytes); - long minByAgeAndSize = Math.max(minByAge, minBySize); - if (minByAgeAndSize == Long.MIN_VALUE) { + final long minByAgeAndSize; + if (minBySize == Long.MIN_VALUE && minByAge == Long.MAX_VALUE) { // both size and age are disabled; minByAgeAndSize = Long.MAX_VALUE; + } else { + minByAgeAndSize = Math.max(minBySize, minByAge); } return Math.min(minByAgeAndSize, Math.min(minByView, minTranslogGenerationForRecovery)); }

…on_size_age

bleskes · 2017-06-15T09:47:56Z

core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

+            }
+            translog.getDeletionPolicy().setRetentionAgeInMillis(randomBoolean() ? 100 : -1);
+            assertBusy(() -> {
+                try {


I'm going to clean this up but I need another PR

bleskes · 2017-06-15T09:48:28Z

@s1monw @jasontedor can you take another look?

…on_size_age

s1monw

LGTM thanks

jasontedor

For me I think the only question is whether or not we should fsync the metadata when we close a writer into a reader. I do not seeing this being a question that should hold up this PR. I also do not see this being a challenging thing to do, we already sync (without syncing metadata) when we close into a reader, so I think we can merely push a boolean down to change this. Again, separate PR! LGTM.

…on_size_age

bleskes · 2017-06-16T07:10:12Z

Thx @s1monw & @jasontedor for the multiple iterations

…y-context * 'master' of github.com:elastic/elasticsearch: (21 commits) [DOCS] Clarify expected availability of HDFS for the HDFS Repository (elastic#25220) Remove some redundant 140 character checkstyle suppressions [Docs] more fix for the parent-join docs [Docs] Fix cross reference for parent-join field More advices around search speed and disk usage. (elastic#25252) Add documentation for the new parent-join field (elastic#25227) [analysis-icu] Allow setting unicodeSetFilter (elastic#20814) Introduce translog size and age based retention policies (elastic#25147) Add needs methods for specific variables to Painless script context factories. (elastic#25267) Improves snapshot logging and snapshoth deletion error handling (elastic#25264) Add unit test for PathHierarchyTokenizerFactory (elastic#24984) Deprecate tribe service Moved more token filters to analysis-common module. [Test] Make sure that SearchAfterSortedDocQueryTests uses a single threaded searcher [DOCS] Defined es-test-dir and plugins-examples-dir in index.asciidoc. (elastic#25232) Test fix - removed superfluous assertion (elastic#25247) [Test] restore BWC for parent-join now that the new mapping format is in 5.x Add a section named "relations" in the ParentJoinFieldMapper (elastic#25248) test: Ported more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests. (elastic#25173) fix: Sort Processor does not have proper behavior with targetField (elastic#25237) ...

#25147 added the translog deletion policy but didn't enable it by default. This PR enables a default retention of 512MB (same maximum size of the current translog) and an age of 12 hours (i.e., after 12 hours all translog files will be deleted). This increases to chance to have an ops based recovery, even if the primary flushed or the replica was offline for a few hours. In order to see which parts of the translog are committed into lucene the translog stats are extended to include information about uncommitted operations. Views now include all translog ops and guarantee, as before, that those will not go away. Snapshotting a view allows to filter out generations that are not relevant based on a specific sequence number. Relates to #10708

bleskes added 2 commits June 9, 2017 10:43

retention settings

3e32b7a

Merge remote-tracking branch 'upstream/master' into translog_rententi…

5f01ce6

…on_size_age

bleskes added :Sequence IDs v6.0.0 labels Jun 9, 2017

bleskes requested review from s1monw and jasontedor June 9, 2017 09:59

s1monw requested changes Jun 9, 2017

View reviewed changes

bleskes added 6 commits June 9, 2017 18:48

Merge remote-tracking branch 'upstream/master' into translog_rententi…

7da706c

…on_size_age # Conflicts: # core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

feedback

da56a0d

better indication of disabling size checks

9d0cf47

fix stats

a520407

clearer disable value

659fb6c

Merge branch 'master' into translog_rentention_size_age

08a2b7a

# Conflicts: # core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

bleskes requested a review from s1monw June 11, 2017 21:23

clintongormley added the >feature label Jun 12, 2017

s1monw approved these changes Jun 14, 2017

View reviewed changes

jasontedor reviewed Jun 15, 2017

View reviewed changes

bleskes added 5 commits June 15, 2017 08:14

Merge remote-tracking branch 'upstream/master' into translog_rententi…

88d7a32

…on_size_age

moved to relying on the FS mtime for translog age

3b62254

feedback

1a638cf

line formatting

f1dd56f

add some cleaning testing code that doesn't mock

a3c4f2f

bleskes commented Jun 15, 2017

View reviewed changes

bleskes requested review from s1monw and jasontedor June 15, 2017 09:48

bleskes added 3 commits June 15, 2017 13:25

Merge remote-tracking branch 'upstream/master' into translog_rententi…

384b795

…on_size_age

remove unneeded runtime exception

5d97182

Merge remote-tracking branch 'upstream/master' into translog_rententi…

10ac31a

…on_size_age

s1monw approved these changes Jun 15, 2017

View reviewed changes

jasontedor approved these changes Jun 15, 2017

View reviewed changes

Merge remote-tracking branch 'upstream/master' into translog_rententi…

923152d

…on_size_age

bleskes merged commit 9ddea53 into elastic:master Jun 16, 2017

bleskes deleted the translog_rentention_size_age branch June 16, 2017 07:09

bleskes mentioned this pull request Jun 19, 2017

Enable a long translog retention policy by default #25294

Merged

clintongormley added v6.0.0-beta1 and removed v6.0.0 labels Jul 25, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce translog size and age based retention policies #25147

Introduce translog size and age based retention policies #25147

bleskes commented Jun 9, 2017

s1monw left a comment

s1monw Jun 9, 2017

bleskes Jun 9, 2017

s1monw Jun 9, 2017

bleskes Jun 9, 2017

s1monw Jun 9, 2017

bleskes Jun 9, 2017

s1monw Jun 14, 2017

bleskes Jun 14, 2017

jasontedor Jun 14, 2017

bleskes Jun 14, 2017

s1monw Jun 9, 2017

s1monw left a comment

jasontedor left a comment

jasontedor Jun 15, 2017

bleskes Jun 15, 2017

jasontedor Jun 15, 2017

jasontedor Jun 15, 2017

jasontedor Jun 15, 2017

jasontedor Jun 15, 2017

bleskes Jun 15, 2017

jasontedor Jun 15, 2017

bleskes Jun 15, 2017

bleskes Jun 15, 2017

jasontedor Jun 15, 2017

bleskes Jun 15, 2017

bleskes Jun 15, 2017

bleskes commented Jun 15, 2017

s1monw left a comment

jasontedor left a comment

bleskes commented Jun 16, 2017

Introduce translog size and age based retention policies #25147

Introduce translog size and age based retention policies #25147

Conversation

bleskes commented Jun 9, 2017

s1monw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw left a comment

Choose a reason for hiding this comment

jasontedor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes commented Jun 15, 2017

s1monw left a comment

Choose a reason for hiding this comment

jasontedor left a comment

Choose a reason for hiding this comment

bleskes commented Jun 16, 2017