Introduce soft-deletes retention policy based on global checkpoint #30335

dnhatn · 2018-05-02T13:21:41Z

This commit introduces a soft-deletes retention merge policy based on
the global checkpoint. Some notes on this simple retention policy:

This policy keeps all operations whose seq# is greater than the
persisted global checkpoint and configurable extra operations prior to
the global checkpoint. This is good enough for querying history changes.
This policy is not watertight for peer-recovery. I will make it in a
follow-up.
This policy is too simple to support rollback.

Relates #29530

This commit introduces a soft-deletes retention merge policy based on the global checkpoint. Some notes on this simple retention policy: - This policy keeps all operations whose seq# is greater than the persisted global checkpoint and configurable extra operations prior to the global checkpoint. This is good enough for querying history changes. - This policy is not watertight for peer-recovery. I will make it in a follow-up. - This policy is too simple to support rollback.

dnhatn · 2018-05-02T13:23:11Z

/cc @jasontedor and @martijnvg

bleskes · 2018-05-02T13:36:28Z

Thanks Nhat. I will look as soon as I can.

This policy is not watertight for peer-recovery. I will make it in a
follow-up.
This policy is too simple to support rollback.

Can you elaborate on these?

dnhatn · 2018-05-02T13:48:20Z

This policy is not watertight for peer-recovery. I will make it in a follow-up.

We send the safe-commit in peer-recovery, thus we need to also send all operations after the local checkpoint of that commit. This is analog to the min translog gen for recovery.

This policy is too simple to support rollback.

This policy considers multiple operations of a single document individually instead of a chain. It might not work in this case.

Index doc1 then flush
Index a large num of docs then flush (i.e. greater than the retained ops)
Delete doc1 then flush
At step 3, this retention policy may not keep the actual doc1 but only the tombstone, thus we may not be able to un-delete doc1.

elasticmachine · 2018-05-03T02:24:26Z

Pinging @elastic/es-distributed

s1monw

this LGTM but I think we should wait until https://issues.apache.org/jira/browse/LUCENE-8290 is in and we have a new snapshot. Otherwise the retention policy is broken :(

bleskes

LGTM. Thanks for clarifying your first remarks. Makes sense.

bleskes · 2018-05-03T08:56:45Z

server/src/main/java/org/elasticsearch/index/engine/InternalEngine.java

@@ -2016,6 +2019,17 @@ private IndexWriterConfig getIndexWriterConfig() {
        return iwc;
    }

+    private Query softDeletesRetentionQuery() {


nit: java docs please - something like "documents that are soft deleted and match this query should be retained and not cleaned up by merges"

bleskes · 2018-05-03T09:04:14Z

server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java

+                engine.index(indexForDoc(doc));
+                liveDocs.add(doc.id());
+            }
+            for (int i = 0; i < numDocs; i++) {


question - do we also want to override documents, rather than just deleting them?

question - do we also want to override documents, rather than just deleting them?

+1. I added updates.

s1monw · 2018-05-03T12:04:11Z

@dnhatn https://issues.apache.org/jira/browse/LUCENE-8290 is in, you can pull a new snap if you want.

s1monw · 2018-05-03T12:08:51Z

also @dnhatn you might wanna follow this #30357

dnhatn · 2018-05-04T20:43:53Z

CI failed because of WatchMetadataTests.
@elasticmachine test this please.

dnhatn · 2018-05-04T23:46:03Z

@elasticmachine test this please

dnhatn · 2018-05-04T23:49:14Z

Thanks @simonw and @bleskes for the review.

* origin/ccr: (166 commits) Introduce soft-deletes retention policy based on global checkpoint (elastic#30335) Enable MockHttpTransport in ShardChangsIT Remove old sha files from dated Lucene snapshot Update InternalEngine tests on ccr side for elastic#30121 Set the new lucene version for 6.4.0 [ML][TEST] Clean up jobs in ModelPlotIT Upgrade to 7.4.0-snapshot-1ed95c097b (elastic#30357) Watcher: Ensure trigger service pauses execution (elastic#30363) [DOCS] Added coming qualifiers in changelog [DOCS] Commented out empty sections in the changelog to fix the doc build. (elastic#30372) Security: reduce garbage during index resolution (elastic#30180) Make RepositoriesMetaData contents unmodifiable (elastic#30361) Change quad tree max levels to 29. Closes elastic#21191 (elastic#29663) Test: use trial license in qa tests with security [ML] Add integration test for model plots (elastic#30359) SQL: Fix bug caused by empty composites (elastic#30343) [ML] Account for gaps in data counts after job is reopened (elastic#30294) InternalEngineTests.testConcurrentOutOfOrderDocsOnReplica should use two documents (elastic#30121) Change signature of Get Repositories Response (elastic#30333) Tests: Use different watch ids per test in smoke test (elastic#30331) ...

* origin/ccr: (127 commits) Introduce soft-deletes retention policy based on global checkpoint (elastic#30335) Enable MockHttpTransport in ShardChangsIT Remove old sha files from dated Lucene snapshot Update InternalEngine tests on ccr side for elastic#30121 Set the new lucene version for 6.4.0 [ML][TEST] Clean up jobs in ModelPlotIT Upgrade to 7.4.0-snapshot-1ed95c097b (elastic#30357) Watcher: Ensure trigger service pauses execution (elastic#30363) [DOCS] Added coming qualifiers in changelog [DOCS] Commented out empty sections in the changelog to fix the doc build. (elastic#30372) Security: reduce garbage during index resolution (elastic#30180) Make RepositoriesMetaData contents unmodifiable (elastic#30361) Change quad tree max levels to 29. Closes elastic#21191 (elastic#29663) Test: use trial license in qa tests with security [ML] Add integration test for model plots (elastic#30359) SQL: Fix bug caused by empty composites (elastic#30343) [ML] Account for gaps in data counts after job is reopened (elastic#30294) InternalEngineTests.testConcurrentOutOfOrderDocsOnReplica should use two documents (elastic#30121) Change signature of Get Repositories Response (elastic#30333) Tests: Use different watch ids per test in smoke test (elastic#30331) ...

* origin/ccr: (166 commits) Introduce soft-deletes retention policy based on global checkpoint (elastic#30335) Enable MockHttpTransport in ShardChangsIT Remove old sha files from dated Lucene snapshot Update InternalEngine tests on ccr side for elastic#30121 Set the new lucene version for 6.4.0 [ML][TEST] Clean up jobs in ModelPlotIT Upgrade to 7.4.0-snapshot-1ed95c097b (elastic#30357) Watcher: Ensure trigger service pauses execution (elastic#30363) [DOCS] Added coming qualifiers in changelog [DOCS] Commented out empty sections in the changelog to fix the doc build. (elastic#30372) Security: reduce garbage during index resolution (elastic#30180) Make RepositoriesMetaData contents unmodifiable (elastic#30361) Change quad tree max levels to 29. Closes elastic#21191 (elastic#29663) Test: use trial license in qa tests with security [ML] Add integration test for model plots (elastic#30359) SQL: Fix bug caused by empty composites (elastic#30343) [ML] Account for gaps in data counts after job is reopened (elastic#30294) InternalEngineTests.testConcurrentOutOfOrderDocsOnReplica should use two documents (elastic#30121) Change signature of Get Repositories Response (elastic#30333) Tests: Use different watch ids per test in smoke test (elastic#30331) ...

…30335) This commit introduces a soft-deletes retention merge policy based on the global checkpoint. Some notes on this simple retention policy: - This policy keeps all operations whose seq# is greater than the persisted global checkpoint and configurable extra operations prior to the global checkpoint. This is good enough for querying history changes. - This policy is not watertight for peer-recovery. We send the safe-commit in peer-recovery, thus we need to also send all operations after the local checkpoint of that commit. This is analog to the min translog generation for recovery. - This policy is too simple to support rollback. Relates #29530

This PR integrates Lucene soft-deletes (LUCENE-8200) into Elasticsearch. Highlight works in this PR include: 1. Replace hard-deletes by soft-deletes in InternalEngine 2. Use _recovery_source if _source is disabled or modified (elastic#31106) 3. Soft-deletes retention policy based on the global checkpoint (elastic#30335) 4. Read operation history from Lucene instead of translog (elastic#30120) 5. Use Lucene history in peer-recovery (elastic#30522) These works have been done by the whole team; however, these individuals (lexical order) have significant contribution in coding and reviewing: Co-authored-by: Adrien Grand <jpountz@gmail.com> Co-authored-by: Boaz Leskes <b.leskes@gmail.com> Co-authored-by: Jason Tedor <jason@tedor.me> Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Simon Willnauer <simonw@apache.org>

This PR integrates Lucene soft-deletes(LUCENE-8200) into Elasticsearch. Highlight works in this PR include: - Replace hard-deletes by soft-deletes in InternalEngine - Use _recovery_source if _source is disabled or modified (#31106) - Soft-deletes retention policy based on the global checkpoint (#30335) - Read operation history from Lucene instead of translog (#30120) - Use Lucene history in peer-recovery (#30522) Relates #30086 Closes #29530 --- These works have been done by the whole team; however, these individuals (lexical order) have significant contribution in coding and reviewing: Co-authored-by: Adrien Grand jpountz@gmail.com Co-authored-by: Boaz Leskes b.leskes@gmail.com Co-authored-by: Jason Tedor jason@tedor.me Co-authored-by: Martijn van Groningen martijn.v.groningen@gmail.com Co-authored-by: Nhat Nguyen nhat.nguyen@elastic.co Co-authored-by: Simon Willnauer simonw@apache.org

This PR integrates Lucene soft-deletes(LUCENE-8200) into Elasticsearch. Highlight works in this PR include: - Replace hard-deletes by soft-deletes in InternalEngine - Use _recovery_source if _source is disabled or modified (elastic#31106) - Soft-deletes retention policy based on the global checkpoint (elastic#30335) - Read operation history from Lucene instead of translog (elastic#30120) - Use Lucene history in peer-recovery (elastic#30522) Relates elastic#30086 Closes elastic#29530 --- These works have been done by the whole team; however, these individuals (lexical order) have significant contribution in coding and reviewing: Co-authored-by: Adrien Grand <jpountz@gmail.com> Co-authored-by: Boaz Leskes <b.leskes@gmail.com> Co-authored-by: Jason Tedor <jason@tedor.me> Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Simon Willnauer <simonw@apache.org>

This PR integrates Lucene soft-deletes(LUCENE-8200) into Elasticsearch. Highlight works in this PR include: - Replace hard-deletes by soft-deletes in InternalEngine - Use _recovery_source if _source is disabled or modified (#31106) - Soft-deletes retention policy based on the global checkpoint (#30335) - Read operation history from Lucene instead of translog (#30120) - Use Lucene history in peer-recovery (#30522) Relates #30086 Closes #29530 --- These works have been done by the whole team; however, these individuals (lexical order) have significant contribution in coding and reviewing: Co-authored-by: Adrien Grand <jpountz@gmail.com> Co-authored-by: Boaz Leskes <b.leskes@gmail.com> Co-authored-by: Jason Tedor <jason@tedor.me> Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Simon Willnauer <simonw@apache.org>

dnhatn added the :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. label May 2, 2018

dnhatn requested review from s1monw and bleskes May 2, 2018 13:21

hub-cap added :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. and removed :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. labels May 3, 2018

s1monw approved these changes May 3, 2018

View reviewed changes

colings86 added the >feature label May 3, 2018

bleskes approved these changes May 3, 2018

View reviewed changes

dnhatn added 3 commits May 4, 2018 10:50

Boaz feedbacks

6313164

Merge branch 'ccr' into soft-deletes-mergepolicy

afe5445

Merge branch 'ccr' into soft-deletes-mergepolicy

bca0b8e

dnhatn merged commit 2c73969 into elastic:ccr May 5, 2018

dnhatn deleted the soft-deletes-mergepolicy branch May 5, 2018 03:19

dnhatn added the backport pending label May 5, 2018

dnhatn mentioned this pull request May 5, 2018

Use soft-deletes to maintain document history #29530

Closed

14 tasks

dnhatn removed the backport pending label May 10, 2018

dnhatn mentioned this pull request Aug 29, 2018

Integrates soft-deletes into Elasticsearch #33222

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce soft-deletes retention policy based on global checkpoint #30335

Introduce soft-deletes retention policy based on global checkpoint #30335

dnhatn commented May 2, 2018 •

edited

dnhatn commented May 2, 2018

bleskes commented May 2, 2018

dnhatn commented May 2, 2018 •

edited

elasticmachine commented May 3, 2018

s1monw left a comment

bleskes left a comment

bleskes May 3, 2018

bleskes May 3, 2018

dnhatn May 4, 2018

s1monw commented May 3, 2018

s1monw commented May 3, 2018

dnhatn commented May 4, 2018 •

edited

dnhatn commented May 4, 2018

dnhatn commented May 4, 2018

Introduce soft-deletes retention policy based on global checkpoint #30335

Introduce soft-deletes retention policy based on global checkpoint #30335

Conversation

dnhatn commented May 2, 2018 • edited

dnhatn commented May 2, 2018

bleskes commented May 2, 2018

dnhatn commented May 2, 2018 • edited

elasticmachine commented May 3, 2018

s1monw left a comment

Choose a reason for hiding this comment

bleskes left a comment

Choose a reason for hiding this comment

bleskes May 3, 2018

Choose a reason for hiding this comment

bleskes May 3, 2018

Choose a reason for hiding this comment

dnhatn May 4, 2018

Choose a reason for hiding this comment

s1monw commented May 3, 2018

s1monw commented May 3, 2018

dnhatn commented May 4, 2018 • edited

dnhatn commented May 4, 2018

dnhatn commented May 4, 2018

dnhatn commented May 2, 2018 •

edited

dnhatn commented May 2, 2018 •

edited

dnhatn commented May 4, 2018 •

edited