Add support for time travel with version in Delta Lake #21052

ebyhr · 2024-03-13T13:17:21Z

Description

Support version as time travel in Delta
Fixes #15894

Add test for v2 checkpoint once Add support for v2Checkpoint in Delta Lake #19507 lands
Add test for CREATE OR REPLACE statement once Implement CREATE OR REPLACE TABLE for delta lake connector #19991 lands
optional: Trace operations for listing files

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake
* Add support for time travel with versions. ({issue}`issuenumber`)

jkylling

Exciting changes! I have some concerns about the performance of finding the last checkpoint. For object storage and large tables, single threaded listing of everything within /_delta_log can be very time consuming. I threw together a prototype for time travel using the "list-after" feature of object stores a while ago. It works by starting at the end version, listing the first batch of files after endVersion-500 and less than end version, if there is a checkpoint stop, if not, pick the next batch at endVersion-1000, and so on. The prototype is here: https://github.com/trinodb/trino/compare/master...jkylling:trino:basic-time-travel-delta?expand=1. This performs well on tables with large Delta logs, but require modifying the file system interfaces.

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

findinpath

Just skimmed.
We should focus on TransactionLogAccess to work with only the necessary amount of read operations.

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeBasic.java

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java

...sts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeTimeTravelCompatibility.java

...in/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeFileOperations.java

findinpath · 2024-03-13T22:22:35Z

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

@@ -191,6 +205,72 @@ public TableSnapshot loadSnapshot(ConnectorSession session, SchemaTableName tabl
        return snapshot;
    }

+    private TableSnapshot loadSnapshotForTimeTravel(TrinoFileSystem fileSystem, SchemaTableName table, String tableLocation, long endVersion)


We should actually get the endVersion first, then the previous version , ..., until we find the checkpoint .

I'm thinking about something similar to io.trino.plugin.deltalake.transactionlog.checkpoint.TransactionLogTail#loadNewTail but backwards.

Doing this would get us to retrieve exactly what we need and not do any unnecessary file listings in the transaction log directory.

24.log 23.log 22.log 21.log 20.log 20.checkpoint

The suggested logic works for classic checkpoint, but it might be difficult in case of multipart checkpoints and UUID-named checkpoints. We can't estimate the file name in my understanding. What do you think?

Ye, you are right.
It was a naive thought.
We don't know for which key to look for :(

It is worth exploring the startAfter listing idea for s3

https://github.com/delta-io/delta/blob/fe3c3c05bf2656ab45b0a4e66fc010df13f5ed92/spark/src/main/scala/org/apache/spark/sql/delta/Checkpoints.scala#L426-L434

Actually nevermind about startAfter - let's not overengineer yet this.

Probably, we can use different logic based on delta.checkpointPolicy table property in the future.
https://github.com/delta-io/delta/blob/537ed8ee0be983579873851855ba3e96b20004bd/docs/source/table-properties.md?plain=1#L159-L170

findinpath · 2024-03-15T14:26:15Z

Pls rebase on master to address the conflicts.

findinpath

The PR is ready to be reviewed (definitely not anymore a draft) from my perspective.

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakePropertiesTable.java

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java

findinpath · 2024-03-17T21:55:26Z

...in/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeFileOperations.java

+                        .add(new FileOperation(TRANSACTION_LOG_JSON, "00000000000000000008.json", "InputFile.exists"))
+                        .addCopies(new FileOperation(DATA, "no partition", "InputFile.newInput"), 8)
+                        .build());
+        assertFileSystemAccesses("SELECT * FROM " + tableName + " FOR VERSION AS OF 13",


Here would be nice to see that there is though a list operation performed to figure out the checkpoint

docs/src/main/sphinx/connector/delta-lake.md

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

...ta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAlluxioCacheFileOperations.java

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

findinpath · 2024-03-23T05:48:04Z

Pls rebase (and squash) to resolve the conflicts.

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java

docs/src/main/sphinx/connector/delta-lake.md

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

krvikash

LGTM. some comments.

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java

...in/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeFileOperations.java

...ta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAlluxioCacheFileOperations.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

findinpath

LGTM % nits.

The one relevant comment is about versioned table check in the reconciliation logic.

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java

ebyhr · 2024-04-03T06:18:28Z

Addressed comments.

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestTransactionLogAccess.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

...lta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeLocalConcurrentWritesTest.java

docs/src/main/sphinx/connector/delta-lake.md

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeTableHandle.java

...delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/DropExtendedStatsProcedure.java

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

ebyhr · 2024-04-08T11:44:52Z

@wendigo Thanks for your review. Addressed comments.

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

wendigo · 2024-04-09T09:45:03Z

LGTM % single nit. Please squash fixups before merging @ebyhr

Co-authored-by: Vikash Kumar <vksh1612@gmail.com>

ebyhr self-assigned this Mar 13, 2024

cla-bot bot added the cla-signed label Mar 13, 2024

github-actions bot added docs tests:hive delta-lake Delta Lake connector labels Mar 13, 2024

findinpath requested a review from pajaks March 13, 2024 19:08

jkylling reviewed Mar 13, 2024

View reviewed changes

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java Show resolved Hide resolved

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java Show resolved Hide resolved

findinpath reviewed Mar 13, 2024

View reviewed changes

ebyhr force-pushed the ebi/delta-timetravel branch from b345429 to 99b46c1 Compare March 15, 2024 08:10

ebyhr force-pushed the ebi/delta-timetravel branch 2 times, most recently from d16def0 to 1f8c458 Compare March 16, 2024 00:25

findinpath reviewed Mar 17, 2024

View reviewed changes

pajaks reviewed Mar 19, 2024

View reviewed changes

ebyhr force-pushed the ebi/delta-timetravel branch 2 times, most recently from 4c0711a to 8a1bdd1 Compare March 22, 2024 06:17

ebyhr marked this pull request as ready for review March 22, 2024 09:24

pajaks reviewed Mar 22, 2024

View reviewed changes

findinpath reviewed Mar 23, 2024

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java Outdated Show resolved Hide resolved

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java Outdated Show resolved Hide resolved

ebyhr force-pushed the ebi/delta-timetravel branch 2 times, most recently from 9bc6a46 to 2ba73c9 Compare March 25, 2024 08:14

pajaks reviewed Mar 25, 2024

View reviewed changes

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java Show resolved Hide resolved

findinpath reviewed Mar 25, 2024

View reviewed changes

docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved

findinpath reviewed Mar 25, 2024

View reviewed changes

...-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java Show resolved Hide resolved

findinpath reviewed Mar 25, 2024

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java Outdated Show resolved Hide resolved

findepi mentioned this pull request Mar 25, 2024

Trino FS Write API #20869

Closed

ebyhr force-pushed the ebi/delta-timetravel branch from 2ba73c9 to 7238857 Compare March 25, 2024 23:38

ebyhr requested a review from Praveen2112 March 25, 2024 23:38

pajaks approved these changes Mar 26, 2024

View reviewed changes

krvikash approved these changes Mar 29, 2024

View reviewed changes

ebyhr force-pushed the ebi/delta-timetravel branch from 72f2705 to a4bbba5 Compare March 31, 2024 23:31

wendigo reviewed Apr 2, 2024

View reviewed changes

findinpath mentioned this pull request Apr 2, 2024

Fix potentially incorrect table read version for writes on Delta Lake #21330

Merged

findinpath reviewed Apr 2, 2024

View reviewed changes

ebyhr mentioned this pull request Apr 3, 2024

Improve speed when listing transaction logs during time travel in Delta Lake #21366

Open

ebyhr force-pushed the ebi/delta-timetravel branch from a4bbba5 to dfa774e Compare April 3, 2024 05:02

jkylling reviewed Apr 3, 2024

View reviewed changes

ebyhr force-pushed the ebi/delta-timetravel branch from dfa774e to 7c1f565 Compare April 3, 2024 10:46

findinpath reviewed Apr 3, 2024

View reviewed changes

...lta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeLocalConcurrentWritesTest.java Outdated Show resolved Hide resolved

ebyhr force-pushed the ebi/delta-timetravel branch from 7c1f565 to 4928540 Compare April 7, 2024 23:03

findinpath reviewed Apr 8, 2024

View reviewed changes

...lta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeLocalConcurrentWritesTest.java Outdated Show resolved Hide resolved

Fix typo

9ff5cc2

ebyhr force-pushed the ebi/delta-timetravel branch from 4928540 to 5a06a1b Compare April 8, 2024 08:48

wendigo reviewed Apr 8, 2024

View reviewed changes

docs/src/main/sphinx/connector/delta-lake.md Outdated Show resolved Hide resolved

wendigo reviewed Apr 8, 2024

View reviewed changes

ebyhr force-pushed the ebi/delta-timetravel branch from 5a06a1b to 95bfba7 Compare April 8, 2024 11:44

ebyhr requested a review from wendigo April 9, 2024 09:35

wendigo reviewed Apr 9, 2024

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java Show resolved Hide resolved

wendigo approved these changes Apr 9, 2024

View reviewed changes

Add support for time travel with version in Delta Lake

837f174

Co-authored-by: Vikash Kumar <vksh1612@gmail.com>

ebyhr force-pushed the ebi/delta-timetravel branch from 95bfba7 to 837f174 Compare April 9, 2024 10:07

ebyhr merged commit 760e55d into trinodb:master Apr 9, 2024
4 of 15 checks passed

ebyhr deleted the ebi/delta-timetravel branch April 9, 2024 10:08

ebyhr mentioned this pull request Apr 9, 2024

Implement CREATE OR REPLACE TABLE for delta lake connector #19991

Merged

github-actions bot added this to the 445 milestone Apr 9, 2024

colebow mentioned this pull request Apr 9, 2024

Add Trino 445 release notes #21472

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for time travel with version in Delta Lake #21052

Add support for time travel with version in Delta Lake #21052

ebyhr commented Mar 13, 2024 •

edited

jkylling left a comment

findinpath left a comment

findinpath Mar 13, 2024 •

edited

ebyhr Mar 15, 2024

findinpath Mar 15, 2024 •

edited

findinpath Mar 18, 2024

ebyhr Apr 2, 2024

findinpath commented Mar 15, 2024

findinpath left a comment

findinpath Mar 17, 2024

findinpath commented Mar 23, 2024

krvikash left a comment

findinpath left a comment

ebyhr commented Apr 3, 2024

ebyhr commented Apr 8, 2024

wendigo commented Apr 9, 2024

Add support for time travel with version in Delta Lake #21052

Add support for time travel with version in Delta Lake #21052

Conversation

ebyhr commented Mar 13, 2024 • edited

Description

Release notes

jkylling left a comment

Choose a reason for hiding this comment

findinpath left a comment

Choose a reason for hiding this comment

findinpath Mar 13, 2024 • edited

Choose a reason for hiding this comment

ebyhr Mar 15, 2024

Choose a reason for hiding this comment

findinpath Mar 15, 2024 • edited

Choose a reason for hiding this comment

findinpath Mar 18, 2024

Choose a reason for hiding this comment

ebyhr Apr 2, 2024

Choose a reason for hiding this comment

findinpath commented Mar 15, 2024

findinpath left a comment

Choose a reason for hiding this comment

findinpath Mar 17, 2024

Choose a reason for hiding this comment

findinpath commented Mar 23, 2024

krvikash left a comment

Choose a reason for hiding this comment

findinpath left a comment

Choose a reason for hiding this comment

ebyhr commented Apr 3, 2024

ebyhr commented Apr 8, 2024

wendigo commented Apr 9, 2024

ebyhr commented Mar 13, 2024 •

edited

findinpath Mar 13, 2024 •

edited

findinpath Mar 15, 2024 •

edited