Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Delta Lake connector #10897

Merged
merged 2 commits into from
Mar 8, 2022
Merged

Add Delta Lake connector #10897

merged 2 commits into from
Mar 8, 2022

Conversation

jirassimok
Copy link
Member

@jirassimok jirassimok commented Feb 1, 2022 •
edited by findepi
Loading

This connector was originally written for the Starburst Enterprise platform and is now being contributed to Trino.

For #11296

@cla-bot cla-bot bot added the cla-signed label Feb 1, 2022
@jirassimok jirassimok force-pushed the open-starburst-delta-lake branch 2 times, most recently from c47598f to f1954e6 Compare February 1, 2022 21:46
@mosabua
Copy link
Member

mosabua commented Feb 2, 2022

💯

@dennyglee dennyglee mentioned this pull request Feb 3, 2022
Closed
@sajjoseph sajjoseph mentioned this pull request Feb 16, 2022
Closed
@sajjoseph
Copy link
Contributor

@jirassimok - thanks for the contribution. Are there any features that are only available in Starburst version of this connector? (Z ordering, delta write capability etc - are they supported in the version here)?

I know that Databricks team created their version of Trino delta connector (#10300). Is there any plan to combine the efforts?

@findepi
Copy link
Member

findepi commented Feb 16, 2022

@sajjoseph thanks for your comments

Are there any features that are only available in Starburst version of this connector? (Z ordering, delta write capability etc - are they supported in the version here)?

writes are supported here
z-order support is TBD and is a bigger effort that we may work on in the future

I know that Databricks team created their version of Trino delta connector (#10300). Is there any plan to combine the efforts?

we're aware and we've been discussing the best way to move forward. cc @vkorukanti

@mosabua
Copy link
Member

mosabua commented Feb 16, 2022

@jirassimok and @findepi .. do you intend to add the connector docs here or should we start a parallel docs PR and work with you on that?

@jirassimok
Copy link
Member Author

I don't plan to work on the docs for a while yet, so it might be easier to keep the PRs separate.

@jirassimok jirassimok force-pushed the open-starburst-delta-lake branch 3 times, most recently from 92954b7 to d5db955 Compare February 23, 2022 14:45
@findepi
Copy link
Member

findepi commented Feb 23, 2022

cc @vkorukanti

@jirassimok jirassimok force-pushed the open-starburst-delta-lake branch 4 times, most recently from 93d8fd5 to fbd36a2 Compare February 24, 2022 15:05
@findepi
Copy link
Member

findepi commented Feb 24, 2022

@nineinchnick can you PTAL the CI changes?

@findepi findepi mentioned this pull request Feb 24, 2022
Closed
@jirassimok jirassimok force-pushed the open-starburst-delta-lake branch 5 times, most recently from 8860515 to 9061fd5 Compare February 25, 2022 17:03
@findepi findepi marked this pull request as ready for review February 28, 2022 09:43
@findepi
Copy link
Member

findepi commented Mar 3, 2022

rebased because master version changerd

findepi
findepi reviewed Mar 3, 2022
View reviewed changes
plugin/trino-delta-lake/pom.xml Show resolved Hide resolved
...trino-delta-lake/src/main/java/io/trino/plugin/deltalake/statistics/DeltaLakeStatistics.java Show resolved Hide resolved
...delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/DropExtendedStatsProcedure.java Show resolved Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java Show resolved Hide resolved
...ake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointBuilder.java Show resolved Hide resolved
...ake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointBuilder.java Show resolved Hide resolved
...lake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointWriter.java Show resolved Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConfig.java Show resolved Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConnectorFactory.java Show resolved Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeErrorCode.java Outdated Show resolved Hide resolved
This was referenced Mar 3, 2022
Open
Closed
@findepi findepi mentioned this pull request Mar 3, 2022
Closed
findepi
findepi requested changes Mar 3, 2022
View reviewed changes
Copy link
Member

@findepi findepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking red so that i don't forget about the comment in the ci.yml

# TODO: Restore condition above: && env.ABFS_ACCOUNT != '' && env.ABFS_CONTAINER != '' && env.ABFS_ACCESS_KEY != ''

@findepi
Copy link
Member

findepi commented Mar 3, 2022

rebased to resolve logical conflict because of #11286

@findepi
Copy link
Member

findepi commented Mar 4, 2022

(rebased because #11312 merged introducing a logical merge conflict)

@findepi findepi force-pushed the open-starburst-delta-lake branch 3 times, most recently from f9958f6 to 7d7ae75 Compare March 7, 2022 19:50
@jirassimok @alexjo2144
@findepi @losipiuk @aalbu @wendigo @raunaqmorarka @hashhar @MiguelWeezardo @Justin-Kwan @kokosing @homar @ksobolew @findinpath @nineinchnick @ebyhr @Praveen2112 @sopel39 @electrum @ssheikin @charlesjmorgan @martint @jvanzyl @dain
Add Delta Lake connector
2aff0d7 
This connecter was originally written for the Starburst Enterprise
platform.

Co-authored by various engineers at Starburst Data:

Co-authored-by: Alex Jo <alex.jo@starburstdata.com>
Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com>
Co-authored-by: Łukasz Osipiuk <lukasz@osipiuk.net>
Co-authored-by: Alex Albu <alex.albu@starburstdata.com>
Co-authored-by: Mateusz Gajewski <mateusz.gajewski@gmail.com>
Co-authored-by: Raunaq Morarka <raunaqmorarka@gmail.com>
Co-authored-by: Ashhar Hasan <ashhar.hasan@starburstdata.com>
Co-authored-by: Michał Ślizak <michal.slizak+github@gmail.com>
Co-authored-by: Justin-Kwan <justin.kwan@starburstdata.com>
Co-authored-by: Grzegorz Kokosiński <grzegorz@starburstdata.com>
Co-authored-by: Arkadiusz Czajkowski <arek@starburstdata.com>
Co-authored-by: Konrad Dziedzic <konraddziedzic@gmail.com>
Co-authored-by: Jacob I. Komissar <jacob.komissar@starburstdata.com>
Co-authored-by: Krzysztof Sobolewski <krzysztof.sobolewski@starburstdata.com>
Co-authored-by: Marius Grama <findinpath@gmail.com>
Co-authored-by: Irek Matysiewicz <ireneusz.matysiewicz@starburstdata.com>
Co-authored-by: Krzysztof Skrzypczynski <krzysztof.skrzypczynski@starburstdata.com>
Co-authored-by: Jan Was <jan.was@starburstdata.com>
Co-authored-by: Yuya Ebihara <yuya.ebihara@starburstdata.com>
Co-authored-by: praveenkrishna <praveenkrishna@tutanota.com>
Co-authored-by: Karol Sobczak <napewnotrafi@gmail.com>
Co-authored-by: David Phillips <david@acz.org>
Co-authored-by: Sasha Sheikin <myminitrue@gmail.com>
Co-authored-by: Charles Morgan <cjmorgan2000@gmail.com>
Co-authored-by: Martin Traverso <mtraverso@gmail.com>
Co-authored-by: Jason van Zyl <jason@vanzyl.ca>
Co-authored-by: Szymon Homa <szymon.homa@starburstdata.com>
Co-authored-by: Dain Sundstrom <dain@iq80.com>
@findepi findepi merged commit 9956543 into trinodb:master Mar 8, 2022
@findepi
Copy link
Member

findepi commented Mar 8, 2022

Thank you for your work on this @jirassimok.

@findepi findepi mentioned this pull request Mar 8, 2022
Closed
@github-actions github-actions bot added this to the 373 milestone Mar 8, 2022
@mosabua mosabua mentioned this pull request Mar 8, 2022
Merged
@ebyhr ebyhr mentioned this pull request Mar 29, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nineinchnick nineinchnick nineinchnick left review comments

@findepi findepi findepi approved these changes

@vkorukanti vkorukanti vkorukanti approved these changes

@alexjo2144 alexjo2144 Awaiting requested review from alexjo2144

Assignees
No one assigned
Labels
cla-signed
Milestone
373
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@jirassimok @mosabua @sajjoseph @findepi @nineinchnick @vkorukanti