-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Delta Lake connector #10897
Add Delta Lake connector #10897
Conversation
c47598f
to
f1954e6
Compare
💯 |
f1954e6
to
2ee7d4d
Compare
@jirassimok - thanks for the contribution. Are there any features that are only available in Starburst version of this connector? (Z ordering, delta write capability etc - are they supported in the version here)? I know that Databricks team created their version of Trino delta connector (#10300). Is there any plan to combine the efforts? |
@sajjoseph thanks for your comments
writes are supported here
we're aware and we've been discussing the best way to move forward. cc @vkorukanti |
@jirassimok and @findepi .. do you intend to add the connector docs here or should we start a parallel docs PR and work with you on that? |
I don't plan to work on the docs for a while yet, so it might be easier to keep the PRs separate. |
92954b7
to
d5db955
Compare
cc @vkorukanti |
93d8fd5
to
fbd36a2
Compare
@nineinchnick can you PTAL the CI changes? |
8860515
to
9061fd5
Compare
e7e41e2
to
0257a7f
Compare
rebased because master version changerd |
...trino-delta-lake/src/main/java/io/trino/plugin/deltalake/statistics/DeltaLakeStatistics.java
Show resolved
Hide resolved
...delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/DropExtendedStatsProcedure.java
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java
Show resolved
Hide resolved
...ake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointBuilder.java
Show resolved
Hide resolved
...ake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointBuilder.java
Show resolved
Hide resolved
...lake/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointWriter.java
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConfig.java
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConnectorFactory.java
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeErrorCode.java
Outdated
Show resolved
Hide resolved
fe7c731
to
85565eb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Marking red so that i don't forget about the comment in the ci.yml
# TODO: Restore condition above: && env.ABFS_ACCOUNT != '' && env.ABFS_CONTAINER != '' && env.ABFS_ACCESS_KEY != ''
8d57adf
to
c71c88b
Compare
rebased to resolve logical conflict because of #11286 |
...-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAdlsConnectorSmokeTest.java
Outdated
Show resolved
Hide resolved
plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAnalyze.java
Outdated
Show resolved
Hide resolved
b131a78
to
9cfa97d
Compare
(rebased because #11312 merged introducing a logical merge conflict) |
f9958f6
to
7d7ae75
Compare
This connecter was originally written for the Starburst Enterprise platform. Co-authored by various engineers at Starburst Data: Co-authored-by: Alex Jo <alex.jo@starburstdata.com> Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> Co-authored-by: Łukasz Osipiuk <lukasz@osipiuk.net> Co-authored-by: Alex Albu <alex.albu@starburstdata.com> Co-authored-by: Mateusz Gajewski <mateusz.gajewski@gmail.com> Co-authored-by: Raunaq Morarka <raunaqmorarka@gmail.com> Co-authored-by: Ashhar Hasan <ashhar.hasan@starburstdata.com> Co-authored-by: Michał Ślizak <michal.slizak+github@gmail.com> Co-authored-by: Justin-Kwan <justin.kwan@starburstdata.com> Co-authored-by: Grzegorz Kokosiński <grzegorz@starburstdata.com> Co-authored-by: Arkadiusz Czajkowski <arek@starburstdata.com> Co-authored-by: Konrad Dziedzic <konraddziedzic@gmail.com> Co-authored-by: Jacob I. Komissar <jacob.komissar@starburstdata.com> Co-authored-by: Krzysztof Sobolewski <krzysztof.sobolewski@starburstdata.com> Co-authored-by: Marius Grama <findinpath@gmail.com> Co-authored-by: Irek Matysiewicz <ireneusz.matysiewicz@starburstdata.com> Co-authored-by: Krzysztof Skrzypczynski <krzysztof.skrzypczynski@starburstdata.com> Co-authored-by: Jan Was <jan.was@starburstdata.com> Co-authored-by: Yuya Ebihara <yuya.ebihara@starburstdata.com> Co-authored-by: praveenkrishna <praveenkrishna@tutanota.com> Co-authored-by: Karol Sobczak <napewnotrafi@gmail.com> Co-authored-by: David Phillips <david@acz.org> Co-authored-by: Sasha Sheikin <myminitrue@gmail.com> Co-authored-by: Charles Morgan <cjmorgan2000@gmail.com> Co-authored-by: Martin Traverso <mtraverso@gmail.com> Co-authored-by: Jason van Zyl <jason@vanzyl.ca> Co-authored-by: Szymon Homa <szymon.homa@starburstdata.com> Co-authored-by: Dain Sundstrom <dain@iq80.com>
b5fbd9d
to
2aff0d7
Compare
Thank you for your work on this @jirassimok. |
This connector was originally written for the Starburst Enterprise platform and is now being contributed to Trino.
For #11296