Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-1492] [Feature] Add support for Iceberg Tables #321

Open
3 tasks done
teej opened this issue Nov 10, 2022 · 32 comments · May be fixed by #961
Open
3 tasks done

[CT-1492] [Feature] Add support for Iceberg Tables #321

teej opened this issue Nov 10, 2022 · 32 comments · May be fixed by #961
Assignees
Labels
enhancement New feature or request

Comments

@teej
Copy link

teej commented Nov 10, 2022

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-snowflake functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Snowflake has announced support for Iceberg Tables, an external table format but with full interoperability as a 1st class snowflake table.

Once this feature goes live, dbt-snowflake should support materializing a model as an external iceberg table.

Describe alternatives you've considered

No response

Who will this benefit?

  • dbt users who want their models available as parquet files in storage they control
  • dbt users who want an easy way to interoperate with non-dbt systems that use the iceberg table format
  • dbt users who want direct access and control over the underlying storage of their data

Are you interested in contributing this feature?

yes

Anything else?

Iceberg Tables are in private preview as of 2022-11-10

@teej teej added enhancement New feature or request triage labels Nov 10, 2022
@github-actions github-actions bot changed the title [Feature] Add support for Iceberg Tables [CT-1492] [Feature] Add support for Iceberg Tables Nov 10, 2022
@erniedeferia
Copy link

+1

@Fleid Fleid self-assigned this Nov 23, 2022
@Fleid Fleid removed the triage label Dec 5, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2023

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

@github-actions github-actions bot added the Stale label Jun 4, 2023
@Fleid Fleid removed the Stale label Jun 5, 2023
@jaklan
Copy link

jaklan commented Jul 24, 2023

Any chance to put more light on that? We really need a Snowflake equivalent of https://github.com/dbt-athena/dbt-athena to push forward the adoption of open table format (which, I believe, would be also pretty crucial for dbt to support in the long term).

@multiprocessingh
Copy link

+1 here! This would be super useful to support for enterprise sas use cases!

@WangCHEN9
Copy link

+1 here!

@DLewisEndeavor
Copy link

+1 here! Looking into Iceberg for enterprise sharing across platforms

@apd-bscally
Copy link

+1, Iceberg is now in public preview on Snowflake

@mourtallah
Copy link

+1 here!

1 similar comment
@VSchmidts
Copy link

+1 here!

@mikealfare mikealfare added the refinement Product or leadership input needed label Jan 17, 2024
@ksdaly
Copy link

ksdaly commented Jan 19, 2024

+1

2 similar comments
@ZergRocks
Copy link

+1

@sfc-gh-ripu
Copy link

+1

@jimvekemans
Copy link

+1, will work on this a bit in the coming weeks and, if successful, open a pull request

@dennysrega1ado
Copy link

+1

@waitsfornone
Copy link

+1 here. We are adopting Iceberg tables as our data lake format and I would love to generate the tables out of Snowflake with dbt.

@mikezusev
Copy link

+1

1 similar comment
@zewang-sanofi
Copy link

+1

@sfc-gh-wriley
Copy link

+1 :-)

@martynydbt martynydbt removed the refinement Product or leadership input needed label Feb 14, 2024
@njrogers
Copy link

+1

@Fleid
Copy link
Contributor

Fleid commented Feb 26, 2024

Hey all! Thanks for "upvoting", please continue to do so.
We are wrapping up all the internals work to support 1.8, plus a few more things to do for quality of the plugins and how we test them, and then Iceberg will be at the top of the list!

@sfc-gh-ftrejo
Copy link

+1

@sfc-gh-elethier
Copy link

+1

4 similar comments
@sfc-gh-amdeshpande
Copy link

+1

@sfc-gh-mpowell
Copy link

+1

@joatkh
Copy link

joatkh commented Mar 6, 2024

+1

@Jemsley0
Copy link

Jemsley0 commented Mar 7, 2024

+1

@ncapeta
Copy link

ncapeta commented Mar 8, 2024

Any ETA on 1.8 ?

@jonask137
Copy link

Any ETA on 1.8 ?

According to the documentation, it should be released as of April 2024: https://docs.getdbt.com/docs/dbt-versions/core#latest-releases

@shayms8
Copy link

shayms8 commented Mar 20, 2024

i'm getting errors when trying to query a MAP iceberg object from dbt:

  002040 (42601): SQL compilation error:
  Unsupported data type 'map(VARCHAR(16777216), VARCHAR(16777216))'.
  compiled Code at target/run/dbt_snowflake_1/models/L1_iceberg/L1__client_events_iceberg.sql

any idea why it happens? can i somehow solve it?
didn't find a documentation suggesting MAP columns are not supported on dbt_snowflake integration

@dataders dataders linked a pull request Mar 28, 2024 that will close this issue
@ShahNewazKhan
Copy link

+1

@sfc-gh-wriley
Copy link

Is this now LIVE...? Was slated for APRIL release?

@sivakanthavel-tigeranalytics

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.