Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow writing data to managed Delta tables on Databricks #839

Closed
zzeekk opened this issue Jun 1, 2024 · 0 comments
Closed

Allow writing data to managed Delta tables on Databricks #839

zzeekk opened this issue Jun 1, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@zzeekk
Copy link
Contributor

zzeekk commented Jun 1, 2024

Is your feature request related to a problem? Please describe.
Tables in Hive Catalog can be managed or external. Data for managed tables is stored in the Warehouse directory and their lifecycle is managed by the catalog, e.g. dropping the table also deletes the data/files on the disk. External Tables have their data stored in a custom directory, which is not managed by the catalog, e.g. dropping the table does not deletes the data/files on the disk.

Currently SDLB supports only writing to external table for Hive & DeltaLake. Configuring a path is mandatory when writing to such a DataObject, and is good for data protection and flexibility. But using managed tables is easier and in some Databricks environments the only official way.

SDLB should support writing to managed DeltaLake Tables on Databricks.

Describe the solution you'd like
Add an additional configuration attribute DeltaLakeTableDataObject.isExternal = true (default).
If isExternal = false, configuring a path for writing to the table should not be needed. This implies that the Spark Table API (saveAsTable/insertInto) is used instead of writer....save().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants