Add support for Databricks Dataset

[tiencuongkieu]

Description

Currently Kedro doesn't support read/write directly to Databricks tables. Should we support this feature?

Context

Databricks is used in many projects. And using Databricks tables, instead of, for example S3, has several benefits like we can see sample data in Databricks UI, we have metadata for tables, it is also faster.

Possible Implementation

I already implemented the dataset and used it in a client project.

[datajoely]

Hi @tiencuongkieu - are you talking about regular databricks tables or delta tables?

You can do the former via the SparkJDBCDataSet today, but we also have a Delta dataset in progress as well #964.

[jiriklein]

I think @tiencuongkieu might also mean the native saveAsTable functionality of tables within environment such as Databricks, in line with https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html#saving-to-persistent-tables
In such case it should be a simple inheritance of SparkDataSet with the _save and _load methods overriden to call the relevant methods.

[datajoely]

Ah I wasn't aware of that! @tiencuongkieu we're always open to PRs :)

[tiencuongkieu]

Yes, exactly as @jiriklein mentioned above, it is for the native saveAsTable and the code is quite simple as he mentioned as well. I'll raise a PR.

[datajoely]

Thank you!

[jiriklein]

Awesome, thank you @tiencuongkieu
I recommend this is rather called SparkTableDataSet or similar - it's not related strictly to databricks, but to any underlying system that allows for table metastore.

Then inheritance should come from SparkDataSet or AbstractVersionedDataSet as parent class, with overriding for __init__ (instead of filepath, you have a "database" and table - or fully-qualified name), then _load and _save methods as discussed.

Let us know if and when you need help with reviews.

[tiencuongkieu]

Thank you @jiriklein for you suggestion. I raised a draft PR. Still need to add tests though.

[tiencuongkieu]

Just copied it over from my project's implementation. Inheritance comes from AbtractDataSet only. Maybe we need versioning functionality as well?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.