[destinations] Databricks

[AstrakhantsevaAA]

Feature description

Implement Databricks destination. It will be quite similar to the Snowflake implementation.

• use Databricks Python Connector
• decide what to use COPY INTO or AutoLoader

Authorization:

• Databricks personal access token
• OAuth with refresh token

Support data sources (staging storage):

• Managed tables
• Amazon S3 (External storage)
• Azure Blob Storage (External storage)

Tests

• we should enable it as destination in tests. all tests using ALL_DESTINATIONS must pass
• it should pass the same common tests which BigQuery job client and sql client is passing
• note that to pass sql_client tests you'll need to map DBApi exceptions into 3 categories that dlt needs: relation not found, terminal exceptions, transient exceptions. see bigquery and postgres implementations. that's manual work

[rudolfix]

@AstrakhantsevaAA let's sync on slack as well. we should invite the implementer as contributor so it is easy to run our CI.

[phillem15]

@AstrakhantsevaAA, this looks great. Under "Support data sources", does this mean that S3 and ADLS would only be supported for the staging part? Or would this also be available for the destination dataset as well? We would likely want the option to have the destination dataset be created as an "external table".

Also, Auto-Loader for Databricks SQL is currently in public preview (see here). I typically use COPY INTO, but having the option for autoloader would be nice to have.

[AstrakhantsevaAA]

Under "Support data sources"

@phillem15 yes, I meant staging data sources. User can use filesystem destination to load files directly to s3, azure, gs storage. Or use filesystem verified source as a source to load files from buckets to the Databricks workspace.

[rudolfix]

@phillem15 are people using databricks file system like they use s3 for athena? actually we could support it in our filesystem destination and source so people can send files to is as a staging (and also read)
https://filesystem-spec.readthedocs.io/en/stable/_modules/fsspec/implementations/dbfs.html
UPDATE: it seems that dbfs is abstraction over bucket storage so the question is if going via dbfs is something that people do, or they just interact with bucket storage directly

[mrhopko]

Is this being worked on? I am happy to contribute if required?

[AstrakhantsevaAA]

Is this being worked on? I am happy to contribute if required?

@phillem15 works on it, I'll check the development status and if it turns out that we need help, I'll get back to you. Thank you for your interest!