[Feature] Databricks Compatibility

[fivetran-catfritz]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

For Databricks Compatibility, add the following:

1. Buildkite testing:
   • Update pre-command (example)
   • Update pipeline.yml (example)
   • Update sample.profiles.yml (example)
   • Add the below to integration_tests/dbt_project.yml if it's not there:

dispatch:
  - macro_namespace: dbt_utils
    search_order: ['spark_utils', 'dbt_utils']

2. For source packages, update src yml so a database won't be passed to spark (example or use below):

sources: 
  - name: <name>
    database: "{% if target.type != 'spark' %}{{ var('<name>_database', target.database) }}{% endif %}"

3. Update any incremental models to update partition_by for databricks and add current strategies if not present:

config(
        materialized='incremental',
        unique_key='<original unique key>',
        partition_by={'field': '<original field>', 'data_type': '<original data type>'} if target.type not in ('spark','databricks') else ['<original field>'],
        incremental_strategy = 'merge' if target.type not in ('postgres', 'redshift') else 'delete+insert',
        file_format = 'delta' 
)

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response