Skip to content

Version 1.3.0

Compare
Choose a tag to compare
@ueshin ueshin released this 14 Oct 18:54
· 25 commits to 1.3.latest since this release

Features

  • Support python model through run command API, currently supported materializations are table and incremental. (dbt-labs/dbt-spark#377, #126)
  • Enable Pandas and Pandas-on-Spark DataFrames for dbt python models (dbt-labs/dbt-spark#469, #181)
  • Support job cluster in notebook submission method (dbt-labs/dbt-spark#467, #194)
    • In all_purpose_cluster submission method, a config http_path can be specified in Python model config to switch the cluster where Python model runs.
      def model(dbt, _):
          dbt.config(
              materialized='table',
              http_path='...'
          )
          ...
  • Use builtin timestampadd and timestampdiff functions for dateadd/datediff macros if available (#185)
  • Implement testing for a test for various Python models (#189)
  • Implement testing for type_boolean in Databricks (dbt-labs/dbt-spark#471, #188)
  • Add a macro to support COPY INTO (#190)

Under the hood

  • Apply "Initial refactoring of incremental materialization" (#148)
    • Now dbt-databricks uses adapter.get_incremental_strategy_macro instead of dbt_spark_get_incremental_sql macro to dispatch the incremental strategy macro. The overwritten dbt_spark_get_incremental_sql macro will not work anymore.
  • Better interface for python submission (dbt-labs/dbt-spark#452, #178)