Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix is_incremental() macro #48

Closed
jtcohen6 opened this issue Jan 28, 2020 · 0 comments · Fixed by #50
Closed

Fix is_incremental() macro #48

jtcohen6 opened this issue Jan 28, 2020 · 0 comments · Fixed by #50

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Jan 28, 2020

The default implementation of is_incremental() bundles the following four conditions:

relation is not none
and relation.type == 'table'
and model.config.materialized == 'incremental'
and not flags.FULL_REFRESH

The second condition is always false on Spark because relation objects don't naturally have a value for type. Instead, relation type is handled by the spark_get_relation_type macro, which infers it from

show tblproperties ... ('view.default.database')

There are a few potential fixes:

  • Reimplement is_incremental() for dbt-spark
  • Update the relation object with its type as part of the incremental materialization, since we're already calling spark_get_relation_type here
  • At the beginning of runs, cache all info on dbt relations, including relation types (similar to get_catalog, although this takes a while to run)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant