-
Notifications
You must be signed in to change notification settings - Fork 128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support require partition filter #112
Comments
This is a noble goal, but every model in the package would need to respect this. We have so much work ahead to even get models like Remember that any model that scans all partitions, by definition, does not use a partition filter. We have lots of those models. So I don't think it's as easy as setting a config variable. |
Manually adding this to every model is not what I intended by opening this issue. What we need to do to is to macro our partition header and where clauses. Some recent updates to our data marts only support dynamic partitioning while our base partitions support both dynamic and static partitions. This is an oversight that would be fixed by having partition macros applies across all models and it would make adding partition filters trivial. Also, I'm working on multisite right now which makes the advantages of templating things like partitioning seem more practical. |
The issue that I've come across is that moving incremental headers into a macro puts
The fix as I see it is to set all partitions macros in the same file. One sets partitions_to_replace. The other two call partitions_to_replace and output the header and where clause code. |
Adding a require partition filter to this structure should be trivial. |
Sorry, I'm not concerned about the mechanics of adding macros for where clauses and configs, I'm concerned about the logical implications of requiring partition filters. Take for example, Or take I definitely want to work on each of these issues, but we have to be careful of the logical implications of adding where filters and incremental materializations. It could change the data in unexpected ways. |
BigQuery has a require partition filter option that can only be set when creating a partitioned table (unlike table expiration which can be edited in the BigQuery console). As a result, the easiest way to set this config is via dbt when it creates or full-refreshes the models.
I expect that anyone wanting to configure this, will want to configure it for all models. The multi-site branch related to #107 has a partition header macro. It would be best to implement this configuration there after it gets merged into main.
The text was updated successfully, but these errors were encountered: