-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CT-899] Update manifest flat_graph after deferral, to support dbt-metrics compilation #5525
Comments
Let's make sure to do a test case here before we close this out |
Current StateAs @jtcohen6 raised in the above issue, there are problems with how the metrics package is currently handling model references. For the curious, this is because metrics can have nested dependencies that resolve as models 1+n nodes upstream. For example, if you have an expression metric that is built on a chain of 4 other metrics, we need to resolve the model(s) reference at the end of that chain. Right now we construct that information from Desired End StateAs most of this will happen behind the scenes, there are few constraints when it comes to resolving this issue. The end goal is that the relationship would be correctly picked up and understood . @jtcohen6 raised a few ideas on how to resolve this:
One thing I do want to call out is that we'd need this functionality to be able to support n+1 node depth. IE not just understanding Would love to hear thoughts about solutions from the languages team! |
@callum-mcdata Thanks for the detailed comment! I just opened it as a new issue, for a spike/investigation: #5637 Let's keep using this ticket for the narrowly scoped fix described above: rebuild |
Metrics + deferral don't work perfectly together, because of how the
dbt-metrics
package is doing the "lookup" for a metric's model parent.I can envision two ways to solve this problem:
ref
in thedbt-metrics
package here, instead of re-constructing the model relation viagraph
lookup. I'd certainly feel better about it, if we did it this way :)flat_graph
after deferral. This is actually very simple, and could be a good way to go regardless. It will also enable users who are extracting metadata from thegraph
variable (e.g.dbt_artifacts
package rewrite: Release 1.0.0b1 brooklyn-data/dbt_artifacts#132). There are two places we could do this: b26c7caThose aren't mutually exclusive. Both could be good ideas in their own right.
Reproduction case
I have two profile targets set up:
prod
(withschema: analytics
) anddev
(withschema: dbt_jcohen
). The default isdev
.dbt run --target prod
target/manifest.json
) into a new folder, e.g.state/
sql: id
→sql: user_id
)dev
target.dbt ls -s state:modified+ --state state/ --target dev
to confirm thatsome_metric
+model_b
are both selected, andmodel_a
is not selecteddrop schema dbt_jcohen cascade
)dbt run -s state:modified+ --state state/ --defer --target dev
Resolution
Following proposal 2 above, after either of the additions in b26c7ca, this works:
Any concerns?
dbt-core/core/dbt/contracts/graph/manifest.py
Lines 690 to 701 in 1071a46
This does require us to rebuild
flat_graph
after parsing is over, at the beginning of execution, since that's where deferral happens. Deferral is resolved before any individual nodes are run in threads, though, so I don't believe we'd need to worry about thread safety.The text was updated successfully, but these errors were encountered: