Add more template fields to DbtBaseOperator

[dwreeves]

This partially addresses #754 via allowing for built-in templating support for the DbtBaseOperator.

I also noticed --full-refresh was not documented so I added that in.

Still missing

Manual run pattern is not documented; the fact that these fields are templated is not documented. I don't really know where in the docs to put this. The docs are very API-focused more than narrative-based or suggestive, and Cosmos's maintainers prefer this style of documentation so it's hard to find a spot for this. It's possible that that's fine and we just keep this as a feature for more advanced users who dig into source code to discover for themselves? 🤷

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name	Link
🔨 Latest commit	d44424b
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/65e10b172f877300087b29de
😎 Deploy Preview	https://deploy-preview-786--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.39%. Comparing base (1bf78b4) to head (d44424b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #786      +/-   ##
==========================================
+ Coverage   95.35%   95.39%   +0.03%     
==========================================
  Files          57       57              
  Lines        2737     2759      +22     
==========================================
+ Hits         2610     2632      +22     
  Misses        127      127              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

Hey @dwreeves, thanks a lot for the contribution!

Great catch on the missing docstring and docs.

The docs are more API-focused than narrative-based or suggestive, and Cosmos's maintainers prefer this documentation style, so it's hard to find a spot for this.

We can and should certainly improve Cosmos docs. I agree they are more API-focused, but it isn't a maintainers' preference. We're open to having other styles of docs, and I'd be super happy if we could follow https://diataxis.fr/. It has not been a priority so far - but we'd welcome help.

Regarding template fields, given the existing docs, I think we could either:

(1) Create a new page inside docs/configuration with a list of existing template fields

(2) Highlight the Cosmos arguments that are template fields where we expose them. Eg. In the case of select, exclude and selection, this could be in:
https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html

In the case of env and vars, we could probably set it in dbt_vars and env_vars in:
https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html

We should probably add full_refresh together with models:
https://astronomer.github.io/astronomer-cosmos/configuration/operator-args.html

As of compiled SQL, it seems we have a dedicated section for it:
https://astronomer.github.io/astronomer-cosmos/configuration/compiled-sql.html
Which could probably be merged into operator_args.

I'm happy for us to merge this PR once we have some docs related to the template fields.

[joppevos]

@dwreeves oops I did not document it when adding full-refresh. Thanks for taking care of it 🙇

[dwreeves]

I think the thing that would make the most sense for now is to just add a section under /operator-args.html that is just about the templated variables. With the exception of --full-refresh, all of these variables are either (1) potentially used for rendering if DBT_LS is true (vars and env), or (2) used for model selection (select, selector, models, exclude) and are thus typically out of scope for operator_args inside of the dbt->Airflow converter context. A note should be included that clarifies this, although it's a little confusing because --full-refresh is essentially the only variable where it would make sense to do something like: operator_args={"full_refresh": "{{ params.full_refresh }}"}.

(2) Highlight the Cosmos arguments that are template fields where we expose them. Eg. In the case of select, exclude and selection, this could be in:
https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html

So this is a bit weird because templating does not work when rendering a DAG, only when executing the task. I will keep in mind that a note should be made about this.

---

This also does beg a question though about the recent treatment of the dbt_vars and dbt_env. Right now in validate_initial_user_config(), an error is raised when both operator_args={"vars": ...} and project_config.dbt_vars are set. Since both rendering and execution use --vars and environment variables, this means that rendering and execution are strongly coupled and it's not possible to have these differ.

This was something I was thinking about irrespective of this particular PR. I think it's possible that it should be possible to set both operator_args={"vars": ...} and project_config.dbt_vars, and that the behavior would be that project_config.dbt_vars is used for rendering and operator_args={"vars": ...} could either merge or overwrite. Alternatively, render config and execution config could each have their own notion of dbt vars that can get set to the project configs' vars if either is not defined at the execution or rendering level.

In general the render and execution configs share a lot of configuration in the case that DBT_LS needs to be run, and maybe a separate discussion but I think there should be some sort of way to make the configurations more decoupled at the user level, but more DRY in their under-the-hood implementations. If that makes sense.

---

The reason why this discussion is potentially relevant to this PR is because of my note above about how the only operator args you can seemingly reliably utilize templating for at Cosmos's "highest level" API (i.e. DbtDag + DbtTaskGroup) is full_refresh.

I mean, you can access vars and env technically, but these will either:

• if set in operator_args, they will raise deprecation warnings as of 1.3
• if set in project_config and LoadMode.DBT_LS is used, then it will not properly render at all during LoadMode.DBT_LS rendering, i.e. the string literal will be used for either the dbt var or env var, since it never gets passed through Jinja and the scheduler wouldn't have potentially relevant context variables anyway.

This all feels both somewhat suboptimal, and also extremely confusing, to users. It's strange that ProjectConfig(dbt_vars={'logical_date': '{{ context["logical_date"] }}') is valid code, and works perfectly fine and as-intended if the user is using LoadMode.DBT_MANIFEST, but does not behave so well when using LoadMode.DBT_LS. It's tricky to convey this in documentation as a general statement. "Templating does not work with ProjectConfig(dbt_vars=...)" is not a true statement, but it is not a false statement either. The truth is templating works only in execution, and may or may not fail in rendering. Here are 2 example cases when it fails in rendering:

• Since executing dbt ls does involve dbt parsing all the Jinja, and if your SQL query has something like {{ modules.datetime.datetime.fromisoformat(var("logical_date")) }} then the code will fail when rendering even though it would succeed when executing. In this case I think the user would want to either not have logical_date set at all (and rely on a default instead), or alternatively they should be allowed to set logical_date differently in rendering and execution.

• Env vars can be a little more realistic a problem if, say, an env var is being used as part of a connection or to enable or disable models. For example, imagine the user sets a dbt profile with something like role: {{ env_var("DBT_ROLE") }}, and in their DAG they want to set DAG(..., params={"dbt_role": Param(...)}) and inside a DbtTaskGroup they want to set DbtTaskGroup(..., project_config=ProjectConfig(..., dbt_vars={"dbt_role": "{{ params.dbt_role }}"})). Here, again, it works fine in execution, but it does not work in the case where the load method is LoadMode.DBT_LS specifically.

That said, I am aware these are potentially extreme edge cases-- for example in most cases passing the string literal {{ logical_date }} into the dbt rendering will have zero impact on parsing with dbt ls and everything will render fine. So this adds to the confusion! The fully qualified correct statement is: Templating works in execution but does not work in rendering when the load method is LoadMode.DBT_LS; it may or may not matter depending on your dbt project. And that's just way more confusing than saying it doesn't work with dbt ls fully unqualified, and instead pushing users toward a separate layer of abstraction in the event that they want access to templating.

---

OK, this is all very confusing and in the weeds so I hope what I am saying makes sense. TLDR is:

• Probably just stick this in operator-args.html as a separate section.
• I would not mention this at all in ProjectConfig; and if we do, we may want to revisit some of the 1.3 changes regarding deprecation of vars= and env= in the operator_args?
• The use of templating is actually really complicated and isn't fully addressed by the abstractions. It may be best to revisit this.
• For now, without any further code changes, the only real thing that can be templated via DbtDag or DbtTaskGroup is full_refresh. (Maybe also warn users that render_template_as_native_obj=True should be true for that to work, since right now it cannot handle strings; alternatively make it so full_refresh can handle strings??).

[dwreeves]

Cov requirements are still failing. There is no universe where I get this done before the weekend, just as an FYI.

[tatiana]

Thanks, @dwreeves , I just managed to get some time to add the missing coverage bit - it will be released as part of the alpha today!

[dwreeves]

Yes! Thank you! Can't wait for this 😁

[tatiana]

Looks great, thank you very much, @dwreeves both for the contribution and persistence to get this into the finishing line!