[CT-76] [Feature] Allow vars to be declared in selectors

[jjuanramos]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

As of now, selectors are a great way of allowing dbt users to version control the dbt cloud jobs we run. Nevertheless, when it comes to variables we are still limited to declare them on the dbt Cloud UI.

It would be great if we could set the variables in the selector itself.
This could look like this:

selectors:
  - name: update_warehouse
    description: >
      Selection of models that we want to run when we
      update the whole warehouse.
    definition:
      union:
        - method: fqn
          value: "*"
        - exclude:
            - 'tag:tag_1'
            - 'base.kafka'
      vars:
        - treat_base_sources_as_relations: true
        - excluded_base_sources_from_relation_treatment: ['kafka']

Achieving this in the dbt Cloud UI:
dbt build --selector update_warehouse

Describe alternatives you've considered

Keep declaring them on the UI:
dbt build --selector update_warehouse --vars "{treat_base_sources_as_relations: true, excluded_base_sources_from_relation_treatment: ['kafka']}"

Who will this benefit?

Any dbt user that wants to maximize how much of the dbt flow is version controlled.

More specifically, dbt users that rely heavily on variables will improve the reliability and ease of their release process. Right now, any dbt user that has a staging/production environment has to make sure the variables are manually updated in all of their dbt Cloud jobs whenever necessary.

Are you interested in contributing this feature?

Yes!

Anything else?

No response

[gshank]

I can see that this would be a useful feature. This would probably be easiest to implement if we consider the vars in the selectors as an alternative way of specifying cli vars, so that they are just added (internally) to any other cli vars. This would generally require a full re-parse (as providing different cli vars does today), though we are looking at enabling more granular re-parsing in the future.

@jtcohen6 would have to weigh in on how this would fit in to current development plans.

[jtcohen6]

The proposal in this issue feels similar to a few others, in that they're all solving a similar problem. Effectively, by writing vars in selectors.yml, you can version control the env-dependent logic and save copy-paste across dbt Cloud jobs.

I'm hesitant to add vars right alongside selection logic in selectors.yml, though I see the mental connection between the two. I'd be more excited about:

• Supporting env-specific logic in the vars dictionary in dbt_project.yml, by Jinja rendering them ([Feature] env_var does not work in an object under the vars section of dbt_project.yml #4314 + target-based Jinja rendering of vars in dbt_project.yml #3105) — that way, you can "flip" a number of vars by means of a single env var, and keep the most important logic in version control
• Rather than continuing to over-load dbt_project.yml (and further complicate its parsing order + many rendering contexts), we should stick vars in a new separate file, vars.yml (Defining vars, folder-level configs outside dbt_project.yml #2955)
• A lot of this is also solved if dbt Cloud could offer better support for file-based, version-controlled job definitions ([Feature] Add an --input-file option to run-operation #4139)

@gshank Your points about partial parsing are also spot on, and the performance considerations are a big blocker to the ergonomics of vars (wherever they're defined).

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.