[CT-3438] UserConfig in profiles.yml -> flags in dbt_project.yml

[jtcohen6]

Housekeeping

• I am a maintainer of dbt-core

Short description

We are deprecating config as a thing that is settable in profiles.yml. Instead, we will support flags in dbt_project.yml

For dbt-core v1.8, we should see if the user has defined config in profiles.yml.

• If they have, continue to respect it, and raise a deprecation warning.
• If they haven't, great. Instead, we will be reading flags from dbt_project.yml. The two are mutually exclusive.

In a future release, we will remove UserConfig from profiles.yml entirely.

Acceptance criteria

In dbt_project.yml, it will look like:

# dbt_project.yml
flags:
  send_anonymous_usage_stats: True

Going forward, some flags will be settable in:

1. "Project flags" (dbt_project.yml) only. These flags define behaviors tied to the source code in a project, and should remain consistent across all users, environments, and invocations of that project.
2. CLI option / env var only. These flags are highly specific to the running invocation, and don't make sense to define in version-controlled code. Several of these are file system paths, such as --project-dir or --log-path. Some want to vary across commands, such as --full-refresh. Environment variables are best suited to flags that want different settings in different environments, such as turning DBT_FAIL_FAST on in dev/CI but not in prod.
3. Both places. In this case, a project wants to set a default behavior, but it may be appropriate to override it for different environments/invocations.

If set in both places, the inheritance hierarchy remains the same:

1. CLI var (handled by click)
2. Env var (handled by click)
3. User config Project flags

Implementation Notes

• We should rename the internal UserConfig to ProjectFlags
• User config is read when Flags is instantiated here. This is probably still the right place to do it, but the logic now becomes:
  • read_project_flags:
    • tries to load a user config from profiles.yml. If found, read it and raise a deprecation warning
    • if not found, tries to load user config from dbt_project.yml (can be a minimal yaml parse for just the 'flags' key - no jinja parsing necessary)
    • otherwise, use default user config settings (Nones are appropriate to ensure flag precedence works as expected for flags that can be set via CLI or env var)
• this will also need to be updated in core.dbt.flags.set_from_args

Testing:

• let's add a unit test that ensures that: every attribute of ProjectFlags that has a corresponding click option in params.py should be set to None by default (except for anon user tracking). Going forward, flags can have non-None defaults if they do not have a corresponding CLI option/env var. These will be used to control backwards incompatible interface or behaviour changes.

Impact to Other Teams

N/A

Will backports be required?

no :)

Context

Implications of deprecation

Why was this called UserConfig before? Well... hypothetically, if Michelle wants colorized logs, and Jeremy doesn't, they should be able to set user-specific preferences that apply across all their projects, without checking those preferences into those projects' version control (because profiles.yml lives in the HOME directory on each machine). While we are removing UserConfig from profiles.yml, it will remain possible to set user-specific preferences by setting environment variables (in .bashrc, .zshrc, etc).

[jtcohen6]

From chatting with @gshank: Let's see what the complexity will be for implementation a "soft" deprecation. If it's a lot of additional work, we should instead opt for a "hard" deprecation in v1.8 (with a lot of prior notice), i.e. stop reading config from profiles.yml entirely.

[vianmora]

Hello :)
I'm comming from this Issue : #6207 (comment)
and before, this one : #8026

I'm trying to understand, are you going to allow the use of .env (or dbt.env) files? Or is the project abandoned as the issue is closed ? :)

[jtcohen6]

Hey @vianmora! Thanks for following up here. We kicked that part out of scope for now, to refocus on:

• moving flags (f.k.a. "user configs") from profiles.yml → dbt_project.yml
• defining the intended use of flags for managing behavior changes

A few questions for you:

• Are you still interested in using .env files to set environment variables?
• Are these env vars for global/runtime configs (things like DBT_FAIL_FAST), or for env vars that modify behavior in your project (accessed via {{ env_var(...) }})?
• How are you developing/deploying dbt? (using dbt Cloud or otherwise)

[nikohunt]

Hey @vianmora! Thanks for following up here. We kicked that part out of scope for now, to refocus on:

• moving flags (f.k.a. "user configs") from profiles.yml → dbt_project.yml
• defining the intended use of flags for managing behavior changes

A few questions for you:

• Are you still interested in using .env files to set environment variables?
• Are these env vars for global/runtime configs (things like DBT_FAIL_FAST), or for env vars that modify behavior in your project (accessed via {{ env_var(...) }})?
• How are you developing/deploying dbt? (using dbt Cloud or otherwise)

@jtcohen6
I'm still interested in better handling of .env., please.

• The env vars are for holding profile config (e.g., host, username etc.) and so not checked in to a repo.
• I'm using dbt Core, orchestrated by Dagster.

[conorpp]

It's be great to support loading variables from .env as this is more consistent with other projects (e.g. meltano, dagster). If we can put secrets in .env then we don't need to also duplicate them to dbt's profiles.yml.

[nikohunt]

@conorpp, I've found that adding the prefix DBT_ or DBT_ENV_SECRET_ to my variables in .env seem to cause these .env files to be used by dbt, negating the need for using set -a; source .env; set +a.

[elyobo]

Would this possibly have caused this issue in sqlfluff (that has definitely been caused by something in 1.7.14)? sqlfluff/sqlfluff#5841

[martanthony]

For dbt-core v1.8, we should see if the user has defined config in profiles.yml.

...

Will backports be required?

no :)

Curious to understand why this was also released to 1.6 and 1.7? There's nothing on this ticket or the pull request to indicate this should have affected versions other than 1.8

[jtcohen6]

@elyobo Thanks for flagging - I've responded on the linked sqlfluff issue.

@martanthony You're right, it wasn't our initial intent to backport this change. We decided to do this recently in order to provide existing users with the deprecation warning and mechanism to opt into more-secure (less-surprising) behavior around how built-in materializations from packages are overridden:

• Raise deprecation warning if installed package overrides built-in materialization #9981
• https://docs.getdbt.com/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization

Apologies for not reflecting that on this issue in advance. Has it caused any issues for you, besides the one noted above for sqlfluff?

[martanthony]

@jtcohen6 understood – thanks for the prompt response. We have a step in our CI that consumes the output of dbt ls and the deprecation warning appears to have broken the way that output is handled by us. Easily mitgated by moving the profile configs to dbt_project flags though.