Default to current working directory for profiles.yml and fall back to ~/.dbt

[dbeatty10]

resolves #5411

uses #5715

Description

Use the following priority order to search for the profiles.yml configuration file:

1. --profiles-dir command-line argument
2. DBT_PROFILES_DIR environment variable
3. current working directory 👈 NEW
4. HOME directory of the user (i.e. ~/.dbt/)

Checklist

• I have read the contributing guide and understand what's expected of me
• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
  • Default to current working directory for profiles.yml and fall back to ~/.dbt docs.getdbt.com#2024
• I have run changie new to create a changelog entry

Outstanding questions

• We're testing that dbt debug produces the correct output; how many other subcommands do we need to test?
• Do we need to test that core/dbt/lib.py handles this new case properly? If so, then how?
• Is there any special testing that needs to be done for dbt Cloud or dbt Server?

Some non-critical curiosities while we're here:

• Looks like this is no longer used (after being moved in #4183):

  • dbt-core/core/dbt/task/debug.py

    Lines 24 to 26 in 49e7bdb

    	PROFILE_DIR_MESSAGE = """To view your profiles.yml file, run:
    	{open_cmd} {profiles_dir}"""

• Can references to dbt.config.profile.DEFAULT_PROFILES_DIR be replaced with dbt.flags.DEFAULT_PROFILES_DIR?

  • dbt-core/core/dbt/flags.py

    Line 13 in 49e7bdb

    DEFAULT_PROFILES_DIR = os.path.join(os.path.expanduser("~"), ".dbt")

  • dbt-core/core/dbt/config/profile.py

    Line 26 in 49e7bdb

    DEFAULT_PROFILES_DIR = os.path.join(os.path.expanduser("~"), ".dbt")

  • dbt-core/core/dbt/config/profile.py

    Line 47 in 49e7bdb

    profiles_file=DEFAULT_PROFILES_DIR

  • dbt-core/core/dbt/main.py

    Line 45 in 49e7bdb

    from dbt.config.profile import DEFAULT_PROFILES_DIR, read_user_config

  • dbt-core/test/unit/test_flags.py

    Line 8 in 49e7bdb

    from dbt.config.profile import DEFAULT_PROFILES_DIR

    • Last one isn't even used?

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

[jtcohen6]

@dbeatty10 Everything here makes sense to me! One point of clarification, and responses to your questions.

"Look here first" - which "here"?

3. current working directory of dbt_project.yml NEW

As discussed during our meeting on Wednesday, this should just be the current working directory. In 95% of cases, we should expect the current working directory == the directory of the active root project (dbt_project.yml). In cases where users set --project-dir, I think the cleanest behavior is:

• They also set a custom profile dir via CLI flag or env var → use it!
• They don't set a custom profile dir → use the current working directory (rather than implicitly taking direction from --project-dir), then fall back to the HOME directory

---

Questions

We're testing that dbt debug produces the correct output; how many other subcommands do we need to test?

Seems sufficient to me! dbt debug uses the same standard RuntimeConfig as all other tasks (except deps + clean, which don't use a profile at all).

Do we need to test that core/dbt/lib.py handles this new case properly? If so, then how?

I don't think so. The profiles_dir is manually set there currently, and only allows two options: env var or home dir.

dbt-core/core/dbt/lib.py

Lines 15 to 18 in f5a94fc

	if os.getenv("DBT_PROFILES_DIR"):
	profiles_dir = os.getenv("DBT_PROFILES_DIR")
	else:
	profiles_dir = os.path.expanduser("~/.dbt")

Is there any special testing that needs to be done for dbt Cloud or dbt Server?

dbt Cloud uses the --profiles-dir CLI flag, which always takes top precedence. I just checked to confirm.

---

Help text

I realized our CLI help text for --profiles-dir includes the default location (HOME dir). Should we update it to be "If not set, dbt will look in the current working directory first, otherwise the HOME?"

We have an old + new CLI now :)

dbt-core/core/dbt/main.py

Lines 261 to 264 in f5a94fc

	Which directory to look in for the profiles.yml file. Default = {}
	""".format(
	DEFAULT_PROFILES_DIR
	),

dbt-core/core/dbt/cli/params.py

Line 164 in f5a94fc

help=f"Which directory to look in for the profiles.yml file. Default = {PurePath.joinpath(Path.home(), '.dbt')}",

[dbeatty10]

Okay, I've made all those updates @jtcohen6:

1. Just be the current working directory (regardless of the location of the project directory)
2. Updated help text in both CLIs
   1. Updates to new CLI to align with current CLI (since the current CLI has no default values for --profiles-dir and --project-dir)
3. Remove a couple unused variables

[jtcohen6]

Ready to rock & roll, whether here or there

[lostmygithubaccount]

this is gonna make quickstart guides so much easier!

[leahwicz]

@dbeatty10 does that change anything to this doc #5425?

Sorry I owe you a review here...

[dbeatty10]

@dbeatty10 does that change anything to this doc #5425?

@leahwicz I'll double check if that ADR needs any changes.

This PR does implement this exactly:

However, we did not implement anything in relation to the non-existent DBT_PROJECT_DIR environment variable described here:

[gshank]

If you could merge main to handle the merge conflicts message, this looks good to me!

[dbeatty10]

@gshank merge conflicts handled now 👍