Give profiles some love

[clrcrl]

I recently rewrote some docs on connecting to your data warehouse when using dbt Core.

In doing so, I started to thing pretty hard about whether the names/structure we've used so far are still a good fit.

Some random thoughts:

• I think folks sometimes think that because targets are named a “target”, there’s also somewhere to define a “source”. Is “target” a misleading word? Is envs a better word instead?
• The current structure has some weirdness in it, e.g. “targets” are listed under the “outputs” key.
• The keys “schema” and “default” are not descriptive enough — is “schema” the schema I’m reading from or writing to? “default” what? Oh, default output default target.
• When defining multiple targets for a profile, you have to repeat connection details

Here's my proposed new schema:

version: 2

profiles: # should this be connections instead?
- name: jaffle_shop
  default_env: dev
  connection_details: # this isn’t the right key name... yet, but this is the default connection details, which you can override in a particular target. 
    type: redshift
    host: xyz
    ....
    user: abc
    pw: pas55w0rd
  envs:
    - name: dev
      target_schema: dbt_claire
      connection_details: # here you can override/add to anything defined above
        user: claire
        password: pa55w0rd
    - name: prod
      target_schema: analytics
      connection_details: 
        user: claire_super
        password: pa55w0rd

[clrcrl]

Evidence from Slack:

Nick S Today at 11:04 AM
Hi Everyone, I'm new to dbt and trying to figure out something which may be frowned upon.
We have a lot of data sources coming in to the warehouse, each source going to its own schema.
There may be cases where we would want to create a model using data from multiple schemas. Is this possible at all with dbt?
...

Nick S 3 minutes ago
Ooo ok I missed where you can override schema in sources. How well does this play with a schema set in profile?

Nick S 2 minutes ago
Ah I think I understand what I've missed here

Nick S 2 minutes ago
the profile specifies the target - where the models will be built. But doesn't specify the source schema.

[drewbanin]

Thanks for raising this issue @clrcrl - I agree that we'll want to update the profile specification syntax, and I do like the structure you've outlined here.

The addition of a version number is a really good idea - it will make the upgrade path for users significantly easier, especially since this is a cosmetic change that users don't need to make right away.

I think a good name for connection_details could just be connection. Is there a specific reason you don't like that name?

Some thoughts that pop into my head as I look at this:

• if we change targets to envs, we'll probably also want to update the --target flag on the CLI
• this is more complicated to write than the existing profiles.yml structure. I think it's better, but we should probably anticipate that folks who are new to yaml could have trouble with the distinction between lists and dicts.
• I love that this format doesn't require you to repeat creds for dev/prod envs
• since the list of envs is a list (instead of a dict), we don't strictly need a default_env key to be present. Users could add that if they want to set a default, but dbt could otherwise just use the first env in the list (just a thought).

Let me know what you think about all of this

[clrcrl]

More evidence:

Stephen Lloyd
I’m trying to understand how to set up profiles to have more than one environment. I don’t fully understand the difference between target and outputs.

I somehow missed this reply:

• connection_details vs connection: happy to use connection here!
• I like the default env idea!

[jaypeedevlin]

I just spent a good amount of time trying to work out how targets relate to outputs in order to help a colleague get setup in dbt cloud. A big +1 on this!!

[jtcohen6]

Let's give this some consideration in advance of v1.0. I'm going to close #1165 in favor of this proposal.

[leahwicz]

• Update the start-project profile when we do this work

[nathaniel-may]

• What is the exit criteria for this issue?
  • target renamed to envs
  • no more output section:
    • name, default_env, connection, envs for each yaml array element in profiles
    • name, target_schema, connection_details (overrides) for each array element in envs.
  • update --target flag on cli to a new name (TODO: to what? env?)
  • add alias for the jinja context variable called target with the new name
  • update init command to use the new version syntax
  • backwards compatible: you can use either v1 or v2.
• What are the high-level items of the work that need to be done (i.e. create x, split out y, etc.)
  • add version 2 to profiles parser
  • parse new structure
  • update target flag
  • update init command
  • add/modify integration tests to test the new profiles version
• What are the open questions on this issue that still need answers?
  • What is the new target flag name?
• Are there blockers/prerequisites to starting this work?
  • I think it could be started.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[jaypeedevlin]

Commenting to stop this one being auto-closed, although dbt Labs folks feel free to overrule me.

I think this remains a confusing part of the dbt experience, which is especially important as it's often one of the first things a new dbt-core user has to work out. I'd love to see this targeted for 2.0 (and would be happy to contribute in part as much as I have the technical ability to).

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.

[jtcohen6]

Reopening for now - I'd be excited to refactor profiles.yml next year :)

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.