check env variables for profile configs

[drewbanin]

This could work by:

• Specifying a path to a valid profiles.yml file
• Specifying a JSON blob containing valid profiles info
• Specifying config-level options (eg. DBT_PROFILE_USER='...')

[enkeboll]

I (and others) would love to see this. Our ops team wants us to avoid writing arbitrary flat files as a part of the deploy process, and we can't check in secrets to our repositories.

Specifying a path to a valid profiles.yml file

This seems little different than the --profiles-dir option. While not a bad idea, it doesn't solve the problem

Specifying a JSON blob containing valid profiles info

This would definitely work, but might get unwieldy depending on how many connections you have

Specifying config-level options (eg. DBT_PROFILE_USER='...')

This again could get hairy if you have multiple connections, but it's closest to what I have in mind. I could also imagine something where in a secrets-free profiles.yml, there's a pass-env-var-name: key that lets you pluck that secret for an arbitrary environment variable.

[drewbanin]

This is really great feedback, thanks @enkeboll.

Here's where my head goes:

If we parse the profiles.yml file as jinja code, we could put an env function into the jinja context. Then you could somewhat naturally pull in arbitrarily named environment variables. Eg:

my-profile:
  outputs:
    prod:
      type: redshift
      host: "{{ env('DBT_HOST') }}"
      user: "{{ env('DBT_USER') }}"
      pass: "{{ env('DBT_PASS') }}"
      port: 5439
      dbname: analytics
      schema: analytics
      threads: 1
  target: prod

Definitely open to alternatives here though.

[enkeboll]

Love it! I think that's an ideal solution.