Support referencing across YAML config (e.g. jinja suppport)

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3112

Originally created by @kgpayne on 2021-12-09 10:31:49

---

In order to support resource references (#3108), it would be advantageous to support some kind of referencing or templating capability within the Meltano config. YAML reference support is available, but not as full featured enough to allow attribute selection or string formatting (e.g. to compile a resource URI from component user, password, host etc.).

As Meltano is built in python, Jinja comes to mind as an obvious choice. Meltano could provide a set of built-in macros for referencing config elements, akin to the ref and source macros provided in dbt.

# Example use
inventory:
  prod-snowflake-load:
    type: snowflake_warehouse
    properties:
      account: "<snowflake account name>"
      database: "<snowflake database name>"
      region: "<warehouse region>"
      warehouse: "<snowflake warehouse name>"
      username: "<user name>"
      password: "<password>" # ideally this would be a ref to a secret value added separately, or reference an env var
      role: "<snowflake role>"
      session_parameters: "<session level parameters>"

# Example usage
environments:
- name: prod
  config:
    plugins:
      loaders:
      - name: target-snowflake
        config:
          # Macro usage: `ref('<config object key path>', '<config object attribute>')`
          db_name: "{{ ref('inventory.prod-snowflake-load', 'database') }}" # References resources defined above
          account: "{{ ref('inventory.prod-snowflake-load', 'account') }}"
          user: "{{ ref('inventory.prod-snowflake-load', 'username') }}"
          password: "{{ ref('inventory.prod-snowflake-load', 'password') }}"
          role: "{{ ref('inventory.prod-snowflake-load', 'role') }}"
          load_schema: "${TAP_NAMESPACE}_raw"  # Templated string, value pulled from env var at tap execution

CLI Implications

This will have implications for meltano config <plugin> set <name> <value>. Whilst it is verbose, we could rely on straight string passthrough:

meltano config target-snowflake set db_name "{{ ref('inventory.prod-snowflake-load', 'database') }}"

or explicit --ref-key and --ref-value optional arguments that compose the ref ninja string:

meltano config target-snowflake set db_name --ref-key 'inventory.prod-snowflake-load' --ref-value 'database'

[MeltyBot]

View 5 previous comments from the original issue on GitLab

[tayloramurphy]

We have a user interested in this:

• https://app.hubspot.com/contacts/20712484/company/8160896903