[FEATURE] Make use of meta config when defining model metadata

[alexjreilly]

Make use of the meta dictionary either in the model config block or properties (.yml) file(s) when defining the model metadata. (e.g. src_pk etc.).
I think this would be cleaner and could improve readability of the models.

Docs for reference.

AB#5376

[DVAlexHiggs]

This is a great request and something we are planning to experiment with.

We want to ensure a smooth transition to this feature and we're working out whether this would be replacement functionality or whether it would work alongside macro parameters (and then whether that would be a good idea!)

We originally suggested this change here so we're excited for the possibilities for this.

Thanks again!

[alexjreilly]

Perhaps the simplest solution with minimal impact to users might be to set default values for macro arguments, taking those defined in config. Something like:

{%- macro hub(
    src_pk=config.get('meta')['src_pk'], 
    src_nk=config.get('meta')['src_nk'], 
    src_ldts=config.get('meta')['src_ldts'], 
    src_source=config.get('meta')['src_source'], 
    source_model=config.get('meta')['source_model']
) -%}

    {{- adapter.dispatch('hub', 'dbtvault')(src_pk=src_pk, src_nk=src_nk,
                                            src_ldts=src_ldts, src_source=src_source,
                                            source_model=source_model) -}}

{%- endmacro -%}

This implementation should effectively be backward compatible.
This could potentially allow you to define your metadata in the properties files and your .sql file could resolve to a single macro call with no parameters passed to it.

schema.yml

version: 2

models:
  - name: my_hub
    description: "lorem ipsum"
    meta:
        src_pk: foo
        src_nk: bar
        src_ldts: baz
        src_source: my_source
        src_model: my_model

my_hub.sql

{{ dbtvault.hub() }}

[DVAlexHiggs]

Yes this was the kind of thing we had in mind! Just hadn't had a chance to play around with it yet. Thanks for this :)

[DVAlexHiggs]

Hi! I did some investigation on this yesterday, and discovered it might not be possible yet. I've submitted an issue to dbt here. Essentially meta isn't available to the model context at compile time, i.e. the values are undefined/empty, so we cannot us them to generate SQL currently.

[alexjreilly]

@DVAlexHiggs interesting, thanks for looking at this. And thanks for raising the ticket in dbt-core.

I suspect this might be an issue related to the timing of when dbt parses the properties file(s). The reason I say this is because I've created macros that behave in a similar way, but where the config options are specified in the config block of the model, they work as expected. Do you know what the precedence is for model parsing?

Are you having the same problem if you define meta in the config block?

[DVAlexHiggs]

I think I did try this. Will clarify. It would be better if it would work either way, however.

[DVAlexHiggs]

No unfortunately that does not work either.

{{ config(materialized='table', meta={'param_1': 'COLUMN_1', 
                                     'param_2': 'COLUMN_2', 
                                     'param_3': 'COLUMN_3', 
                                     'param_4': 'COLUMN_4', 
                                     'source_model_name': 'stg_orders'}
) }}

{% do log('meta: ' ~ config.get('meta'), true) %}
{% do log('param_1: ' ~ config.get('meta')['param_1'], true) %}
{% do log('param_2: ' ~ config.get('meta')['param_2'], true) %}
{% do log('param_3: ' ~ config.get('meta')['param_3'], true) %}
{% do log('param_4: ' ~ config.get('meta')['param_4'], true) %}
{% do log('source_model_name: ' ~ config.get('meta')['source_model_name'], true) %}

{% set param_1 = config.get('meta')['param_1'] -%}
{% set param_2 = config.get('meta')['param_2'] -%}
{% set param_3 = config.get('meta')['param_3'] -%}
{% set param_4 = config.get('meta')['param_4'] -%}
{% set source_model_name = config.get('meta')['source_model_name'] -%}

SELECT 1 AS "{{ param_1 }}"
       , 2 AS "{{ param_2 }}"
       , 3 AS "{{ param_3 }}"
       , 4 AS "{{ param_4 }}"

FROM {{ ref(source_model_name) }}

12:26:07  Running with dbt=1.0.3
12:26:07  Unable to do partial parsing because a project config has changed
12:26:09  meta: 
12:26:09  param_1: 
12:26:09  param_2: 
12:26:09  param_3: 
12:26:09  param_4: 
12:26:09  source_model_name: 
12:26:10  Encountered an error:
Compilation Error in model my_meta_table (models/my_meta_table.sql)
  The name argument to ref() must be a string, got <class 'dbt.clients.jinja.create_undefined.<locals>.Undefined'>

[DVAlexHiggs]

Minor update @alexjreilly

dbt-labs/dbt-core#4776 (comment)