Data Lineage for Macro Based Dynamic Models

[amalkumar]

Describe the feature

A clear and concise description of what you want to happen.

Data Lineage for Macro based model basis the dynamic input instead of model file name.

Use case:
I have 2 models i.e. m1.sql and m2.sql, wherein m2 depends on m1.
Linear graph shows, m1->m2.

These two models are used for transformations for different entities eg. employee, department, employer etc. dynamically generating the transformation rules (SQL Statements), source and target using DBT Macros.

My Use case is, is there a way I can see the Lineage graph with materialization alias name instead of model file name.
For eg:
Current Lineage : m1 -> m2

Model m1.sql target materialization is table and alias is employee_source (passed via Macros)
Model m2.sql target materialization is table and alias is employee_target (passed via Macros)

Expected Lineage : employee_source -> employee_target.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

none known

Additional context

Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here.
Not specific to any database

Who will this benefit?

What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly.

This will enable to use the macro based models (re-usable models) without the cost of loosing the model level lineage

Are you interested in contributing this feature?

Let us know if you want to write some code, and how we can help.

[JC-Lightfold]

Hey @amalkumar this is potentially interesting - in your usecase above, are you saying that aliasing the name of the model as it appears in the DAG would be sufficient? Or are you saying that M1 and M2 are dynamically utilised to create materialisations many times with different names and you need to track those materialisations as dependancies?

[ThomasRolfsnes-EP]

@JC-Lightfold Hi! We're also interested in this feature. We have a use-case where we create a version of all fact/dim tables with norwegian column names (for BI purposes). I have macro that does this automatically, but it would be really nice to also track lineage here. Ex: dim_customer -> dim_customer_NO.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[ivoalbrecht]

I am interested in this as well. I am using a macro to union all models in a certain folder (folder X).

This however breaks the lineage of the model, which makes it not dependent on the models in folder X.

Any suggestions?

[mjr267]

ditto - i am trying to script out a lot of the boiler plate joins/unions that need to happen through the use of a macro and am losing the lineage aspect.