[CT-2562] [Feature] support "depend on macro" selector in --select

[tema-popov]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

In case of updating the macro code, I want to select and run all models, that could depend on the macro. That includes

• models that use the specified macro directly
• models that reference models that use the specified macro directly
• models that use macros that depend on the specified macro through the graph of macros calls

something like
dbt ls|run --select macro:macro_name could work well for me.

Describe alternatives you've considered

Discussion about workaround https://discourse.getdbt.com/t/is-there-a-way-to-select-models-that-depend-on-specific-macro/8217/2

Who will this benefit?

Anybody, who supports a lot of models in a project where it is dangerous or expensive to run all models in case of code change and it is hard to find all dependent models from the specified macro where the code and logic were changed.

Are you interested in contributing this feature?

no

Anything else?

No response

[dbeatty10]

Thanks for opening @tema-popov -- clearly this a popular feature request across the developer team at FingerprintJS 😎

Could you share more detail about the potential use case and how it would benefit your team and others? Do you have a workaround that you are using today?

[tema-popov]

Could you share more detail about the potential use case

Sure! If I change a model and want to re-run all models that depend on it I have a selector for that (I believe it is dbt run --select @some_changed_model)
Macros are one of the most used features by our team as it allows us to avoid SQL code duplication.
If I change a macro that could be used in other macros and models, I don't have any clear way to select all models that depend on the macro code so I could re-run only models, that will change after the code change. As the dependencies could be quite unexpected in a large dbt project, I could forget to update models that depend on the changed code.

and how it would benefit your team and others?

• Less manual work and potential for making mistakes during dbt project code updates and its deployment. No
• Another case is to use it in a changelog generation for all affected models by code diff so we could find pull requests that could change some model behavior in an unexpected way.

Do you have a workaround that you are using today?`

Suppose, I changed the macro that was used in 2 other macros and several models. And all these macros and models also have models that depend on them. Right now I will just look through the code to see what models depend on changed macros, which requires manual work.

Other options I have:

• run all models that are safe to run (some of the models require dynamic parameters from variables or iterative, so I can't really run them every time. Perhaps it is expected to run all models on every code update as a general pattern, but I try to run only potentially affected models.
  It could work if I tag all "unsafe" to run on every code update model with a special tag and exclude them and if I'm okay with time spent or re-running all models.

[dbeatty10]

Thanks for that background @tema-popov !

Did you try state-based selection, by any chance? See relevant caveats for state-based selection here.

If you have tried it before, will be curious how it did or didn't meet your needs.

If not, I'll provide a short example below.

Example using the "state" selector method

1. Create these two files in your dbt project:

models/my_model.sql

select {{ guid() }} as id

macros/guid.sql

{% macro guid() %}
    {{ return("'1'") }}
{% endmacro %}

2. Run all the models:

dbt run

3. Save a copy of the manifest for later:

cp target/manifest.json .

4. Only run any models that are categorized as being modified versus the last saved state (which should be none):

dbt run -s state:modified --state .

5. Modify the macro:
   macros/guid.sql

{% macro guid() %}
    {{ return("'A'") }}
{% endmacro %}

6. Re-run any models that are categorized as being modified versus the last saved state (which should be one):

dbt run -s state:modified --state .

[tema-popov]

Oh, great, I will definitely try the state selector, it seems to do, what I want from macro selectors. It is a bummer it was hard for me to new about it as https://docs.getdbt.com/reference/node-selection/syntax don't have any clear reference to new selectors added in 1.5

[dbeatty10]

We'll be interested to hear how the state selector works for you!

With the belief that it will do what you want, I'm going to close this issue as "not planned". But feel free to reach out if you think we should reconsider.

Docs for state-based selection

Thank you for mentioning that this feature was hard to know about. We definitely want you to know about all the tools at your disposal! 🔧

The state-based selector was added in 0.18.0, but many people either don't know about it or haven't used it before.

If you have any suggestions to improve our documentation, we would welcome the feedback. There's two ways to do so:

1. Propose an edit
2. Open a docs issue

Propose an edit

If you come across a page that lacks clear references or information about relevant features, you can help by submitting proposed edits. To do this, simply click on the "edit this page" link located near the bottom of every documentation page. This will allow you to suggest changes or additions to the content.

Thank you for bringing this up, and please let me know if there's anything else I can assist you with!