[CT-1303] Respect node selection in catalog queries run by docs generate

[jtcohen6]

When users specify dbt docs generate --select <selection_criteria>, that limits the set of project resources for which dbt will generate compile SQL. This is implicit behavior in the GenerateTask, because it inherits from the CompileTask, where that behavior is defined.

We should additionally make docs generate respect node selection for the purposes of generating catalogs, which can be very very expensive.

To really have an effect, this change would want to be paired with #4997, which would filter down our catalog queries to only the ones that are actually selected. Without that change, while we would successfully limit ourselves to cataloging metadata on only the schema(s) containing the selected resource(s), we'd still catalog the entire schema, which could include hundreds/thousands of unrelated resources.

Use case

• I'm working in a project with hundreds/thousands of models
• I just made changes to one model, e.g. by updating its SQL and adding descriptions
• I ran that one model: dbt run --select my_one_model
• Now I want to view documentation for just that one model: dbt docs generate --select my_one_model
• I understand that catalog info (columns, table stats, etc) will be missing for all other models

Technical implementation

• Pass in a set of relations (from get_node_selector, plus selected sources) into adapter.get_catalog here
• Instead of adapter._get_catalog_schemas accessing manifest.nodes and manifest.sources directly, it should limit itself to the nodes/sources actually included in the selection logic

Note

To be clear, this would not accomplish what users tend to ask for when asking for more powerful docs generate --select, which is to produce a manifest.json with only the selected resources. Relevant discussions: #5096, #5244

[scottsoithongsuk]

Thanks @jtcohen6 - the comment around not accomplishing what users tend to ask for (I'm in that boat!) - I've had a look at the discussions referenced, but wanted to ask if there's any indication on potential solution or timeline?
I'm looking for basically what's been released here, but to filter down the catalog.

[JavierLopezT]

Thanks @jtcohen6 for pointing me to this from #7813.

This is exactly what I am looking for :)

[kevinneville]

Thanks @jtcohen6 for pointing me to this from #7813.

This is exactly what I am looking for :)

Please upvote this to get higher priority of this :D

[dbeatty10]

In the meantime, here is an example override for default__get_catalog that I protyped for use with dbt-postgres (postgres__get_catalog ).

It includes a diff that can be used as a guide to make a similar override for other adapters.