Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-3050] [Applied State] Implement Core Changes for Filtered Catalog Queries #8521

Closed
Tracked by #8316
peterallenwebb opened this issue Aug 30, 2023 · 0 comments · Fixed by #8648
Closed
Tracked by #8316
Assignees
Labels
Impact: Adapters performance user docs [docs.getdbt.com] Needs better documentation

Comments

@peterallenwebb
Copy link
Contributor

peterallenwebb commented Aug 30, 2023

Short description

Building on some of the research done for #4997, we want to improve catalog creation performance by selecting only the needed relation information, instead of information for every relation in a schema.

Acceptance criteria

  • In dbt's base adapter implementation (base/impl.py), the _get_one_catalog method has been modified to accept an additional new parameter, relations_by_schema: Optional[Dict[str, Optional[List[BaseRelation]]]] = None, mapping schema names to the list of relations in that schema for which catalog information should be retrieved.
  • When the new parameter defaults to None, behavior should be the same as before.
  • When _get_one_catalog invokes the get_catalog macro, it should include the new parameter.
  • Adapters which do not yet implement the new get_catalog macro parameter should continue to work as before. Therefore we need to evaluate the macro without generating an error and we also need to continue post-filtering the results.
  • When the value of the relations parameter is None, the _get_one_catalog method, and get_catalog macros should continue to retrieve all relation information for every schema listed in the schema parameter, matching historical behavior.

Impact to Other Teams

The adapter teams will need to implement changes to the get_catalog macro in each adapter implementation, in order to take advantage of the new parameter:

@peterallenwebb peterallenwebb added the user docs [docs.getdbt.com] Needs better documentation label Aug 30, 2023
@peterallenwebb peterallenwebb self-assigned this Aug 30, 2023
@github-actions github-actions bot changed the title [Applied State] Implement Core Changes for Filtered Catalog Queries [CT-3050] [Applied State] Implement Core Changes for Filtered Catalog Queries Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Impact: Adapters performance user docs [docs.getdbt.com] Needs better documentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants