Add inheritance to materialization macro resolution #5348
Conversation
There was a problem hiding this comment.
Thanks so much for picking this up @volkangurel!
Even though we're not achieving full consistency (between macro dispatch / materialization candidacy), as proposed in #4646, it is a step on the way there, and a big benefit to adapter developers in the meantime.
I've tagged this for review by both Team:Language + Team:Adapters, since it touches code relevant to both. Sign-off from one or the other is probably enough to see this in for v1.2.
| # order matters for dispatch: | ||
| # 1. current adapter | ||
| # 2. any parent adapters (dependencies) | ||
| # 3. 'default' | ||
| return get_adapter_type_names(adapter_type) + ["default"] |
There was a problem hiding this comment.
This order looks right to me. Thanks for the clear annotation!
Question: Would we be better served by incorporating actual dispatch logic here, rather than reimplementing it? I get that materialization macros are a special case (using a different / older naming structure), so there may not be a way to do this without expanding the scope of this PR significantly.
There was a problem hiding this comment.
Good question. I'm not very familiar with how providers work and how they can be injected into the manifest class instance here. But the approach in this PR doesn't work I'm happy to go down that path with some guidance from others.
There was a problem hiding this comment.
I'm fine with this approach. However, @volkangurel could you leave a comment here to this effect? Something that notes that this is duplicated logic and could instead be incorporating actual dispatch logic.
There was a problem hiding this comment.
Added a comment here.
jtcohen6
added
Team:Adapters
There was a problem hiding this comment.
Thanks for the work here, @volkangurel! @McKnight-42 and I just went though the changes, and it looks pretty good to us. We just want one additional set of tests, and some comments. Otherwise this looks good to merge.
| expected=(project, 'foo'), | ||
| ) for project in ['root', 'dep', 'dbt'] | ||
| ) | ||
|
|
There was a problem hiding this comment.
This additional set of test inputs looks good. I also want to check that if macros for both foo and bar are present, bar gets chosen. Maybe something like this:
sets.extend(
FindMaterializationSpec(
macros=[MockMaterialization(project, adapter_type='foo'), MockMaterialization(project, adapter_type='bar')],
adapter_type='bar',
expected=(project, 'bar'),
) for project in ['root', 'dep', 'dbt']
)
There was a problem hiding this comment.
Good catch! I added this test.
| # order matters for dispatch: | ||
| # 1. current adapter | ||
| # 2. any parent adapters (dependencies) | ||
| # 3. 'default' | ||
| return get_adapter_type_names(adapter_type) + ["default"] |
There was a problem hiding this comment.
I'm fine with this approach. However, @volkangurel could you leave a comment here to this effect? Something that notes that this is duplicated logic and could instead be incorporating actual dispatch logic.
| @@ -355,12 +350,10 @@ def __lt__(self, other: object) -> bool: | |||
|
|
|||
| @dataclass | |||
| class MaterializationCandidate(MacroCandidate): | |||
| specificity: Specificity | |||
| specificity: int | |||
There was a problem hiding this comment.
@volkangurel could you leave a comment on this parameter describing what the int value represents and how it should be set? The previous Specificity class had names that helped with this, but int isn't obvious how it is used by the MaterializationCandidate.
| def _materialization_candidates_for( | ||
| self, | ||
| project_name: str, | ||
| materialization_name: str, | ||
| adapter_type: Optional[str], | ||
| adapter_type: str, | ||
| specificity: int, |
There was a problem hiding this comment.
@volkangurel I might also leave that specificity comment here since this seems to be where it is passed in.
nathaniel-may
added
awaiting_response
and removed
ready_for_review
* Add inheritance to materialization macro resolution * Add changelog entry * Address PR feedback
addresses parts of #4646
Description
Adapter inheritance currently works for macro resolution but doesn't work for materialization macro resolution. This PR updates the logic for the materialization macro resolution to use inheritance as well.
Checklist
changie newto create a changelog entry