[DS][24/n] Remove all AssetSubset references from the context and result objects #21612
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced May 2, 2024
OwenKephart
force-pushed
the
05-01-Update_SchedulingContext_object
branch
from
132764e
to
5744f34
Compare
OwenKephart
force-pushed
the
05-02-Remove_all_AssetSubset_references_from_the_context_and_result_objects
branch
from
b9355f4
to
6baf92a
Compare
OwenKephart
force-pushed
the
05-01-Update_SchedulingContext_object
branch
from
5744f34
to
8bc6892
Compare
OwenKephart
force-pushed
the
05-02-Remove_all_AssetSubset_references_from_the_context_and_result_objects
branch
from
6baf92a
to
e5e6ed8
Compare
schrockn
reviewed
May 3, 2024
Comment on lines
+167
to
+176
| def compute_union(self, other: "AssetSlice") -> "AssetSlice": | ||
| return _slice_from_subset( | ||
| self._asset_graph_view, self._compatible_subset | other.convert_to_valid_asset_subset() | ||
| ) | ||
|
|
||
| def compute_intersection(self, other: "AssetSlice") -> "AssetSlice": | ||
| return _slice_from_subset( | ||
| self._asset_graph_view, self._compatible_subset & other.convert_to_valid_asset_subset() | ||
| ) | ||
|
|
There was a problem hiding this comment.
hmmm i don't think we need to use the compute language here right? This is guaranteed to be fairly fast I think
There was a problem hiding this comment.
I think at the moment there's one case where things get a bit rough, which is if you have an in-memory time-window subset which only contains the time windows, not the partition keys themselves.
in order to compute the intersection, the current logic has to compute the partition keys for those time windows.
we could theoretically make this more efficient if you have time window to time window intersections (just intersect the time windows themselves), but I think there will always be the case where you have one subset that's encoded in terms of partition keys and another subset encoded with time windows, and it's non-negligible to convert one to the other
There was a problem hiding this comment.
Ah and then there's the especially-nasty case of multi-partitions definitions, which would require even more cleverness to make efficient in all cases.
There was a problem hiding this comment.
Got it. Very happy we are doing compute then instead of an operator that might just silently do that!
schrockn
reviewed
May 3, 2024
| @@ -433,7 +435,8 @@ def evaluate_for_asset(self, context: "SchedulingContext") -> "SchedulingResult" | |||
| ignore_subset=context.legacy_context.materialized_requested_or_discarded_since_previous_tick_subset, | |||
| ) | |||
| ) | |||
| return SchedulingResult.create(context, true_subset, subsets_with_metadata) | |||
| true_slice = context.asset_graph_view.get_asset_slice_from_subset(true_subset) | |||
| return SchedulingResult.create(context, true_slice, subsets_with_metadata) | |||
There was a problem hiding this comment.
gentle reminder that i do want to revist the true_slice verbiage
OwenKephart
force-pushed
the
05-01-Update_SchedulingContext_object
branch
from
8bc6892
to
7b1cf10
Compare
OwenKephart
force-pushed
the
05-02-Remove_all_AssetSubset_references_from_the_context_and_result_objects
branch
from
e5e6ed8
to
41e4314
Compare
schrockn
approved these changes
May 3, 2024
This was referenced May 3, 2024
OwenKephart
force-pushed
the
05-01-Update_SchedulingContext_object
branch
from
7b1cf10
to
45c84a3
Compare
OwenKephart
force-pushed
the
05-02-Remove_all_AssetSubset_references_from_the_context_and_result_objects
branch
from
41e4314
to
a03b80f
Compare
Merge activity
|
OwenKephart
force-pushed
the
05-01-Update_SchedulingContext_object
branch
from
45c84a3
to
9fffe56
Compare
OwenKephart
force-pushed
the
05-02-Remove_all_AssetSubset_references_from_the_context_and_result_objects
branch
from
a03b80f
to
2fb533a
Compare
OwenKephart
deleted the
05-02-Remove_all_AssetSubset_references_from_the_context_and_result_objects
branch
cmpadden
pushed a commit
that referenced
this pull request
May 6, 2024
…ult objects (#21612) ## Summary & Motivation As title -- for the new codepaths, helps avoid a lot of convert_to_valid_asset_subset() business. I didn't bother updating the AutoMaterializeRule side of the world in a nicer way, as this code will eventually be removed anyway. Had to add some new methods to replace the & / | / - operators, this time properly labeled with "compute_" ## How I Tested These Changes
This was referenced May 6, 2024
alangenfeld
reviewed
May 7, 2024
Comment on lines
+71
to
+72
| # construct is used here for performance | ||
| return SchedulingContext.construct( |
This was referenced May 7, 2024
danielgafni
pushed a commit
to danielgafni/dagster
that referenced
this pull request
Jun 18, 2024
…ult objects (dagster-io#21612) ## Summary & Motivation As title -- for the new codepaths, helps avoid a lot of convert_to_valid_asset_subset() business. I didn't bother updating the AutoMaterializeRule side of the world in a nicer way, as this code will eventually be removed anyway. Had to add some new methods to replace the & / | / - operators, this time properly labeled with "compute_" ## How I Tested These Changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
As title -- for the new codepaths, helps avoid a lot of convert_to_valid_asset_subset() business.
I didn't bother updating the AutoMaterializeRule side of the world in a nicer way, as this code will eventually be removed anyway.
Had to add some new methods to replace the & / | / - operators, this time properly labeled with "compute_"
How I Tested These Changes