[DS][36/n] Create ScheduledSince condition #21712
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced May 7, 2024
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
77c8447
to
bcdc712
Compare
This was referenced May 7, 2024
OwenKephart
force-pushed
the
05-07-Create_ChildCondition_condition
branch
from
594d77c
to
88dce77
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
bcdc712
to
d6b9e0e
Compare
OwenKephart
force-pushed
the
05-07-Create_ChildCondition_condition
branch
from
88dce77
to
bde7a62
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
d6b9e0e
to
0d35118
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
0fd4a9d
to
1bbbd9a
Compare
OwenKephart
changed the base branch from
05-07-Create_ChildCondition_condition
to
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
1bbbd9a
to
3174f9a
Compare
OwenKephart
force-pushed
the
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
branch
from
4946a2f
to
eb565e8
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
3174f9a
to
6dae5df
Compare
OwenKephart
changed the title
OwenKephart
force-pushed
the
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
branch
from
eb565e8
to
b9a986c
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
6dae5df
to
c9aee95
Compare
OwenKephart
force-pushed
the
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
branch
from
b9a986c
to
97839e1
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
c9aee95
to
8ede4ae
Compare
schrockn
approved these changes
May 14, 2024
Comment on lines
+65
to
+70
| slices_with_metadata = [ | ||
| AssetSliceWithMetadata( | ||
| asset_slice.compute_difference(context.previous_requested_slice), metadata | ||
| ) | ||
| for asset_slice, metadata in previous_slices_with_metadata | ||
| ] |
There was a problem hiding this comment.
This definitely speaks to me to the the need of a generalized abstraction encapsulate a set of slices with metadata we can operate. I was calling this a "space" in early prototypes which wasn't quire right
but
space.difference_with(context.previous_requested_slice)
would be very nice.
Comment on lines
+85
to
+98
| return [ | ||
| asset_slice_with_metadata | ||
| for asset_slice_with_metadata in slices_with_metadata | ||
| if not ( | ||
| asset_slice_with_metadata.asset_slice.is_empty | ||
| or cast( | ||
| float, | ||
| asset_slice_with_metadata.metadata.get( | ||
| _REQUEST_TIMESTAMP_METADATA_KEY, MetadataValue.float(0) | ||
| ).value, | ||
| ) | ||
| < self._get_minimum_timestamp(context) | ||
| ) | ||
| ] |
There was a problem hiding this comment.
Similarly here this feels like it should be a primitive on a slightly higher level of abstraction.
OwenKephart
force-pushed
the
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
branch
from
97839e1
to
fc4c2a0
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
8ede4ae
to
5823a56
Compare
OwenKephart
force-pushed
the
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
branch
from
fc4c2a0
to
a5d3f72
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
5823a56
to
e704541
Compare
OwenKephart
force-pushed
the
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
branch
from
a5d3f72
to
407cd58
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
e704541
to
91e489e
Compare
OwenKephart
force-pushed
the
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
branch
from
407cd58
to
b6d8453
Compare
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
91e489e
to
bd76584
Compare
Merge activity
|
OwenKephart
force-pushed
the
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
branch
from
b6d8453
to
e374d83
Compare
Base automatically changed from
05-10-Replace_asset_subsets_with_metadata_with_asset_slices_with_metadata
to
master
May 14, 2024 15:04
OwenKephart
force-pushed
the
05-07-Create_RecentlyRequested_condition
branch
from
bd76584
to
55b9187
Compare
danielgafni
pushed a commit
to danielgafni/dagster
that referenced
this pull request
Jun 18, 2024
## Summary & Motivation Creates a condition which determines if an asset partition has been requested within some lookback period. It does this by keeping track of a set of (essentially) tuples of (AssetSlice, last_requested_timestamp). These tuples are evicted from the list whenever they age out of the lookback window. It uses asset_subsets_with_metadata instead of the more generic extra_state field because asset_subsets_with_metadata get rendered in the UI. this means that users will be able to see exactly when an asset partition was last requested (if it was requested within the given lookback period). ## How I Tested These Changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Creates a condition which determines if an asset partition has been requested within some lookback period.
It does this by keeping track of a set of (essentially) tuples of (AssetSlice, last_requested_timestamp). These tuples are evicted from the list whenever they age out of the lookback window.
It uses asset_subsets_with_metadata instead of the more generic extra_state field because asset_subsets_with_metadata get rendered in the UI. this means that users will be able to see exactly when an asset partition was last requested (if it was requested within the given lookback period).
How I Tested These Changes