[DS][31/n] Create static constructor for eager condition

[OwenKephart]

Summary & Motivation

This creates a static constructor for an analog to the current-day eager condition. It is not identical, as it does not consider historical time partitions. It also manages state differently.

In the current world, the "parent_newer" analog actually encodes a lot of different information. From its perspective, an asset partition is True if:

A parent has been materialized more recently than the asset, and the run that materialized it does not target this asset,
AND this asset hasn't been requested for materialization by AMP since that run came in

This is a pretty confusing thing to wrap your head around, but it does have two good qualities:

it avoids repeatedly requesting the downstream asset for materialization while the asset computes (techincally, the parent will remain newer than the child until the child successfully materializes, which may in practice span a large number of scheduling evaluations)
it avoids repeatedly materializing the asset in the case that the requested materialization failed (similar to the above issue)
This policy handles issue (1) by explicitly avoiding materializing this asset if a run is currently in progress, but does not handle case (2).

This will need to be handled with a recently_requested condition, which can keep track of the asset partitions which have been requested within some timedelta.

How I Tested These Changes

[OwenKephart]

• [DS][40/n] Create an as_auto_materialize_policy method on SchedulingCondition #21788
• [DS][39/n] Use FailedSchedulingCondition and ScheduledSince condition in eager policy #21741
• [DS][38/n] Create FailedSchedulingCondition #21740
• [DS][37/n] Reorganize AssetGraphView methods #21739
• [DS][36/n] Create ScheduledSince condition #21712
• [DS][35/n] Replace asset_subsets_with_metadata with asset_slices_with_metadata #21793
• [DS][34/n] Remove some AssetCondition references #21704
• [DS][33/n] Create static constructor for cron condition #21703
• [DS][32/n] Update UpdatedSinceCronCondition #21677
• [DS][31/n] Create static constructor for eager condition #21737 👈
• [DS][30/n] Create new unique ids for each child based off of index #21671 : 1 other dependent PR (#21705 )
• [DS][29/n] Create ParentNewerCondition #21641 : 1 other dependent PR (#21670 )
• [DS][28/n] Create WillBeRequestedCondition #21640
• [DS][27/n] Make candidate_slice and true_slice Optional #21648
• [DS][26/n] Update internal representation of InLatestTimeWindow #21615
• [DS][25/n] Narrow extra_state type #21613
• [DS][24/n] Remove all AssetSubset references from the context and result objects #21612
• [DS][23/n] Update SchedulingContext object #21573
• [DS][22/n] Convert SchedulingResult to DagsterModel #21546
• [DS][21/n] Rename AssetConditionResult to SchedulingResult #21545
• [DS][20/n] Port static constructors from AssetCondition to SchedulingCondition #21541
• [DS][19/n] Create SchedulingCondition object to replace AssetCondition #21540
• [DS][18/n] Move serialized objects into separate file #21539
• [DS][17/n] Move legacy files to separate legacy folder #21538
• [DS][16/n] Rename AssetConditionEvaluationContext to LegacyRuleEvaluationContext #21537
• [DS][15/n] Move operators to separate folder #21536
• [DS][14/n] Remove unused methods and properties #21535
• [DS][13/n] Convert SchedulingContext to DagsterModel #21521
• [DS][12/n] Rename SchedulingContext object #21520
• [DS][11/n] Create InProgressCondition #21511 : 1 other dependent PR (#21512 )
• [DS][10/n] Create UpdatedSinceCronCondition #21510
• [DS][9/n] Add lookback_timedelta to InLatestTimeWindow condition #21508
• [DS][8/n] Create InLatestTimeWindow condition #21507
• [DS][7/n] Create MaterializedSchedulingCondition #21505
• [DS][6/n] Create Any/AllDepsMatchSchedulingConditions #21504
• [DS][5/n] Use AssetSlice in AssetConditionResult #21503
• [DS][4/n] Rename asset_condition folder and add README #21502
• [DS][3/n] Update unique_id creation logic #21501
• [DS][2/n] Update signature of AutoMaterializeRule #21500
• [DS][1/n] Create SchedulingConditionEvaluationContext #21499
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @OwenKephart and the rest of your teammates on Graphite

[schrockn]

Super exciting to see this. Just want to make sure I understand 100% our default policy here and have it in the docblock

[schrockn]

Please add test that ensures this actually constructs.

All I am looking for is

assert isinstance(SchedulingCondition.eager(), SchedulingCondition)

[OwenKephart]

a test gets added upstack, which also tests the behavior of the composite condition

[OwenKephart]

Merge activity

• May 14, 10:38 AM EDT: @OwenKephart started a stack merge that includes this pull request via Graphite.
• May 14, 10:53 AM EDT: Graphite rebased this pull request as part of a merge.
• May 14, 10:54 AM EDT: @OwenKephart merged this pull request with Graphite.