[DS][27/n] Make candidate_slice and true_slice Optional

[OwenKephart]

Summary & Motivation

This helps avoid a class of errors using the type system. We have these concepts of "AssetSubsets" vs. "ValidAssetSubsets", where valid asset subsets are known to have been created with the asset's current-tick PartitionsDefinition, and AssetSubsets have gone through a serdes layer and therefore may have a differen partitions definition than they did at serialization time.

This means that care needs to be taken when using a bare AssetSubset.

Previously, we had these fancy operators which would treat AssetSubsets which had invalid partitions_definitions as the empty subset, but that is not actually always accurate. i.e. you may have been keeping track of all the materialized partitions of an asset using a subset, then you update the partitions definition. The correct thing to do here is to recompute the materialized partitions from scratch, NOT assume that this susbet is equivalent to the empty subset.

By explicitly creating a None value for candidate_slice / true_slice in these cases, we force authors of conditions to handle these cases through the type system. This already uncovered a potential bug in the new UpdatedSinceCron condition, in which it would return an empty set in the case that a partition was updated since the latest cron tick, but then the partitions definition was updated.

How I Tested These Changes

[OwenKephart]

• [DS][29/n] Create ParentNewerCondition #21641 : 2 dependent PRs (#21670 , #21671 )
• [DS][28/n] Create WillBeRequestedCondition #21640
• [DS][27/n] Make candidate_slice and true_slice Optional #21648 👈
• [DS][26/n] Update internal representation of InLatestTimeWindow #21615
• [DS][25/n] Narrow extra_state type #21613
• [DS][24/n] Remove all AssetSubset references from the context and result objects #21612
• [DS][23/n] Update SchedulingContext object #21573
• [DS][22/n] Convert SchedulingResult to DagsterModel #21546
• [DS][21/n] Rename AssetConditionResult to SchedulingResult #21545
• [DS][20/n] Port static constructors from AssetCondition to SchedulingCondition #21541
• [DS][19/n] Create SchedulingCondition object to replace AssetCondition #21540
• [DS][18/n] Move serialized objects into separate file #21539
• [DS][17/n] Move legacy files to separate legacy folder #21538
• [DS][16/n] Rename AssetConditionEvaluationContext to LegacyRuleEvaluationContext #21537
• [DS][15/n] Move operators to separate folder #21536
• [DS][14/n] Remove unused methods and properties #21535
• [DS][13/n] Convert SchedulingContext to DagsterModel #21521
• [DS][12/n] Rename SchedulingContext object #21520
• [DS][11/n] Create InProgressCondition #21511 : 1 other dependent PR (#21512 )
• [DS][10/n] Create UpdatedSinceCronCondition #21510
• [DS][9/n] Add lookback_timedelta to InLatestTimeWindow condition #21508
• [DS][8/n] Create InLatestTimeWindow condition #21507
• [DS][7/n] Create MaterializedSchedulingCondition #21505
• [DS][6/n] Create Any/AllDepsMatchSchedulingConditions #21504
• [DS][5/n] Use AssetSlice in AssetConditionResult #21503
• [DS][4/n] Rename asset_condition folder and add README #21502
• [DS][3/n] Update unique_id creation logic #21501
• [DS][2/n] Update signature of AutoMaterializeRule #21500
• [DS][1/n] Create SchedulingConditionEvaluationContext #21499
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @OwenKephart and the rest of your teammates on Graphite

[OwenKephart]

Merge activity

• May 14, 10:38 AM EDT: @OwenKephart started a stack merge that includes this pull request via Graphite.
• May 14, 10:43 AM EDT: Graphite rebased this pull request as part of a merge.
• May 14, 10:44 AM EDT: @OwenKephart merged this pull request with Graphite.