Improve must_consolidate
setting in single-time dataflow refinement
#18732
Labels
must_consolidate
setting in single-time dataflow refinement
#18732
As observed in #18546 (comment), there is an opportunity to set the
must_consolidate
field of monotonic operators tofalse
on single-time plans where the input to the operator is already consolidated upstream. For example, if the input given to a monotonic top-k operator originates from a reduce operator, then we know that consolidation in the monotonic operator is unnecessary. The reduce operator already arranges its output, implicitly consolidating it; the latter results in a monotonic stream in a single-time context. Similarly, inputs originating from indexes orArrangeBy
nodes would become monotonic in a single-time context, and remain so if other intermediate operators do not break monotonicity.This issue is a request to revisit the single-time dataflow refinement in
finalize_dataflow
that is introduced by #18546 and introduce additional reasoning to selectively setmust_consolidate
tofalse
whenever possible, as explained above.The text was updated successfully, but these errors were encountered: