-
Notifications
You must be signed in to change notification settings - Fork 459
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
compute: tokenize linear join closure
This commit adds a shutdown token check to the closure we pass to the differential join operator. When the dataflow is shutting down, this makes the join closure drain all input data, rather than processing it. As a result, differential join operators shut down faster an emit less data, which in turn speeds up shutdown of downstream operators. Unfortunately, the new shutdown logic interferes with the fueling of the differential join operator. Fuel is consumed based on the number of updates emitted. When the token is dropped, the join closure stops producing updates, which means the operator stops consuming fuel, so it does not yield anymore until it has drained all its inputs. If there are many inputs left, the replica may not accept commands for potentially quite a long time.
- Loading branch information
Showing
1 changed file
with
16 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters