Parallel, Join and Window operators may have unexpected behavior: Messages possibly received _after_ final callback, causing “stalls” and missed messages

[AshleyF]

The issue is that there is no guarantee that final callback will be handled in upstream to downstream order, thus it’s possible for such components chained together to finalize and then see upstream messages generated. This can cause unexpected behavior as a pipeline is shutting down. More perniciously, the Parallel operator uses dynamic subpipelines internally which causes this (sub)pipeline shutdown behavior to potentially occur throughout the life of a running application.

Components registering to be called with Pipeline.RegisterPipelineFinalHandler(…) expect to be called only once no further upstream messages are forthcoming. Source components are stopped, scheduled messages throughout the system are drained, and then the final callbacks are invoked. However, components are free to post further messages as a result of these callbacks. For example,Join and the RelativeTimeWindow (exposed by the Window(…) operator) do this. Chaining these components together may cause issues due to the finalization ordering between them. Normally this issue manifests at pipeline shutdown, but Parallel and its use of subpipelines may exacerbate this into a steady state issue.

(This issue is currently being investigated)