Copy operatorSummaries before merging stats with running drivers #773
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PipelineContext contains list of running Driver instances.
All fields in PipelineContext are thread safe. However
getPipelineStats and driverFinished methods modify/read
those fields multiple times, therefore they are racy.
This causes PipelineStats to be consistent only when
there are no running drivers.
When driver is finished its OperatorStats stats are
accumulated in PipelineContext map. Because of that
in previous code version getPipelineStats could double
account for DriverStats (once from accumulated fields
and once from a copy of list of running drivers).
In this PR code is changed so that PipelineContext
can only underaccount for finished driver stats.
This is more natural and less confusing as cumulative OperatorStats
of a pipeline do not exceed accumulated PipelineStats themselves.