[BEAM-22] Clean up InProcess Read Evaluators #106
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
R: @bjchambers |
| UnboundedReadEvaluator<OutputT> evaluator = | ||
| new UnboundedReadEvaluator<OutputT>(transform, evaluationContext, evaluatorQueue); | ||
| new UnboundedReadEvaluator<OutputT>( | ||
| transform, evaluationContext, evaluatorQueue, source); |
There was a problem hiding this comment.
Argument order should be the same between these methods (eg., transform, source, context, ...).
|
LGTM. Will wait for green CI before commit. |
tgroh
force-pushed
the
ippr_cleaner_read_evaluators
branch
from
b6ba797
to
71e95b7
Compare
| @@ -117,12 +117,19 @@ | |||
| private final AppliedPTransform<?, PCollection<OutputT>, Bounded<OutputT>> transform; | |||
| private final InProcessEvaluationContext evaluationContext; | |||
| private boolean contentsRemaining; | |||
| /** | |||
| * The source being read from by this {@link BoundedReadEvaluator}. This may not be the same | |||
| * source as returned by {@link Bounded#getSource()} within {@link #transform} due to splitting. | |||
There was a problem hiding this comment.
Maybe shorten:
The source being read from by this {@link BoundedReadEvaluator}.
This may differ from the initial source derived from the {@link #tranfsorm} due to splitting.
(Documenting the path by which it is derived seems brittle to future changes)
This permits use of sources that are not the initial source used in the transform. BoundedSource#splitIntoBundles and UnboundedSource#generateInitialSplits generate multiple source objects for the same transform in order to permit parallelism.
Use BoundedReader instead of Reader. contentsRemaining should be method-scoped not instance-scoped.
tgroh
force-pushed
the
ippr_cleaner_read_evaluators
branch
from
71e95b7
to
a8f862d
Compare
guanjun1
pushed a commit
to guanjun1/beam
that referenced
this pull request
Aug 28, 2023
LISAMZA-31528: Add top level module for improving Beam auto ELR stability and productivity
pl04351820
pushed a commit
to pl04351820/beam
that referenced
this pull request
Dec 20, 2023
* refactor: drop six package use * fix conflicts * fix conflicts * fix conflicts * fix conflicts
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These are a couple of minor improvements to BoundedReadEvaluator
and UnboundedReadEvaluator that enables splitting a source at
evaluation time, as well as minor code cleanup.