[BEAM-4135] Fork the DirectRunner Engine#5225
Conversation
Cleanup to follow
|
R: @youngoli |
youngoli
left a comment
There was a problem hiding this comment.
Reviewed changes between forked files and non-forked files. Some files had further changes than just forking, which I'll sum up here for anyone else reading this:
DirectTransformExecutor.java - Removes usages of ModelEnforcement.
EvaluationContext.java - Changes usages of DirectGraph to ExecutableGraph. Removes side input code.
ExecutorServiceParallelExecutor.java - Changes usages of DirectGraph to ExecutableGraph. Removes usages of ModelEnforcement.
PipelineExecutor.java - Changes usages of DirectGraph to ExecutableGraph.
QuiescenceDriver.java - Changes usages of DirectGraph to ExecutableGraph.
RootProviderRegistry.java - Removes the function for returning a native Java registry.
TransformEvaluatorRegistry.java - Removes code used for creating a native Java SDK registry.
WatermarkManager.java - Changes usages of DirectGraph to ExecutableGraph.
All the changes look good to me.
| */ | ||
| package org.apache.beam.runners.direct.portable; | ||
|
|
||
| import com.google.common.annotations.VisibleForTesting; import java.io.Closeable; |
There was a problem hiding this comment.
Minor nit: Two imports on same line.
Remove currently unsupported implementations. Update types to more generic when obviously appropriate.
For development velocity, the DirectRunner's two implementations can share a
large number of underlying implementations, but due to limitations on intermediate
data type representations and the depth of the dependency chains, partial migrations
end with extremely messy generic types. Forking the two implementations in order
to minimize the need to modify the Java-SDK specific implementations of the DirectRunner
is likely to significantly simplify initial development efforts, with the ability to make
future components which maintain a high degree of complexity generic and shared between
the two processing implementations.
This change is a fork + prune, removing as many classes as seems sensible. Follow-up
changes will migrate Java SDK types (AppliedPTransform and PCollection being the two
primary types) to the corresponding portable representations (PTransformNode and PCollectionNode).
Some of the forked utilities (e.g. WatermarkManager) should be deduplicated,
as they are sufficiently generic at their current state. This will occur in a follow-up change.
Follow this checklist to help us incorporate your contribution quickly and easily:
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue../gradlew buildto make sure basic checks pass. A more thorough check will be performed on your pull request automatically.