Skip to content

Comments

[BEAM-4135] Fork the DirectRunner Engine#5225

Merged
tgroh merged 2 commits intoapache:masterfrom
tgroh:fork_engine
Apr 26, 2018
Merged

[BEAM-4135] Fork the DirectRunner Engine#5225
tgroh merged 2 commits intoapache:masterfrom
tgroh:fork_engine

Conversation

@tgroh
Copy link
Member

@tgroh tgroh commented Apr 25, 2018

For development velocity, the DirectRunner's two implementations can share a
large number of underlying implementations, but due to limitations on intermediate
data type representations and the depth of the dependency chains, partial migrations
end with extremely messy generic types. Forking the two implementations in order
to minimize the need to modify the Java-SDK specific implementations of the DirectRunner
is likely to significantly simplify initial development efforts, with the ability to make
future components which maintain a high degree of complexity generic and shared between
the two processing implementations.

This change is a fork + prune, removing as many classes as seems sensible. Follow-up
changes will migrate Java SDK types (AppliedPTransform and PCollection being the two
primary types) to the corresponding portable representations (PTransformNode and PCollectionNode).

Some of the forked utilities (e.g. WatermarkManager) should be deduplicated,
as they are sufficiently generic at their current state. This will occur in a follow-up change.


Follow this checklist to help us incorporate your contribution quickly and easily:

  • Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
  • Write a pull request description that is detailed enough to understand:
    • What the pull request does
    • Why it does it
    • How it does it
    • Why this approach
  • Each commit in the pull request should have a meaningful subject line and body.
  • Run ./gradlew build to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

@tgroh
Copy link
Member Author

tgroh commented Apr 25, 2018

R: @youngoli

Copy link
Contributor

@youngoli youngoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed changes between forked files and non-forked files. Some files had further changes than just forking, which I'll sum up here for anyone else reading this:

DirectTransformExecutor.java - Removes usages of ModelEnforcement.
EvaluationContext.java - Changes usages of DirectGraph to ExecutableGraph. Removes side input code.
ExecutorServiceParallelExecutor.java - Changes usages of DirectGraph to ExecutableGraph. Removes usages of ModelEnforcement.
PipelineExecutor.java - Changes usages of DirectGraph to ExecutableGraph.
QuiescenceDriver.java - Changes usages of DirectGraph to ExecutableGraph.
RootProviderRegistry.java - Removes the function for returning a native Java registry.
TransformEvaluatorRegistry.java - Removes code used for creating a native Java SDK registry.
WatermarkManager.java - Changes usages of DirectGraph to ExecutableGraph.

All the changes look good to me.

*/
package org.apache.beam.runners.direct.portable;

import com.google.common.annotations.VisibleForTesting; import java.io.Closeable;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit: Two imports on same line.

Remove currently unsupported implementations.

Update types to more generic when obviously appropriate.
@tgroh tgroh merged commit e86cf60 into apache:master Apr 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants