[BEAM-11] Integrate Spark runner with Beam #42
Conversation
…be replaced by a SparkStateInternals implementation
|
I'll take a peek at this one shortly. R: @davorbonaci |
| </dependency> | ||
| <dependency> | ||
| <groupId>com.google.cloud.dataflow</groupId> | ||
| <artifactId>google-cloud-dataflow-java-examples-all</artifactId> |
There was a problem hiding this comment.
It sounds fair to me to prefer the runners logger.. Flink runner does the same.
There was a problem hiding this comment.
The dependency is needed because some of the examples are used to test the Spark runner.
|
LGTM Nice! I think we'll have to go over all |
|
(We should get to the bottom of the Jenkins failure before merging.) |
|
R: @tomwhite as well |
| <!--<transformers>--> | ||
| <!--<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />--> | ||
| <!--</transformers>--> | ||
| <!--</configuration>--> |
There was a problem hiding this comment.
Guava will still need to be relocated to run properly on a cluster, won't it?
There was a problem hiding this comment.
The SDK upgraded to Guava 19 but I guess shading is still necessary for cluster. I'll reinstate the shade configuration.
|
Looks good to me. Thanks for working on it @amitsela. A few comments inline and here:
|
|
Also, the note about the Spark runner on https://github.com/apache/incubator-beam#runners should be updated to say that it's now a part of Beam. |
|
Thanks @tomwhite and @davorbonaci ! I plan to address the following:
@davorbonaci like you said, I think that we need a cross-project pom.xml work to get all components inline, but let's get this runner running first :) |
|
This pull request is till pending additional work so please DON'T MERGE. Thanks! |
|
@tomwhite please review second iteration. Thanks. |
|
+1 from me |
Wire job service API into portable runner PipelineResults
chore: complete URN -> dyn Coder codes
No description provided.