[BEAM-380] Remove Spark runner dependency on beam-examlpes-java #539

amitsela · 2016-06-27T18:05:42Z

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Duplicate WordCount into spark examlpes package.

Duplicate parts of TfIdf from beam examlpes.

Better reuse of WordCount and its parts.

Remove dependency on beam-examples-java

amitsela · 2016-06-27T18:08:17Z

Following @peihe's comment in #516 this PR removes the dependency of the Spark runner on beam-examples-java.
R: @peihe @davorbonaci

peihe · 2016-06-27T18:38:22Z

runners/spark/src/test/java/org/apache/beam/runners/spark/io/NumShardsTest.java

@@ -21,10 +21,10 @@
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertTrue;

-import org.apache.beam.examples.WordCount;


It tests TextIO numShards.
Does it have to use WordCount?

No it doesn't, it's just been like that... Not a good answer 😄 but I'm definitely prioritising.

peihe · 2016-06-27T18:44:10Z

FYI
I am also working towards removing the examples dependency with a different approach in here:
#533 (need to fix the failed build)

This will avoid duplicating codes.

amitsela · 2016-06-27T19:34:25Z

First of all, I'm sorry if I "jumped in", I was looking for an open JIRA because I knew you had this in mind, but when I couldn't find one I opened it and submitted this PR against it.
I don't mind your approach, but the Spark runner needs, for now, it's own WordCount example because it doesn't pass validation when trying to read from HDFS.. probably because the runner still lacks Read.from implementation and IOChannelFactory only has implementations for file/gs.

peihe · 2016-06-27T20:51:14Z

My fault. I should open a JIRA issue before I start on this. I thought it is a quick change.
For the WordCount part, I like to move it to a top level class, and document why Spark runners need its own copy. (+@dhalperi for Read.from doesn't work for HDFS)
For the Tfidf part, I think I am close to run it with Spark runner in beam-examples module. #533
I am having a spark dependency problem in the test (JavaSparkContext class not found):
https://builds.apache.org/job/beam_PreCommit_MavenVerify/1908/console
Could you help me to take a look? And, you can make a call on whether you want to include the Tfidf in this PR or left to me to handle in #533.

amitsela · 2016-06-27T21:00:19Z

About the ClassNotFoundException, it's probably because the Spark runner has a provided dependency on Spark, which works in test, but is not transitive, so the examples module will have to have a provided dependency as well.. Which kinda sucks.
Let me think about it...
See: https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html provided dependencies are just intransitive.

amitsela · 2016-06-28T07:32:39Z

I honestly closed this by accident.. Though I'd be happy to have Jenkins try again since it failed on JavaDoc issues that, while might be valid, are unrelated to this PR.

amitsela · 2016-06-28T07:56:31Z

Anyone got an idea why Jenkins suddenly fails on JavaDoc issues in classes that aren't related to this PR ?

kennknowles · 2016-07-01T02:32:01Z

I was just browsing the output and didn't even see what the error was. FWIW the travis errors are different but also seem unrelated. I've seen the Travis issue before but have forgotten what it was about.

dhalperi · 2016-07-01T04:04:49Z

Hi Amit,

The logs are unfortunately a bit misleading, but there is one Javadoc error:

/home/jenkins/jenkins-slave/workspace/beam_PreCommit_MavenVerify/runners/spark/src/main/java/org/apache/
beam/runners/spark/examples/WordCount.java:38: error: reference not found
 * Duplicated from {@link org.apache.beam.examples.WordCount} to avoid dependency on

The link is not resolved because you removed the dependency on the examples module.

All the other Javadoc stuff is warnings and don't cause breaks.

dhalperi · 2016-07-01T04:07:20Z

Okay, I did a little more digging and here's my summary:

Jenkins - legit Javadoc error.
Travis - broken because the Travis part of the build was broken for a little bit.

A rebase should fix the latter.

Duplicate WordCount into spark examlpes package. Duplicate parts of TfIdf from beam examlpes. Better reuse of WordCount and its parts. Remove dependency on beam-examples-java

iemejia · 2016-07-19T15:42:42Z

Hello, is there any blocking reason why this has not been merged ?
This will make the spark runner lighter and also from an architecture point of view it makes sense too.

dhalperi · 2016-07-23T03:13:50Z

LGTM

peihe reviewed Jun 27, 2016
View reviewed changes

amitsela closed this Jun 28, 2016

amitsela reopened this Jun 28, 2016

amitsela mentioned this pull request Jun 28, 2016

Make beam-examples run with spark runner #533

Closed

Sela added 2 commits July 4, 2016 15:35

Remove dependency on beam-examples-java.

e2f20ce

Duplicate WordCount into spark examlpes package. Duplicate parts of TfIdf from beam examlpes. Better reuse of WordCount and its parts. Remove dependency on beam-examples-java

Remove JavaDoc links to beam-examples.

e957474

amitsela force-pushed the BEAM-380 branch from 2b6bbc7 to e957474 Compare July 4, 2016 12:45

kennknowles mentioned this pull request Jul 8, 2016

[BEAM-431] Move example dependency on runners into a profile #609

Merged

4 tasks

This was referenced Jul 20, 2016

[BEAM-362] Move aggregator scraping API onto Pipeline, make the support code private #699

Merged

[BEAM-124] Flink and Spark running Examples WordCountIT #703

Closed

asfgit closed this in cf14644 Jul 23, 2016

johnjcasey pushed a commit to johnjcasey/beam that referenced this pull request Feb 8, 2023

This closes apache#539

3604bc0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-380] Remove Spark runner dependency on beam-examlpes-java #539

[BEAM-380] Remove Spark runner dependency on beam-examlpes-java #539

amitsela commented Jun 27, 2016 •

edited

amitsela commented Jun 27, 2016

peihe Jun 27, 2016

amitsela Jun 27, 2016

peihe commented Jun 27, 2016

amitsela commented Jun 27, 2016

peihe commented Jun 27, 2016

amitsela commented Jun 27, 2016

amitsela commented Jun 28, 2016

amitsela commented Jun 28, 2016

kennknowles commented Jul 1, 2016

dhalperi commented Jul 1, 2016 •

edited

dhalperi commented Jul 1, 2016 •

edited

iemejia commented Jul 19, 2016

dhalperi commented Jul 23, 2016

[BEAM-380] Remove Spark runner dependency on beam-examlpes-java #539

[BEAM-380] Remove Spark runner dependency on beam-examlpes-java #539

Conversation

amitsela commented Jun 27, 2016 • edited

amitsela commented Jun 27, 2016

peihe Jun 27, 2016

Choose a reason for hiding this comment

amitsela Jun 27, 2016

Choose a reason for hiding this comment

peihe commented Jun 27, 2016

amitsela commented Jun 27, 2016

peihe commented Jun 27, 2016

amitsela commented Jun 27, 2016

amitsela commented Jun 28, 2016

amitsela commented Jun 28, 2016

kennknowles commented Jul 1, 2016

dhalperi commented Jul 1, 2016 • edited

dhalperi commented Jul 1, 2016 • edited

iemejia commented Jul 19, 2016

dhalperi commented Jul 23, 2016

amitsela commented Jun 27, 2016 •

edited

dhalperi commented Jul 1, 2016 •

edited

dhalperi commented Jul 1, 2016 •

edited