Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPARK-2548 [STREAMING] JavaRecoverableWordCount is missing #2564

Closed
wants to merge 3 commits into from
Closed

SPARK-2548 [STREAMING] JavaRecoverableWordCount is missing #2564

wants to merge 3 commits into from

Conversation

srowen
Copy link
Member

@srowen srowen commented Sep 28, 2014

Here's my attempt to re-port RecoverableNetworkWordCount to Java, following the example of its Scala and Java siblings. I fixed a few minor doc/formatting issues along the way I believe.

@SparkQA
Copy link

SparkQA commented Sep 28, 2014

QA tests have started for PR 2564 at commit 179b3c2.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 28, 2014

QA tests have finished for PR 2564 at commit 179b3c2.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public final class JavaRecoverableNetworkWordCount

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20941/

@srowen
Copy link
Member Author

srowen commented Sep 28, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Sep 28, 2014

QA tests have started for PR 2564 at commit 179b3c2.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 28, 2014

QA tests have finished for PR 2564 at commit 179b3c2.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public final class JavaRecoverableNetworkWordCount

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20944/

@tdas
Copy link
Contributor

tdas commented Sep 30, 2014

@srowen Cool!! How did you test the JavaRecoverableWordCount?

tdas
tdas reviewed Sep 30, 2014
View reviewed changes
examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java
* checkpoint data exists in ~/checkpoint/, then it will create StreamingContext from
* the checkpoint data.
*
* To run this example in a local standalone cluster with automatic driver recovery,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this reference to spark standalone cluster mode? These instructions are old and irrelevant. Please remove from both examples :)

@srowen
Copy link
Member Author

srowen commented Oct 1, 2014

@tdas No problem, text removed. I tested the Java example using the instructions in the javadoc, and that worked. I was lazy, and didn't try it on a cluster and try killing the receiver and recovering, on the assumption that the API calls are correct and it's the tests that should make sure it works. It's a straight port of the Scala example, so if that is a valid example, this should be too.

@SparkQA
Copy link

SparkQA commented Oct 1, 2014

QA tests have started for PR 2564 at commit 35f23e3.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Oct 1, 2014

QA tests have finished for PR 2564 at commit 35f23e3.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public final class JavaRecoverableNetworkWordCount

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21107/

@srowen
Copy link
Member Author

srowen commented Oct 12, 2014

Hi @tdas, does this look OK to you? Ready to go from this end. Just a little minor change to examples but I agree it's a worth example to resurrect.

@tdas
Copy link
Contributor

tdas commented Nov 7, 2014

Hey @srowen sorry for dropping the ball on this. I am testing this manually right now. I am afraid there may have been a bug that has been spotted in this PR.
#2735

Could you update the code? Then I can test it manually and merge it right in.

@srowen
Copy link
Member Author

srowen commented Nov 7, 2014

@tdas Done, I added a change like the one in #2735

@srowen srowen mentioned this pull request Nov 7, 2014
Closed
@SparkQA
Copy link

SparkQA commented Nov 7, 2014

Test build #23078 has started for PR 2564 at commit 0d0bf29.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 8, 2014

Test build #23078 has finished for PR 2564 at commit 0d0bf29.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public final class JavaRecoverableNetworkWordCount

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23078/
Test PASSed.

@tdas
Copy link
Contributor

tdas commented Nov 10, 2014

Alright merging this.

asfgit pushed a commit that referenced this pull request Nov 10, 2014
@srowen @tdas
SPARK-2548 [STREAMING] JavaRecoverableWordCount is missing
cdcf546 
Here's my attempt to re-port `RecoverableNetworkWordCount` to Java, following the example of its Scala and Java siblings. I fixed a few minor doc/formatting issues along the way I believe.

Author: Sean Owen <sowen@cloudera.com>

Closes #2564 from srowen/SPARK-2548 and squashes the following commits:

0d0bf29 [Sean Owen] Update checkpoint call as in #2735
35f23e3 [Sean Owen] Remove old comment about running in standalone mode
179b3c2 [Sean Owen] Re-port RecoverableNetworkWordCount to Java example, and touch up doc / formatting in related examples

(cherry picked from commit 3a02d41)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
asfgit pushed a commit that referenced this pull request Nov 10, 2014
@srowen @tdas
SPARK-2548 [STREAMING] JavaRecoverableWordCount is missing
ca3fe8c 
Here's my attempt to re-port `RecoverableNetworkWordCount` to Java, following the example of its Scala and Java siblings. I fixed a few minor doc/formatting issues along the way I believe.

Author: Sean Owen <sowen@cloudera.com>

Closes #2564 from srowen/SPARK-2548 and squashes the following commits:

0d0bf29 [Sean Owen] Update checkpoint call as in #2735
35f23e3 [Sean Owen] Remove old comment about running in standalone mode
179b3c2 [Sean Owen] Re-port RecoverableNetworkWordCount to Java example, and touch up doc / formatting in related examples

(cherry picked from commit 3a02d41)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in 3a02d41 Nov 10, 2014
@srowen srowen deleted the SPARK-2548 branch November 11, 2014 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@srowen @SparkQA @AmplabJenkins @tdas