Skip to content

[SPARK-18410][STREAMING] Add structured kafka example #15849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
uncleGen wants to merge 1 commit into from
Closed

[SPARK-18410][STREAMING] Add structured kafka example #15849

uncleGen wants to merge 1 commit into from

Conversation

@uncleGen
Copy link
Contributor

@uncleGen uncleGen commented Nov 11, 2016

What changes were proposed in this pull request?

This PR provides structured kafka wordcount examples

How was this patch tested?

@SparkQA
Copy link

SparkQA commented Nov 11, 2016

Test build #68510 has finished for PR 15849 at commit 82a4487.

  • This patch fails RAT tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 11, 2016

Test build #68517 has finished for PR 15849 at commit 220ad02.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 14, 2016

Test build #68592 has finished for PR 15849 at commit effaead.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Nov 14, 2016

@koeninger want to review this?

@SparkQA
Copy link

SparkQA commented Nov 14, 2016

Test build #68593 has finished for PR 15849 at commit f8d7e16.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 14, 2016

Test build #68595 has finished for PR 15849 at commit 0abc93a.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 14, 2016

Test build #68598 has finished for PR 15849 at commit d1bb4a7.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 14, 2016

Test build #68599 has finished for PR 15849 at commit e6a5367.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

koeninger
koeninger reviewed Nov 14, 2016
View reviewed changes
Copy link
Contributor

@koeninger koeninger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks OK to me.
One question - am I misremembering that we previously had issues building a single example jar that had both kafka 0.8 and 0.10 dependencies in it? What changed?

...ples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java Outdated
Copy link
Contributor

@koeninger koeninger Nov 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't really a list of topics for assign or subscribePattern. I'd say use a more general description, or stick to just subscribe

Copy link
Contributor Author

@uncleGen uncleGen Nov 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get it

@SparkQA
Copy link

SparkQA commented Nov 15, 2016

Test build #68651 has finished for PR 15849 at commit e0c8b4e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@koeninger
Copy link
Contributor

koeninger commented Nov 15, 2016

LGTM

@uncleGen
SPARK-18410: Add structured kafka example
7126dc9 
update

update

add license

add implicit import

add python example

code clean

bug fix

udate comments

update comments
@uncleGen
Copy link
Contributor Author

uncleGen commented Nov 16, 2016

just rebase

@SparkQA
Copy link

SparkQA commented Nov 16, 2016

Test build #68688 has finished for PR 15849 at commit 7126dc9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public final class JavaStructuredKafkaWordCount

@srowen
Copy link
Member

srowen commented Nov 16, 2016

Merged to master/2.1

@asfgit asfgit closed this in e614577 Nov 16, 2016
asfgit pushed a commit that referenced this pull request Nov 16, 2016
@uncleGen @srowen
[SPARK-18410][STREAMING] Add structured kafka example
6b2301b 
## What changes were proposed in this pull request?

This PR provides structured kafka wordcount examples

## How was this patch tested?

Author: uncleGen <hustyugm@gmail.com>

Closes #15849 from uncleGen/SPARK-18410.

(cherry picked from commit e614577)
Signed-off-by: Sean Owen <sowen@cloudera.com>
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
@uncleGen @uzadude
[SPARK-18410][STREAMING] Add structured kafka example
6992cb7 
## What changes were proposed in this pull request?

This PR provides structured kafka wordcount examples

## How was this patch tested?

Author: uncleGen <hustyugm@gmail.com>

Closes apache#15849 from uncleGen/SPARK-18410.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koeninger koeninger koeninger left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@uncleGen @SparkQA @rxin @koeninger @srowen