Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-26121] [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id) #23103

Closed
wants to merge 7 commits into from
Closed

[SPARK-26121] [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id) #23103

wants to merge 7 commits into from

Conversation

zouzias
Copy link

@zouzias zouzias commented Nov 21, 2018 •
edited

What changes were proposed in this pull request?

Allow the Spark Structured Streaming user to specify the prefix of the consumer group (group.id), compared to force consumer group ids of the form spark-kafka-source-*

How was this patch tested?

Unit tests provided by Spark (backwards compatible change, i.e., user can optionally use the functionality)

mvn test -pl external/kafka-0-10

@zouzias zouzias changed the title [SPARK-26121] init commit [SPARK-26121] [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id) Nov 21, 2018
@zouzias
Copy link
Author

zouzias commented Nov 21, 2018

cc @koeninger

Preliminary PR, todos:

  • Decide on input parameter name for prefix consumer group, i.e., streaming.group_id.prefix.
  • Document the above parameter in Spark SS docs.

@tgravescs
Copy link
Contributor

ok to test

@SparkQA
Copy link

SparkQA commented Nov 21, 2018

Test build #99124 has finished for PR 23103 at commit 4afd3a4.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 21, 2018

Test build #99125 has finished for PR 23103 at commit 9a09e74.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 21, 2018

Test build #99126 has finished for PR 23103 at commit cac7976.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 21, 2018

Test build #99134 has finished for PR 23103 at commit 9d9c944.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 22, 2018

Test build #99167 has finished for PR 23103 at commit 39424dd.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 22, 2018

Test build #99183 has finished for PR 23103 at commit f79ba9f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

koeninger
koeninger approved these changes Nov 22, 2018
View reviewed changes
Copy link
Contributor

@koeninger koeninger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @zouzias

@tgravescs if there's anything else you need on this one chime in, otherwise ill merge in a couple days.

@tgravescs
Copy link
Contributor

Lgtm

@HeartSaVioR
Copy link
Contributor

LGTM. Btw, IMHO, TODOs @zouzias described would be better to be addressed at once since documentation is easy to be forgotten.

@koeninger
Copy link
Contributor

@zouzias can you add the new option to docs/structured-streaming-kafka-integration.md as part of this PR? Instructions for building docs are in docs/README.md , ping me if you need a hand.

@zouzias
Copy link
Author

zouzias commented Nov 23, 2018

@koeninger, I will make the doc changes asap. FYI, I plan to make changes on file structured-streaming-kafka-integration.md seems the most relevant doc for this diff.

@SparkQA
Copy link

SparkQA commented Nov 23, 2018

Test build #99219 has finished for PR 23103 at commit 9196623.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@koeninger
Copy link
Contributor

merging to master, thanks @zouzias

@asfgit asfgit closed this in 2512a1d Nov 26, 2018
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
@jackylee-ch
[SPARK-26121][STRUCTURED STREAMING] Allow users to define prefix of K…
ca4f731 
…afka's consumer group (group.id)

## What changes were proposed in this pull request?

Allow the Spark Structured Streaming user to specify the prefix of the consumer group (group.id), compared to force consumer group ids of the form `spark-kafka-source-*`

## How was this patch tested?

Unit tests provided by Spark (backwards compatible change, i.e., user can optionally use the functionality)

`mvn test -pl external/kafka-0-10`

Closes apache#23103 from zouzias/SPARK-26121.

Authored-by: Anastasios Zouzias <anastasios@sqooba.io>
Signed-off-by: cody koeninger <cody@koeninger.org>
goosalex pushed a commit to goosalex/spark that referenced this pull request Jun 26, 2019
[SPARK-26121][STRUCTURED STREAMING] Allow users to define prefix of K…
a68e4db 
…afka's consumer group (group.id)

## What changes were proposed in this pull request?

Allow the Spark Structured Streaming user to specify the prefix of the consumer group (group.id), compared to force consumer group ids of the form `spark-kafka-source-*`

## How was this patch tested?

Unit tests provided by Spark (backwards compatible change, i.e., user can optionally use the functionality)

`mvn test -pl external/kafka-0-10`

Closes apache#23103 from zouzias/SPARK-26121.

Authored-by: Anastasios Zouzias <anastasios@sqooba.io>
Signed-off-by: cody koeninger <cody@koeninger.org>
goosalex pushed a commit to goosalex/spark that referenced this pull request Jun 26, 2019
[SPARK-26121][STRUCTURED STREAMING] Allow users to define prefix of K…
8deff33 
…afka's consumer group (group.id)

Allow the Spark Structured Streaming user to specify the prefix of the consumer group (group.id), compared to force consumer group ids of the form `spark-kafka-source-*`

Unit tests provided by Spark (backwards compatible change, i.e., user can optionally use the functionality)

`mvn test -pl external/kafka-0-10`

Closes apache#23103 from zouzias/SPARK-26121.

Authored-by: Anastasios Zouzias <anastasios@sqooba.io>
Signed-off-by: cody koeninger <cody@koeninger.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@koeninger koeninger koeninger approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@zouzias @tgravescs @SparkQA @HeartSaVioR @koeninger