[BEAM-11325] Kafka Dynamic Read #13750

boyuanzz · 2021-01-14T04:34:55Z

design doc: https://docs.google.com/document/d/1FU3GxVRetHPLVizP3Mdv6mP5tpjZ3fd99qNjUI5DT5k/edit#

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

boyuanzz · 2021-01-19T03:01:43Z

cc: @aromanenko-dev It is a really early stage PR but just in case you are interested in : )

boyuanzz · 2021-01-20T23:53:16Z

Hi Alexey, this PR is ready to review. I mark WatchKafkaTopicPartitionDoFn as Experimental for now and we can remove it after I have an IT for it.

aromanenko-dev · 2021-01-22T17:08:55Z

@boyuanzz Thanks! I'll take a look on this in the next week.

boyuanzz · 2021-02-01T18:53:17Z

Kindly pinging : )
I want to make it into 2.29.0 release if that's feasible.

aromanenko-dev · 2021-02-02T15:24:31Z

Run Java PreCommit

aromanenko-dev

Thanks, sorry for delay. I took a quick first look and it's fine to me, just a minor notes.

Did you test it against real Kafka cluster with adding/removing topics and/or partitions?

CC: @chamikaramj Could you take a look as well if you have a time?

aromanenko-dev · 2021-02-02T15:28:02Z

sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java

+ *
+ * For a given kafka bootstrap_server, KafkaIO is also able to detect and read from available {@link
+ * TopicPartition} dynamically. For more design details, please refer to
+ * https://docs.google.com/document/d/1FU3GxVRetHPLVizP3Mdv6mP5tpjZ3fd99qNjUI5DT5k/. To enable


Please, put the details here (at least in a short form) instead of the link since it can be expired or disappeared along the time.

Will do. Thanks for pointing this out.

aromanenko-dev · 2021-02-02T15:35:17Z

sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java

@@ -553,6 +587,9 @@
      abstract Builder<K, V> setValueDeserializerProvider(
          DeserializerProvider deserializerProvider);

+      abstract Builder<K, V> setCheckStopReadingFn(


Wasn't this functionality added in #13710 ?

Yeah, I opened this PR prior to mergeing #13710. I'll rebase them. Thanks for catching it : )

aromanenko-dev · 2021-02-02T15:44:12Z

sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/WatchKafkaTopicPartitionDoFn.java

+
+/**
+ * A stateful {@linkl DoFn} that emits new available {@link TopicPartition} regularly. Please refer
+ * to https://docs.google.com/document/d/1FU3GxVRetHPLVizP3Mdv6mP5tpjZ3fd99qNjUI5DT5k/edit# for more


The same note about the link to details.

aromanenko-dev · 2021-02-02T15:51:27Z

Also, ptal on "Java PreCommit" test

16:40:12 Execution failed for task ':sdks:java:io:kafka:checkstyleMain'.

boyuanzz · 2021-02-02T23:08:00Z

I added most javadoc to KafkaIO to explain how it works and potential race condition. I haven't test this functionality with real Kafka cluster so I marked WatchKafkaTopicPartitionDoFn as @Experimental. I'm exploring ways to have ITs for these functionalities. @aromanenko-dev

aromanenko-dev

Thanks! LGTM
I think we can merge it if there are no other objections/comments

aromanenko-dev · 2021-02-04T18:05:07Z

Run Java PostCommit

boyuanzz · 2021-02-04T22:22:37Z

Thanks for your quick review! I'm going to merge this PR.

Cham, feel free to drop any comments if you have concerns around this. I'll follow up if any.

aromanenko-dev · 2021-02-09T09:42:30Z

@boyuanzz I think we forgot one small but important thing - update CHANGES.md about this feature.

boyuanzz · 2021-02-09T18:35:25Z

@boyuanzz I think we forgot one small but important thing - update CHANGES.md about this feature.

Thanks for the remainder! I'll update CHANGES.md later today.

boyuanzz · 2021-02-10T02:35:21Z

@boyuanzz I think we forgot one small but important thing - update CHANGES.md about this feature.

Thanks for the remainder! I'll update CHANGES.md later today.

I just checked CHANGES.md and it seems like we want to add this feature into 2.29.0 release. I'll follow up with this when 2.29.0 release branch is cut.

aromanenko-dev self-requested a review January 20, 2021 17:21

boyuanzz force-pushed the dynamic_kafka branch from 2940091 to 3a6dc14 Compare January 20, 2021 23:33

boyuanzz changed the title ~~[WIP] Kafka Dynamic Read~~ [11325] Kafka Dynamic Read Jan 20, 2021

boyuanzz changed the title ~~[11325] Kafka Dynamic Read~~ [BEAM-11325] Kafka Dynamic Read Jan 20, 2021

aromanenko-dev requested changes Feb 2, 2021

View reviewed changes

[BEAM-11325] Support KafkaIO dynamic read

0fbb21f

boyuanzz force-pushed the dynamic_kafka branch from 3a6dc14 to 0fbb21f Compare February 2, 2021 22:59

aromanenko-dev approved these changes Feb 4, 2021

View reviewed changes

boyuanzz merged commit 47d3326 into apache:master Feb 4, 2021

boyuanzz deleted the dynamic_kafka branch March 11, 2021 18:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-11325] Kafka Dynamic Read #13750

[BEAM-11325] Kafka Dynamic Read #13750

boyuanzz commented Jan 14, 2021 •

edited

boyuanzz commented Jan 19, 2021

boyuanzz commented Jan 20, 2021

aromanenko-dev commented Jan 22, 2021

boyuanzz commented Feb 1, 2021

aromanenko-dev commented Feb 2, 2021

aromanenko-dev left a comment

aromanenko-dev Feb 2, 2021

boyuanzz Feb 2, 2021

aromanenko-dev Feb 2, 2021

boyuanzz Feb 2, 2021

aromanenko-dev Feb 2, 2021

aromanenko-dev commented Feb 2, 2021

boyuanzz commented Feb 2, 2021

aromanenko-dev left a comment

aromanenko-dev commented Feb 4, 2021

boyuanzz commented Feb 4, 2021

aromanenko-dev commented Feb 9, 2021

boyuanzz commented Feb 9, 2021

boyuanzz commented Feb 10, 2021

[BEAM-11325] Kafka Dynamic Read #13750

[BEAM-11325] Kafka Dynamic Read #13750

Conversation

boyuanzz commented Jan 14, 2021 • edited

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

boyuanzz commented Jan 19, 2021

boyuanzz commented Jan 20, 2021

aromanenko-dev commented Jan 22, 2021

boyuanzz commented Feb 1, 2021

aromanenko-dev commented Feb 2, 2021

aromanenko-dev left a comment

Choose a reason for hiding this comment

aromanenko-dev Feb 2, 2021

Choose a reason for hiding this comment

boyuanzz Feb 2, 2021

Choose a reason for hiding this comment

aromanenko-dev Feb 2, 2021

Choose a reason for hiding this comment

boyuanzz Feb 2, 2021

Choose a reason for hiding this comment

aromanenko-dev Feb 2, 2021

Choose a reason for hiding this comment

aromanenko-dev commented Feb 2, 2021

boyuanzz commented Feb 2, 2021

aromanenko-dev left a comment

Choose a reason for hiding this comment

aromanenko-dev commented Feb 4, 2021

boyuanzz commented Feb 4, 2021

aromanenko-dev commented Feb 9, 2021

boyuanzz commented Feb 9, 2021

boyuanzz commented Feb 10, 2021

boyuanzz commented Jan 14, 2021 •

edited