Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-7029] Add KafkaIO.Write as an external transform #8322

Merged
merged 1 commit into from Apr 26, 2019

Conversation

@mxm
Copy link
Contributor

commented Apr 16, 2019

This adds KafkaIO.Write as an external transform and includes a Python
wrapper (WriteToKafka) for convenience.

CC @chamikaramj @ihji @robertwb @tweise

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- --- --- --- ---
Java Build Status Build Status Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status
Python Build Status
Build Status
--- Build Status
Build Status
Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@mxm

This comment has been minimized.

Copy link
Contributor Author

commented Apr 16, 2019

Run Python PreCommit

sdks/java/harness/build.gradle Outdated Show resolved Hide resolved

@mxm mxm force-pushed the mxm:BEAM-7029 branch from 57aee23 to 82e8ae6 Apr 22, 2019

@mxm

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2019

Planning to merge this later today if there are no further comments.

@tweise

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2019

We should allow the user to specify the environment for the external transform. Would like to be able to use EMBEDDED when using the Flink runner.

@mxm

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2019

I already discovered this before opening this PR and created https://jira.apache.org/jira/browse/BEAM-7084. I think this can be solved independently of this PR.

@aromanenko-dev
Copy link
Contributor

left a comment

Added a couple of minor notes.

@chamikaramj
Copy link
Contributor

left a comment

Thanks Max. LGTM other than few minor comments.

sdks/java/container/Dockerfile Show resolved Hide resolved
sdks/java/container/Dockerfile-java11 Show resolved Hide resolved
sdks/python/apache_beam/io/external/kafka.py Outdated Show resolved Hide resolved
self.assertTrue('No resolvable bootstrap urls given in bootstrap.servers'
in str(ctx.exception),
'Expected to fail due to invalid bootstrap.servers, but '
'failed due to:\n%s' % str(ctx.exception))

# We just test the expansion but do not execute.

This comment has been minimized.

Copy link
@chamikaramj

chamikaramj Apr 25, 2019

Contributor

FYI: @ihji is looking into adding a validates runner test suite for cross-language transforms that includes Kafka.

This comment has been minimized.

Copy link
@mxm

mxm Apr 25, 2019

Author Contributor

Yes, I saw the PR (#8397). That will be great to extend our test coverage here.

[BEAM-7029] Add KafkaIO.Write as an external transform
This adds KafkaIO.Write as an external transform and includes a Python
wrapper (WriteToKafka) for convenience.

@mxm mxm force-pushed the mxm:BEAM-7029 branch from 82e8ae6 to aebf831 Apr 25, 2019

@mxm

This comment has been minimized.

Copy link
Contributor Author

commented Apr 26, 2019

Thanks for all the comments!

@mxm mxm merged commit 077223e into apache:master Apr 26, 2019

8 checks passed

Java ("Run Java PreCommit") SUCCESS
Details
JavaPortabilityApi ("Run JavaPortabilityApi PreCommit") SUCCESS
Details
Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit") SUCCESS
Details
Portable_Python ("Run Portable_Python PreCommit") SUCCESS
Details
Python ("Run Python PreCommit") SUCCESS
Details
Python_PVR_Flink ("Run Python_PVR_Flink PreCommit") SUCCESS
Details
RAT ("Run RAT PreCommit") SUCCESS
Details
Spotless ("Run Spotless PreCommit") SUCCESS
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.