Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-10138] Python wrapper for Cross-language Java's KinesisIO #12235

Closed

Conversation

pjotrekk
Copy link

@pjotrekk pjotrekk commented Jul 13, 2020

Add a python wrapper for KinesisIO.Write.

Chronology:

  1. java write
  2. java read
  3. python wrapper
  4. python integration tests

PR for java external transform for write: #12234
PR for java external transform for read: #12297
PR for python integration tests: #12298

This test cannot be run on Jenkins because Beam does not have an AWS account. It has been tested locally on DirectRunner, FlinkRunner and SparkRunner.

R: @TheNeuralBit The same as with Java KinesisIO.Write external registrar - if you wanted to do this review I'd appreciate it :)


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status
Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
--- Build Status ---
XLang --- --- Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@TheNeuralBit
Copy link
Member

Sorry for the delay @piotr-szuberski! I'll try to take a look at these today and/or tomorrow.

One high-level question - do we have a jira about not running KinesisIO tests continuously? If not, please file one and make sure it mentions both direct Java tests as well as xlang tests.

@pjotrekk pjotrekk force-pushed the BEAM-10138-write-python-wrapper branch from c7fe25f to 9842ff1 Compare July 23, 2020 08:13
@pjotrekk
Copy link
Author

pjotrekk commented Jul 23, 2020

Sorry for the delay @piotr-szuberski! I'll try to take a look at these today and/or tomorrow.

No problem at all!

One high-level question - do we have a jira about not running KinesisIO tests continuously? If not, please file one and make sure it mentions both direct Java tests as well as xlang tests.

Yes, there is BEAM-601 - I've added a comment there and I'll mention it in the integration tests comment

@pjotrekk
Copy link
Author

pjotrekk commented Jul 23, 2020

@TheNeuralBit I've just figured out that I could use kinesis SDK which is already included in AWS_REQUIREMENTS (boto3) - It will probably make it possible to separate the read and write test.
EDIT: I followed the suggestion from Jira to use localstack with testcontainers.

@pjotrekk pjotrekk force-pushed the BEAM-10138-write-python-wrapper branch from af235cb to 4afc7df Compare July 24, 2020 12:55
@pjotrekk pjotrekk changed the title [BEAM-10138] Python wrapper for Java's KinesisIO.Write with integration test [BEAM-10138] Python wrapper for Cross-language Java's KinesisIO Jul 24, 2020
@pjotrekk pjotrekk force-pushed the BEAM-10138-write-python-wrapper branch from 4afc7df to 66fb7ce Compare July 24, 2020 13:18
@TheNeuralBit
Copy link
Member

Closing in favor of #12297

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants