Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-10137][BEAM-10138] Python wrapper KinesisIO integration tests #12298

Closed

Conversation

pjotrekk
Copy link

@pjotrekk pjotrekk commented Jul 17, 2020

Add python wrapper for KinesisIO.Read.

Chronology:

  1. java write
  2. java read
  3. python wrapper
  4. python integration tests

PR for java external transform for write: #12234
PR for python wrapper: #12235
PR for java external transform for read: #12297

I haven't managed to run this test yet. In the previous version (1st commit) it succeeded on my AWS account. But when I try to run the same pipelines on localstack I get com.amazonaws.services.kinesis.producer.UserRecordFailedException.

Strange thing is that with below code I can use the container, so it works.

 client.put_record(
      StreamName='beam',
      Data=b'bytes',
      PartitionKey='a',
)
shard_iterator = client.get_shard_iterator(
    StreamName=self.stream_name,
    ShardId='shardId-000000000000',
    ShardIteratorType='TRIM_HORIZON'
)

response = client.get_records(
      ShardIterator=shard_iterator['ShardIterator'],
      Limit=2
)

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status
Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
--- Build Status ---
XLang Build Status --- Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@pjotrekk pjotrekk force-pushed the BEAM-10137-python-wrapper-kinesis-read branch 9 times, most recently from 9804607 to 0f73856 Compare July 22, 2020 12:29
@pjotrekk
Copy link
Author

R: @TheNeuralBit Could I ask you for review?

@pjotrekk pjotrekk force-pushed the BEAM-10137-python-wrapper-kinesis-read branch from 65b7563 to e4bfe7c Compare July 24, 2020 13:17
@pjotrekk pjotrekk changed the title [BEAM-10137] Python wrapper for KinesisIO.Read with integration test [BEAM-10137] Python wrapper KinesisIO integration tests Jul 24, 2020
@pjotrekk pjotrekk changed the title [BEAM-10137] Python wrapper KinesisIO integration tests [BEAM-10137][BEAM-10138] Python wrapper KinesisIO integration tests Jul 24, 2020
@TheNeuralBit TheNeuralBit self-requested a review July 25, 2020 00:20
@TheNeuralBit
Copy link
Member

Closing in favor of #12297

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants