Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-7636] Migrate SqsIO to AWS SDK for Java 2 #9935

Merged
merged 7 commits into from Nov 22, 2019

Conversation

cmachgodaddy
Copy link
Contributor

@cmachgodaddy cmachgodaddy commented Oct 30, 2019

This PR is adding SqsIO v2 to amazon-web-services2 submodule, and which uses AWS SDK V2. There are a few changes comparing to SqsIO V1:

  1. Adding SendMessageRequestCoder, since it's not serializable in SDK V2
  2. Adding MessageCoder, since it's not serializable in SDK V2
  3. Adding a custom SqsMessage, since the attributes of the Message is not serializable, and to return attributes to downstream users. We will get an exception DefaultSdkContructionMap is not serializable if we don't do this.
  4. Refactor unit test: have a separate class for EmbeddedSqlServer
  5. Remove SqsConfiguration class, and add withXXX methods() to take in AwsSecretsProvider

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
--- Build Status
Build Status
Build Status
Build Status
--- --- Build Status
XLang --- --- --- Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@aromanenko-dev
Copy link
Contributor

Thanks for contribution!
Please, run gradlew spotlessApply before committing.

@cmachgodaddy cmachgodaddy changed the title [WIP][Beam-7636] Migrate SqsIO to AWS SDK for Java 2 [Beam-7636] Migrate SqsIO to AWS SDK for Java 2 Nov 3, 2019
@aromanenko-dev
Copy link
Contributor

Run Java PreCommit

@aromanenko-dev aromanenko-dev changed the title [Beam-7636] Migrate SqsIO to AWS SDK for Java 2 [BEAM-7636] Migrate SqsIO to AWS SDK for Java 2 Nov 12, 2019
Copy link
Contributor

@aromanenko-dev aromanenko-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I did a review mostly over new added part and I expect that all other functionality/tests are left "as it was before".
Also, I think that @JohnRudolfLewis, as initial author of SqsIO, could be interested in review too.

sdks/java/io/amazon-web-services2/build.gradle Outdated Show resolved Hide resolved
sdks/java/io/amazon-web-services2/build.gradle Outdated Show resolved Hide resolved

@ProcessElement
public void processElement(ProcessContext processContext) throws Exception {
sqs.sendMessage(processContext.element());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it send message synchronically? Would it make sense to batch the messages of the whole bundle and send them as a batch for better performance?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that make send, but then our API has to take in SendMessageBatchRequestEntry, instead of SendMessageRequest. Now, there are three ways to deal with this:

  1. Take in SendMessageBatchRequestEntry, instead of SendMessageRequest. This way will impact all of the existing users
  2. Keep the existing api, which take in SendMessageRequest, and convert it to SendMessageBatchRequestEntry manually in our IO. Bascially, mapping each attribute of SendMessageRequest to SendMessageBatchRequest's
  3. Add another API, called writeBatch, and this one takes in SendMessageBatchRequestEntry, backward compatibility support
    I prefer [BEAM-22] Support Unbounded PCollections in same-process execution #3, since it makes a clear distinction, users know what they are doing, and nobody get "hurt" ;-)
    @aromanenko-dev let me know what you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok to keep it as it was before for now. Perhaps, it would make sense to add TODO comment for the future about that.

@aromanenko-dev
Copy link
Contributor

Please, take a look on failing tests in Java PreCommit job on Jenkins

@aromanenko-dev
Copy link
Contributor

@cmachgodaddy Please, fix Spotless issue.

@aromanenko-dev
Copy link
Contributor

aromanenko-dev commented Nov 20, 2019

@cmachgodaddy There are 2 failed "old" SqsIOTest tests. I think it's related to using the same host/port for EmbeddedSqsServer. Could you fix this, please?
PS: other fails are not related to this PR, I think

@aromanenko-dev
Copy link
Contributor

Run CommunityMetrics PreCommit

@aromanenko-dev
Copy link
Contributor

Run Python PreCommit

1 similar comment
@cmachgodaddy
Copy link
Contributor Author

Run Python PreCommit

Copy link
Contributor

@aromanenko-dev aromanenko-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM

@aromanenko-dev aromanenko-dev merged commit 07d952f into apache:master Nov 22, 2019
@aromanenko-dev aromanenko-dev changed the title [BEAM-7636] Migrate SqsIO to AWS SDK for Java 2 [BEAM-7636] Migrate SqsIO to AWS SDK V2 for Java Nov 22, 2019
@aromanenko-dev aromanenko-dev changed the title [BEAM-7636] Migrate SqsIO to AWS SDK V2 for Java [BEAM-7636] Migrate SqsIO to AWS SDK for Java 2 Nov 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants