-
Notifications
You must be signed in to change notification settings - Fork 808
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pubsub/awssnssqs: aws sqs expose receiver max batch #3412
Conversation
Signed-off-by: pxp928 <parth.psu@gmail.com>
Signed-off-by: pxp928 <parth.psu@gmail.com>
Out of curiousity, why do you need to set this? |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3412 +/- ##
==========================================
+ Coverage 73.12% 73.14% +0.01%
==========================================
Files 113 113
Lines 14864 14870 +6
==========================================
+ Hits 10870 10876 +6
Misses 3219 3219
Partials 775 775 ☔ View full report in Codecov by Sentry. |
Hey @vangent, the use case is to allow for flexibility to change the max batch size if needed. For example, having a smaller batch size could result in us being able to scale out our process service so that there are fewer messages "in-flight" (SQS visibility timeout). So the process service can be scaled in or out as needed based on the configured batch size that each process service can handle. |
Have you tried doing that without manually tuning it? The So, for example, if you only have 2 worker goroutines processing messages, it is unlikely that it will be fetching 10 messages at a time and letting them sit there for a long time (unless the processing time is very fast, in which you do want more messages to be queued so that the workers aren't idle). Basically, the package does a lot to try to make it so that you don't have to manually tune this, that's one of the benefits. If you are manually tuning it because what I've described isn't working the way you want for some reason, that's one thing, but I don't want you to add complexity manaully tuning something that shouldn't need it. |
oh, interesting I did not see that in https://gocloud.dev/howto/pubsub/subscribe/. Is there any documentation around this behavior? Is there a way to know what the current batch size is set to and what it changes to via logs? Thank You for pointing this out. |
I don't think it's well-documented, as it is not really part of the public interface, it's internal implementation detail. You can see constants for the algorithm here: and the main code is here: No, the batch size isn't currently logged, but you can patch a local copy and add some logging. |
Thanks @vangent for the information! |
This PR exposes the receiver max batch size as a URL parameter for AWS SQS via
receivermaxbatch
.For example:
awssqs://sqs.us-east-2.amazonaws.com/99999/my-queue?receivermaxbatch=5
Based on the
recvBatcherOpts
:go-cloud/pubsub/awssnssqs/awssnssqs.go
Lines 118 to 123 in be1b4ae
and the limitations of SQS, any value above 10 would default back to 10.