You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a situation where we need to process a large amount of incoming webhooks, with bursty traffic.
To do so, the messages should be put on an SQS Queue for later processing.
Exploring reactive messaging to achieve this (using Quarkus), the current approach looks more or less like this:
@Inject@Channel("webhooks_enqueue")
MutinyEmitter<WebhookPayload> messageEmitter;
@POST@RunOnVirtualThreadpublicvoidreceiveWebhook(Webhookw) {
// wait for message to be enqueued in case of buffer overflow, but respect// SLAs from webhook caller.messageEmitter.sendMessage( Message.of( w ) ).await().atMost( Duration.ofMillis( 250 ) );
}
However, during intensive benchmarking the io.smallrye.mutiny.TimeoutException is often thrown, indicating that the underlying producer can't keep up. Increasing the connection pool for the AWS SDK client has no effect, indicating that this isn't the limiting factor. Additionally, using the SQSClient directly instead of an Emitter results in much higher throughput when enqueueing messages, without the webhook endpoint timing out.
Are there options for increasing parallelism with Emitters? For incoming channels there are options for defining consumer concurrency and worker pool threads, but this doesn't appear to be the case for outgoing channels.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
I have a situation where we need to process a large amount of incoming webhooks, with bursty traffic.
To do so, the messages should be put on an SQS Queue for later processing.
Exploring reactive messaging to achieve this (using Quarkus), the current approach looks more or less like this:
However, during intensive benchmarking the
io.smallrye.mutiny.TimeoutExceptionis often thrown, indicating that the underlying producer can't keep up. Increasing the connection pool for the AWS SDK client has no effect, indicating that this isn't the limiting factor. Additionally, using theSQSClientdirectly instead of an Emitter results in much higher throughput when enqueueing messages, without the webhook endpoint timing out.Are there options for increasing parallelism with Emitters? For incoming channels there are options for defining consumer concurrency and worker pool threads, but this doesn't appear to be the case for outgoing channels.
All reactions