Is Dapr input binding with Azure Event Hub processing messages synchronously?

[marxxxx]

In what area(s)?

/area runtime
/area docs

Ask your question here

We are using dapr along with an input binding to consume IoT Events from Azure Event Hub. The event hub has 32 partitions and the sender uses the device id as partition key. The controller receiving the events delegates them to dapr Actors for processing.

Now my expectation would be, that the messages in each partition are processed in parallel, resulting in parallel requests to the controller receiving the events allowing for up to 32 concurrent event batches to be processed.

However tests show that the events are received synchronously. The consequent event is received just after the full processing of the previous event has completed.

Now i know that actors process messages one after the other by design but i did not read anything like this for input bindings according to the documentation.

Is there anything i'm completely missing? Otherwise i could not image how this system could scale.

We are using dapr 0.11 along with ASP.NET Core 3.1 running in an AKS Cluster.

FYI: I've also posted this question on StackOverflow

[yaron2]

Transferred issue to components-contrib.

The Event Hubs binding processes events in order and synchronously. It would be easy to add a concurrency option (like the one that exists on some pub/sub components) and allow for concurrent batches.

Is this something you might be interested in contributing?

[marxxxx]

Thank you for your answer!

So do i understand you correctly? The current implementation of the binding ignores the partition and does only allow the processing of one message after the other no matter which partition the message is coming from?

I suppose this is the implementation of the binding? https://github.com/dapr/components-contrib/tree/master/bindings/azure/eventhubs

[yaron2]

Thank you for your answer!

So do i understand you correctly? The current implementation of the binding ignores the partition and does only allow the processing of one message after the other no matter which partition the message is coming from?

I suppose this is the implementation of the binding? https://github.com/dapr/components-contrib/tree/master/bindings/azure/eventhubs

Not quite, you can specify a Dapr component per partition ID: https://docs.dapr.io/operations/components/setup-bindings/supported-bindings/eventhubs/

If you set the partitionID field on the component metadata, the consumer will pull all messages for that partition ID.
Otherwise, it'll pull from all partitions.

[marxxxx]

Parallel execution would certainly improve the situation. However if load increases I would expect some sort of scale out similar to Azure Functions / KEDA. Is this something that can/should be done at the binding level or would this happen some levels above?

Update: According to the docs this seems to work already for the Kafka Binding (https://docs.dapr.io/developing-applications/integrations/autoscale-keda/). So i as far as i can tell this would be the ideal solution for the Event Hub binding as well.

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[dapr-bot]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.