Multiple KafkaFlow Hosted Services in one instance

[kjsulls]

Currently, we have several hosted background services, consumers, in a single instance. Each hosted services is using Kafka library to consume and process a message one at a time. Nothing fancy. See snippet below.

I am currently in the process of implementing KafkaFlow, in order to leverage batch consumer and more, and wondering if it's possible to have several services.AddKafkaFlowHostedService() in one instance? As far as I can tell, this is not possible. What would be the recommended path moving forward? My thought is to use the one hosted service (AddKafkaFlowHostedService()) and add each change handler as consumer like so... Thoughts?

[gsferreira]

Hi @kjsulls!

If the reason behind the multiple hosted services was to have multiple consumers in the same application, you don't need that on KafkaFlow.

As you say, you can register multiple consumers (if the goal is to consume multiple topics). Your example looks like in the right direction.

Let me know if you have any remaining doubts.

[kjsulls]

Hi @gsferreira,

Thank you for the information. Makes perfect sense.

I do have one more question :)

In my Batch Consume Middleware, I'm getting messageContext.GetMessagesBatch(); then grouping by message key. That's all working just fine. The intention here is I want to process each group in parallel and for each group send the list of messages to the next middleware. I want the next middleware (typed handler) to accept a List. In the typed handler, I want to insert/update N number of messages to my persistence layer, then SaveChanges once at the end to reduce the number of calls to the persistence layer. I have not been able to figure out a way to have a typed handler accept a List. Is this possible or is it just better to process the messages in my Batch Consume Middleware?

[gsferreira]

Hi @kjsulls

1. On the KafkaFlow side, there's no imposed limitation. By the way, I'm aware of applications using KafkaFlow with more than 20 consumers.

2. The problem is not the fact that it is a list. In the way Batch Consumer works, you can't change the "Chain" Message Value. I would say you need to address that outside of the Middleware pipeline.

[kjsulls]

Hi @gsferreira,

Excellent! Thank you for the information. I'll be in touch if I have any other questions. Cheers!