[Feature Request]: Dynamic Worker Pool Scaling

[filipeesch]

Is your request related to a problem you have?

Yes. At present, the KafkaFlow worker count is statically set at the application setup. While we can alter this count manually via admin interfaces, doing so results in substantial partition rebalancing operations that can be disruptive to applications. Dynamic workloads and varying consumer lags demand a more flexible worker scaling approach.

Describe the solution you'd like

I propose enhancing the IConsumerConfigurationBuilder with a new overload that allows for dynamic worker count adjustment:

/// <summary>
/// Configures a custom function to dynamically calculate the number of workers.
/// </summary>
/// <param name="calculator">A function that takes a WorkersCountContext object and returns a Task yielding the new workers count</param>
/// <param name="evaluationInterval">The interval that the calculator will be called</param>
/// <returns>The IConsumerConfigurationBuilder instance for method chaining</returns>
IConsumerConfigurationBuilder WithWorkersCount(
    Func<WorkersCountContext, IDependencyResolver, Task<int>> calculator,
    TimeSpan evaluationInterval);

Here's a breakdown of the proposed changes:

Internal Decoupling: Decouple certain internal consumer structures, enabling only the Consumer Feeder and the Worker Pool to restart with a varied worker count. This procedure would be lightweight and would sidestep the need for a partition rebalance.

Dynamic Worker Count Calculation: Implement a client-side function that can adjust the worker pool size based on external metrics (e.g., consumer lag or application workload). This function would leverage the WorkersCountContext to access consumer details and the IDependencyResolver for DI container abstractions.

Regular Invocation: The client function would be invoked at specified intervals, enabling KafkaFlow to adjust dynamically based on the ongoing workload.

Are you able to help bring it to life and contribute with a Pull Request?

No

Additional context

Introducing dynamic worker scaling will significantly boost KafkaFlow's adaptability and scalability. This feature would empower KafkaFlow to optimize resource consumption efficiently in environments with fluctuating workloads, ensuring optimal performance without manual intervention.