Adding load balancing for Services

[Jenscaasen]

With many Azure OpenAI Endpoints available to Azure customers, with each having a limited number of Tokens per Minute (and possibly other vendors in the future as well), being limited to one endpoint at a time in SK can lead to an exhaustion of available token quotas.

The idea is to extend the KernelBuilder or Kernel with an attribute named "EnableLoadBalancing". In the Kernel GetRequiredService currently only the last Service is returned. This can easily be modified to return a random service, hence adding load balancing when multiple are registered in the Kernel.

I would take on the development myself

[stephentoub]

In the Kernel GetRequiredService currently only the last Service is returned. This can easily be modified to return a random service, hence adding load balancing when multiple are registered in the Kernel.

Can you use GetAllServices and just pick one based on whatever policy you want? e.g.

IEnumerable<T> services = kernel.GetAllServices<T>();
T service = services.ElementAt(Random.Shared.Next(services.Count()));

[markwallace-microsoft]

We also have an IAIServiceSelector abstraction which will allow you to control which service gets used when executing a prompt. This example Example62_CustomAIServiceSelector shows how you would implement this.

[Jenscaasen]

We also have an IAIServiceSelector abstraction which will allow you to control which service gets used when executing a prompt. This example Example62_CustomAIServiceSelector shows how you would implement this.

If the IAIServiceSelector is invoked every time a ChatCompletionService is requested from the Kernel (by Planners or other call to InvokePromptAsync that you can not influence) then this is the solution, thank you! I will look into it. Will close the issue