Performance issue related to Grpc client #1064
Description
I am trying to load test Grpc ( client and service )using .Net Core 3.1 in AKS, I have three microservices written ,NET core and deployed to AKS cluster and all are injected using DI during startup
services.AddGrpcClient<Validator.ValidatorClient>(o =>
{
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
o.Address = new Uri(validatorUrl);
// o.ChannelOptionsActions.Add("grpc.keepalive_permit_without_calls", 1);
}).EnableCallContextPropagation(o => o.SuppressContextNotFoundErrors = true)
.ConfigurePrimaryHttpMessageHandler(() =>
{
var handler = new HttpClientHandler();
handler.MaxConnectionsPerServer = 300;
return handler;
});
services.AddGrpcClient<Translator.TranslatorClient>(o =>
{
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
o.Address = new Uri(translatorUrl);
}).EnableCallContextPropagation(o => o.SuppressContextNotFoundErrors = true)
.ConfigurePrimaryHttpMessageHandler(() =>
{
var handler = new HttpClientHandler();
handler.MaxConnectionsPerServer = 300;
return handler;
});
services.AddGrpcClient<Connect.ConnectClient>(o =>
{
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
o.Address = new Uri(connectorUrl);
}).EnableCallContextPropagation(o => o.SuppressContextNotFoundErrors = true)
.ConfigurePrimaryHttpMessageHandler(() =>
{
var handler = new HttpClientHandler();
handler.MaxConnectionsPerServer = 300;
return handler;
});
and these are injected into the Business class
private readonly ILogger<OrchestratorService> _logger;
private readonly Validator.ValidatorClient _validator;
private readonly Translator.TranslatorClient _translator;
and being called while executing the code.
var validatorResponse = await _validator.ValdiateAsync(validatorRequest);
I use Jmeter to send a http post to a RESTful service and that service invoke these microservice using DI.
As these are deployed in Kubernetes as pods, I have used Service ulrs while injecting these microservices in the proxy.
during the performance testing, randomly it is taking longer to get the response or execute the grpc methods. generally it takes 100 to 200ms, but occasionally it is taking about 20 to 40 seconds. There is no consistency in the throughput. I thought this could httpclient socket exhaustion case ( I did not observe any exceptions) and I added handler.MaxConnectionsPerServer = 300; thinking this will improve but still I am seeing the same issue. My question is, what are the parameters/Properties I can look to improve the performance of Grpc client or service to fine tune. The business logic in these microservices are very simple. The AKS is built on Linux OS.
The same functionality written in Go and deployed to the same cluster works well and demonstrate consistent throughput.
Please let me know if you need more information.
Thanks
Ram