Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to configure a gRPC client to allow OpenShift (HAProxy) to balance requests? #2444

Open
SeminDM opened this issue May 27, 2024 · 0 comments
Labels
question Further information is requested

Comments

@SeminDM
Copy link

SeminDM commented May 27, 2024

My gRPC service is deployed in the OpenShift. I configured a route in such way:

kind: Route
apiVersion: route.openshift.io/v1
spec:
  tls:
    termination: reencrypt
    certificate: <...>
    key: <...>
    caCertificate: <...>
    destinationCACertificate: <...>

Such TLS termination allows HAProxy to spread requests between pods.
I had used "Grpc 2.46.6" for client and service LB worked pretty well.
But recently I've changed my client using "Grpc.Net.Client 2.63.0" and LB fails - HAProxy sends requests to single pod.

Question: how to configure a dotnet gRPC client for successful service load balancing?

Service:

  builder.WebHost.ConfigureKestrel(options =>
  {
  	options.ListenAnyIP(settings.Http2Port, listenOptions =>
  	{
  		// Http2 and https for gRPC.
  		listenOptions.Protocols = HttpProtocols.Http2;
  		listenOptions.UseHttps(settings.PathToPfx, settings.PfxPassword);
  	});
  	options.ListenAnyIP(settings.Http1Port, listenOptions =>
  	{
  		// Http1 for Prometheus and OpenShift probes.
  		listenOptions.Protocols = HttpProtocols.Http1;
  	});
  	options.Limits.MaxRequestBodySize = null;
  });
  builder.Services.AddGrpc(options =>
  {
  	options.EnableDetailedErrors = true;
  	options.MaxReceiveMessageSize = int.MaxValue;
  	options.MaxSendMessageSize = int.MaxValue;
  });
  var app = builder.Build();
  app.MapGrpcService<GrpcService>();
  app.MapGrpcService<GrpcStreamService>();

Grpc client:

 var options = new()
  {
     new(ChannelOptions.MaxSendMessageLength, int.MaxValue),
     new(ChannelOptions.MaxReceiveMessageLength, int.MaxValue),
     new("grpc.max_metadata_size", 10 * 1024 * 1024),
     new("grpc.enable_http_proxy", 0),
     new("grpc.lb_policy_name", "round_robin")
  };
  ChannelCredentials credentials = new SslCredentials(Resource.RootCert);
  var asyncAuthInterceptor = new AsyncAuthInterceptor((_, metadata) =>
  {
  	var spn = $"HTTP/{address}";
  	var token = KerberosTokenProvider.GetToken(spn);
  	metadata.Add("authorization-bin", token);
  	return Task.CompletedTask;
  });
  credentials = ChannelCredentials.Create(credentials, CallCredentials.FromInterceptor(asyncAuthInterceptor));
  _channel = new Channel(address, credentials, options);
  Client = CreateClient(_channel);

Grpc.Net.Client client.

  var credentials = CallCredentials.FromInterceptor((_, metadata) =>
  {
  	var spn = $"HTTP/{address}";
  	var token = KerberosTokenProvider.GetToken(spn);
  	metadata.Add("authorization-bin", token);
  	return Task.CompletedTask;
  });  
  var options = new GrpcChannelOptions
  {
  	MaxReceiveMessageSize = int.MaxValue,
  	MaxSendMessageSize = int.MaxValue,
  	ThrowOperationCanceledOnCancellation = true,
  	Credentials = ChannelCredentials.Create(ChannelCredentials.SecureSsl, credentials),
  	HttpHandler = new SocketsHttpHandler
  	{
  		EnableMultipleHttp2Connections = true,
  		UseProxy = false,
  	}
  };
  _channel = GrpcChannel.ForAddress($"https://{address}", options);
  Client = CreateClient(_channel);
@SeminDM SeminDM added the question Further information is requested label May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant