Schema Registry Config SslCaLocation doesn't work

[danielbojczuk]

Description

Nuget Version: Confluent.SchemaRegistry 2.3.0.
Apache Kafka version: Doesn't have influence
Operating system: Windows

SchemaRegistry Client is adding the CA certificates from the configuration SslCaLocation and SslKeystoreLocation to the HttpClientHandler.ClientCertificates Property:

• confluent-kafka-dotnet/src/Confluent.SchemaRegistry/CachedSchemaRegistryClient.cs

  Line 298 in 007a8ba

  private List<X509Certificate2> SetSslConfig(IEnumerable<KeyValuePair<string, string>> config)

• confluent-kafka-dotnet/src/Confluent.SchemaRegistry/Rest/RestService.cs

  Line 89 in 007a8ba

  private static HttpClientHandler CreateHandler(List<X509Certificate2> certificates, bool enableSslCertificateVerification)

But from the documentation, only the SSlKeystoreLocation should be placed on this HttpHandlers property for MTLS. The CA certificates should be handled differently: https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.clientcertificates?view=netcore-2.0

So, the current implementation doesn't allow you to connect to a schema registry server if it uses a certificate signed by a local CA. It throws the following exception:

Local: Key serialization error
 ---> System.Net.Http.HttpRequestException: [SERVER_URL] HttpRequestException: The SSL connection could not be established, see inner exception.
   at Confluent.SchemaRegistry.RestService.ExecuteOnOneInstanceAsync(Func`1 createRequest) in...

How to reproduce

Execute this code using a SchemaRegistry server using a locally signed SSL certificate:

var schemaConfig = new SchemaRegistryConfig
{
    Url = SCHEMA_REGISTRY_URL,
    SslKeystoreLocation = CERTIFICATE_PATH,
    SslKeystorePassword = CERTIFICATE_PASSWORD,
    SslCaLocation = TRUSTED_CA_FILEPATH,
};

var schemaregistry = new CachedSchemaRegistryClient(schemaConfig);
var keySerializer = new AvroSerializer<Application>(schemaregistry).AsSyncOverAsync();
var valueSerializer = new AvroSerializer<Event>(schemaregistry).AsSyncOverAsync();


var privKeyBytes = Convert.FromBase64String(PRIVATE_KEY_BASE64);
var privKeyPem = PemEncoding.Write("RSA PRIVATE KEY", privKeyBytes);

var producerConfig = new ProducerConfig
{
    BootstrapServers = BOOTSTRAP_SERVERS,

    SocketTimeoutMs = 30_000,
    ReconnectBackoffMaxMs = 10_000,

    SecurityProtocol = SecurityProtocol.Ssl,
    SslEndpointIdentificationAlgorithm = SslEndpointIdentificationAlgorithm.None,
    EnableSslCertificateVerification = true,
    SslCertificateLocation = PUBLIC_KEY_FILE_PATH,
    SslCaCertificateStores = "My",
    SslKeyPem = new string(privKeyPem),
};

var producer = new ProducerBuilder<Application, Event>(producerConfig)
    .SetKeySerializer(keySerializer)
    .SetValueSerializer(valueSerializer)
.Build();

var message = new Message<Application, Event>
{
    Key = EventHelper.CreateApplication(),
    Value = EventHelper.CreateEvent()
};

var result = await producer.ProduceAsync(TOPIC, message);

Console.WriteLine($"Written to offset {result.Offset.Value}");

The SchemaRegistry is hiding the InnerException, but with some debugging, you can check the real exception on

confluent-kafka-dotnet/src/Confluent.SchemaRegistry/Rest/RestService.cs

Line 233 in 007a8ba

catch (HttpRequestException e)

The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Confluent.SchemaRegistry.RestService.ExecuteOnOneInstanceAsync(Func`1 createRequest) in

Workarounds

• Disable the SSL Certificate Checking using the EnableSslCertificateVerification=false, which is not recommended at all.

• If you add the same file used in TRUSTED_CA_FILEPATH to the Trust Root Certificate Authorities in Windows Certificate Stores, you can execute the code with success. The problem is that in some cases (Azure Function Apps), you can't do this operation
  

Sugestion for solution

We could use the same approach as the Confluent.Kafka.ProducerConfig: being able to specify which certificate store you want to use:

confluent-kafka-dotnet/src/Confluent.Kafka/Config_gen.cs

Line 759 in 007a8ba

public string SslCaCertificateStores { get { return Get("ssl.ca.certificate.stores"); } set { this.SetObject("ssl.ca.certificate.stores", value); } }

From .NET 5 and on, it is possible to add new Certificate Stores to the HttpClientHandler
dotnet/runtime#39835 (comment)

I would be happy to send a PR implementing the suggestion or another approach.

Checklist

Please provide the following information:

• A complete (i.e., we can run it) minimal program demonstrating the problem. No need to supply a project file.
• Confluent.Kafka nuget version.
• Apache Kafka version.
• Client configuration.
• Operating system.
• Provide logs (with "debug" : "..." as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.

[ahmar-husain]

I too face the same !

[teimyBr]

any news about that when the pr getting merged ?