mTLS handshake not working on Linux from .NET 5.0.15/6.0.x

[maxarm]

Description

I am trying to establish a communication from our client to a server using mTLS and therefore send certificates from client to the server. Server expects 2 certs (1 Leaf + 1 Intermediate) and I'm using the HttpClientHandler to send the certificates from a pem file.
This worked until .net 5.0.14; but stopped working with .net 5.0.15 and does not work with any .net 6.0.x version.
I can see on the Wireshark that only leaf certificate is written on the wire for .net 6.0.x.
For .NET 5.0.15ff even not a single mTLS handshake entry is visible in Wireshark.

Platform - Linux Container

This issue is hindering us to migrate to .NET 6.

Expected behavior

Client sends the leaf and intermediate certificate to the server and client/server communication works.

Actual behavior

The actual behavior was logged with Wireshark.

mcr.microsoft.com/dotnet/runtime:5.0.14

Client sends the leaf and intermediate certificate to the server when running in docker container and hence the communication works.

mcr.microsoft.com/dotnet/runtime:6.0.x

Client sends only the leaf certificate but no intermediate to the server when running in docker container resulting in an Unknown CA TLS error.

mcr.microsoft.com/dotnet/runtime:5.0.17

Client performs even no mTLS handshake when running in docker container resulting in failing communication.

Additional information

• the certificates of the pem file are installed in the docker container using update-ca-certificates

• .net 5.0.14 container runs "OpenSSL 1.1.1d 10 Sep 2019" whereas .net 5.0.15/.net 6.0.x containers run "OpenSSL 1.1.1n 15 Mar 2022"

• executing the following curl-command works in every of the 3 containers, performs a proper handshake and receives the response

    curl -E contains-all-3-certs-and-pk.pem https://the-server.com:443/ --insecure
  

  • without the --insecure switch the following error is logged in all 3 different containers

      curl: (60) SSL certificate problem: self signed certificate in certificate chain
    

• the client code is

        var clientCertificate = X509Certificate2.CreateFromPemFile(certificatePath);
        var chain = new X509Chain();
        chain.ChainPolicy.VerificationFlags = X509VerificationFlags.AllFlags;
        chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
        chain.ChainPolicy.DisableCertificateDownloads = false;
        var status = chain.Build(clientCertificate);
        //... dump status and element count ...
        var handler = new HttpClientHandler
            {
                UseProxy = false, 
                UseCookies = false, 
                AllowAutoRedirect = false, 
                SslProtocols = SslProtocols.Tls12, 
                CheckCertificateRevocationList = false, 
                AutomaticDecompression = DecompressionMethods.All,
                ClientCertificateOptions = ClientCertificateOption.Manual, 
                ClientCertificates = {clientCertificate}, 
                ServerCertificateCustomValidationCallback = (sender, certificate, chain, errors) => true
            };
        var response = await new HttpClient(handler, true)
            //... set base adress and content type ...
            .GetAsync(url);

• .net 5.0.14: X509Chain.Build(cert) results in

    ChainBuild.Status: True
    ChainBuild.ElementCount: 3
  

• .net 5.0.17/.net 6.0.x: X509Chain.Build(cert) results in

    ChainBuild.Status: False
    ChainBuild.ElementCount: 1
  

• .net 6.0.6 exception

The SSL connection could not be established, see inner exception.
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
    at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
    at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
    at Program.<Main>$(String[] args) in /src/Program.cs:line 25
Authentication failed, see inner exception.
    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
    at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
    at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca

• .net 5.0.17 exception

The SSL connection could not be established, see inner exception.
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
    at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
    at <Program>$.<<Main>$>d__0.MoveNext() in /src/Program.cs:line 28
 Authentication failed, see inner exception.
    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
 The type initializer for 'SslMethods' threw an exception.
    at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions)
    at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
    at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
 The type initializer for 'Ssl' threw an exception.
    at Interop.Ssl.SslV2_3Method()
    at Interop.Ssl.SslMethods..cctor()
 The type initializer for 'SslInitializer' threw an exception.
    at Interop.Ssl..cctor()
 error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib
    at Interop.SslInitializer..cctor()

• Certificates of PEM file

    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  9 21:49:37 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Authority Key Identifier:
                    keyid:83...
                X509v3 Key Usage: critical
                    Digital Signature, Key Agreement
                X509v3 Basic Constraints:
                    CA:FALSE
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  9 21:49:37 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Authority Key Identifier:
                    keyid:83...
                X509v3 Key Usage: critical
                    Digital Signature, Key Agreement
                X509v3 Basic Constraints:
                    CA:FALSE
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 1 (0x1)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  1 04:37:11 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Certificate Sign, CRL Sign
                X509v3 Basic Constraints: critical
                    CA:TRUE
                X509v3 Subject Key Identifier:
                    21...
                X509v3 Authority Key Identifier:
                    keyid:21...
                Authority Information Access:
                    CA Issuers - URI:...
                X509v3 CRL Distribution Points:
                    Full Name:
                    URI:...
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  1 04:37:11 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Certificate Sign, CRL Sign
                X509v3 Basic Constraints: critical
                    CA:TRUE, pathlen:0
                X509v3 Subject Key Identifier:
                    83...
                X509v3 Authority Key Identifier:
                    keyid:21...
                Authority Information Access:
                    CA Issuers - URI:...
                X509v3 CRL Distribution Points:
                    Full Name:
                    URI:...
        Signature Algorithm: ecdsa-with-SHA256
            ...
  

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I am trying to establish a communication from our client to a server using mTLS and therefore send certificates from client to the server. Server expects 2 certs (1 Leaf + 1 Intermediate) and I'm using the HttpClientHandler to send the certificates from a pem file.
This worked until .net 5.0.14; but stopped working with .net 5.0.15 and does not work with any .net 6.0.x version.
I can see on the Wireshark that only leaf certificate is written on the wire for .net 6.0.x.
For .NET 5.0.15ff even not a single mTLS handshake entry is visible in Wireshark.

Platform - Linux Container

This issue is hindering us to migrate to .NET 6.

Expected behavior

Client sends the leaf and intermediate certificate to the server and client/server communication works.

Actual behavior

The actual behavior was logged with Wireshark.

mcr.microsoft.com/dotnet/runtime:5.0.14

Client sends the leaf and intermediate certificate to the server when running in docker container and hence the communication works.

mcr.microsoft.com/dotnet/runtime:6.0.x

Client sends only the leaf certificate but no intermediate to the server when running in docker container resulting in an Unknown CA TLS error.

mcr.microsoft.com/dotnet/runtime:5.0.17

Client performs even no mTLS handshake when running in docker container resulting in failing communication.

Additional information

• the certificates of the pem file are installed in the docker container using update-ca-certificates

• .net 5.0.14 container runs "OpenSSL 1.1.1d 10 Sep 2019" whereas .net 5.0.15/.net 6.0.x containers run "OpenSSL 1.1.1n 15 Mar 2022"

• executing the following curl-command works in every of the 3 containers, performs a proper handshake and receives the response

    curl -E contains-all-3-certs-and-pk.pem https://the-server.com:443/ --insecure
  

  • without the --insecure switch the following error is logged in all 3 different containers

      curl: (60) SSL certificate problem: self signed certificate in certificate chain
    

• the client code is

    var clientCertificate = X509Certificate2.CreateFromPemFile(certificatePath);
    var chain = new X509Chain();
    chain.ChainPolicy.VerificationFlags = X509VerificationFlags.AllFlags;
    chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
    chain.ChainPolicy.DisableCertificateDownloads = false;
    var status = chain.Build(clientCertificate);
    ... dump status and element count ...
    var handler = new HttpClientHandler
        {
            UseProxy = false, 
            UseCookies = false, 
            AllowAutoRedirect = false, 
            SslProtocols = SslProtocols.Tls12, 
            CheckCertificateRevocationList = false, 
            AutomaticDecompression = DecompressionMethods.All,
            ClientCertificateOptions = ClientCertificateOption.Manual, 
            ClientCertificates = {clientCertificate}, 
            ServerCertificateCustomValidationCallback = (sender, certificate, chain, errors) => true
        };
    var response = await new HttpClient(handler, true)
        ... set base adress and content type ...
        .GetAsync(url);
  

• .net 5.0.14: X509Chain.Build(cert) results in

    ChainBuild.Status: True
    ChainBuild.ElementCount: 3
  

• .net 5.0.17/.net 6.0.x: X509Chain.Build(cert) results in

    ChainBuild.Status: False
    ChainBuild.ElementCount: 1
  

• .net 6.0.6 exception

The SSL connection could not be established, see inner exception.
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
    at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
    at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
    at Program.<Main>$(String[] args) in /src/Program.cs:line 25
Authentication failed, see inner exception.
    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
    at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
    at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca

• .net 5.0.17 exception

The SSL connection could not be established, see inner exception.
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
    at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
    at <Program>$.<<Main>$>d__0.MoveNext() in /src/Program.cs:line 28
 Authentication failed, see inner exception.
    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
 The type initializer for 'SslMethods' threw an exception.
    at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions)
    at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
    at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
 The type initializer for 'Ssl' threw an exception.
    at Interop.Ssl.SslV2_3Method()
    at Interop.Ssl.SslMethods..cctor()
 The type initializer for 'SslInitializer' threw an exception.
    at Interop.Ssl..cctor()
 error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib
    at Interop.SslInitializer..cctor()

• Certificates of PEM file

    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  9 21:49:37 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Authority Key Identifier:
                    keyid:83...
                X509v3 Key Usage: critical
                    Digital Signature, Key Agreement
                X509v3 Basic Constraints:
                    CA:FALSE
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  9 21:49:37 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Authority Key Identifier:
                    keyid:83...
                X509v3 Key Usage: critical
                    Digital Signature, Key Agreement
                X509v3 Basic Constraints:
                    CA:FALSE
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 1 (0x1)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  1 04:37:11 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Certificate Sign, CRL Sign
                X509v3 Basic Constraints: critical
                    CA:TRUE
                X509v3 Subject Key Identifier:
                    21...
                X509v3 Authority Key Identifier:
                    keyid:21...
                Authority Information Access:
                    CA Issuers - URI:...
                X509v3 CRL Distribution Points:
                    Full Name:
                    URI:...
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  1 04:37:11 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Certificate Sign, CRL Sign
                X509v3 Basic Constraints: critical
                    CA:TRUE, pathlen:0
                X509v3 Subject Key Identifier:
                    83...
                X509v3 Authority Key Identifier:
                    keyid:21...
                Authority Information Access:
                    CA Issuers - URI:...
                X509v3 CRL Distribution Points:
                    Full Name:
                    URI:...
        Signature Algorithm: ecdsa-with-SHA256
            ...
  

Author:	maxarm
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I am trying to establish a communication from our client to a server using mTLS and therefore send certificates from client to the server. Server expects 2 certs (1 Leaf + 1 Intermediate) and I'm using the HttpClientHandler to send the certificates from a pem file.
This worked until .net 5.0.14; but stopped working with .net 5.0.15 and does not work with any .net 6.0.x version.
I can see on the Wireshark that only leaf certificate is written on the wire for .net 6.0.x.
For .NET 5.0.15ff even not a single mTLS handshake entry is visible in Wireshark.

Platform - Linux Container

This issue is hindering us to migrate to .NET 6.

Expected behavior

Client sends the leaf and intermediate certificate to the server and client/server communication works.

Actual behavior

The actual behavior was logged with Wireshark.

mcr.microsoft.com/dotnet/runtime:5.0.14

Client sends the leaf and intermediate certificate to the server when running in docker container and hence the communication works.

mcr.microsoft.com/dotnet/runtime:6.0.x

Client sends only the leaf certificate but no intermediate to the server when running in docker container resulting in an Unknown CA TLS error.

mcr.microsoft.com/dotnet/runtime:5.0.17

Client performs even no mTLS handshake when running in docker container resulting in failing communication.

Additional information

• the certificates of the pem file are installed in the docker container using update-ca-certificates

• .net 5.0.14 container runs "OpenSSL 1.1.1d 10 Sep 2019" whereas .net 5.0.15/.net 6.0.x containers run "OpenSSL 1.1.1n 15 Mar 2022"

• executing the following curl-command works in every of the 3 containers, performs a proper handshake and receives the response

    curl -E contains-all-3-certs-and-pk.pem https://the-server.com:443/ --insecure
  

  • without the --insecure switch the following error is logged in all 3 different containers

      curl: (60) SSL certificate problem: self signed certificate in certificate chain
    

• the client code is

    var clientCertificate = X509Certificate2.CreateFromPemFile(certificatePath);
    var chain = new X509Chain();
    chain.ChainPolicy.VerificationFlags = X509VerificationFlags.AllFlags;
    chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
    chain.ChainPolicy.DisableCertificateDownloads = false;
    var status = chain.Build(clientCertificate);
    ... dump status and element count ...
    var handler = new HttpClientHandler
        {
            UseProxy = false, 
            UseCookies = false, 
            AllowAutoRedirect = false, 
            SslProtocols = SslProtocols.Tls12, 
            CheckCertificateRevocationList = false, 
            AutomaticDecompression = DecompressionMethods.All,
            ClientCertificateOptions = ClientCertificateOption.Manual, 
            ClientCertificates = {clientCertificate}, 
            ServerCertificateCustomValidationCallback = (sender, certificate, chain, errors) => true
        };
    var response = await new HttpClient(handler, true)
        ... set base adress and content type ...
        .GetAsync(url);
  

• .net 5.0.14: X509Chain.Build(cert) results in

    ChainBuild.Status: True
    ChainBuild.ElementCount: 3
  

• .net 5.0.17/.net 6.0.x: X509Chain.Build(cert) results in

    ChainBuild.Status: False
    ChainBuild.ElementCount: 1
  

• .net 6.0.6 exception

The SSL connection could not be established, see inner exception.
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
    at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
    at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
    at Program.<Main>$(String[] args) in /src/Program.cs:line 25
Authentication failed, see inner exception.
    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
    at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
    at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca

• .net 5.0.17 exception

The SSL connection could not be established, see inner exception.
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
    at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
    at <Program>$.<<Main>$>d__0.MoveNext() in /src/Program.cs:line 28
 Authentication failed, see inner exception.
    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
 The type initializer for 'SslMethods' threw an exception.
    at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions)
    at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
    at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
 The type initializer for 'Ssl' threw an exception.
    at Interop.Ssl.SslV2_3Method()
    at Interop.Ssl.SslMethods..cctor()
 The type initializer for 'SslInitializer' threw an exception.
    at Interop.Ssl..cctor()
 error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib
    at Interop.SslInitializer..cctor()

• Certificates of PEM file

    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  9 21:49:37 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Authority Key Identifier:
                    keyid:83...
                X509v3 Key Usage: critical
                    Digital Signature, Key Agreement
                X509v3 Basic Constraints:
                    CA:FALSE
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  9 21:49:37 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Authority Key Identifier:
                    keyid:83...
                X509v3 Key Usage: critical
                    Digital Signature, Key Agreement
                X509v3 Basic Constraints:
                    CA:FALSE
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 1 (0x1)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  1 04:37:11 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Certificate Sign, CRL Sign
                X509v3 Basic Constraints: critical
                    CA:TRUE
                X509v3 Subject Key Identifier:
                    21...
                X509v3 Authority Key Identifier:
                    keyid:21...
                Authority Information Access:
                    CA Issuers - URI:...
                X509v3 CRL Distribution Points:
                    Full Name:
                    URI:...
        Signature Algorithm: ecdsa-with-SHA256
            ...
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number: 2 (0x2)
            Signature Algorithm: ecdsa-with-SHA256
            Issuer:
            Validity
                Not Before: Dec  1 04:37:11 2020 GMT
                Not After : Dec 31 23:59:59 9999 GMT
            Subject:
            Subject Public Key Info:
                Public Key Algorithm: id-ecPublicKey
                    Public-Key: (256 bit)
                    pub:
                        ...
                    ASN1 OID: prime256v1
                    NIST CURVE: P-256
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Certificate Sign, CRL Sign
                X509v3 Basic Constraints: critical
                    CA:TRUE, pathlen:0
                X509v3 Subject Key Identifier:
                    83...
                X509v3 Authority Key Identifier:
                    keyid:21...
                Authority Information Access:
                    CA Issuers - URI:...
                X509v3 CRL Distribution Points:
                    Full Name:
                    URI:...
        Signature Algorithm: ecdsa-with-SHA256
            ...
  

Author:	maxarm
Assignees:	-
Labels:	area-System.Security, untriaged
Milestone:	-

[wfurt]

What is your intermediate and root certificate?

[maxarm]

intermediate is the 4th in above's list
root is the 3rd in above's list
the 1st and 2nd are equal (redundant?) and are the server certificates

[maxarm]

@wfurt did you have any time to investigate further. Or any workaround ideas?

I tried again with a new docker image with .net 6.0.7 instead of 6.0.6. However, still received the error "tlsv1 alert unknown ca". One difference compared to 6.0.6 however was that now the ChainBuild.Status was true instead of false.

[wfurt]

Sorry for the delay @maxarm, is this still problematic for you? Is there any chance you can try 8.0? There is now API that allows you to specify the intermediates explicitly. (and it is more reliable and performant)