DotNet HttpClient doesn't give SSL Error While curl and Java fails

[mddubey]

Hello Team,

We are using DotNet to connect to one of our APIs which is using Let's Encrypt Certificate. There are some issues in the certificate and it is giving SSL issues when we try to connect using curl or java or wget but it works quite fine with DotNet HttpClient. Isn't this a security issue because it might be trusting a certificate which it should not?

Please find the code/command and errors from different tools below:

Java-11:

        HttpURLConnection connection = null;
        try {
            URL url = new URL(String.format("https://customer-portal.ngl.com/abc"));
            connection = (HttpURLConnection) url.openConnection();
            connection.setRequestMethod("GET");
            System.out.println(connection.getResponseCode());
        } finally {
            if (connection != null) {
                connection.disconnect();
            }
        }

Error Message:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Curl

curl https://customer-portal.ngl.com/abc -vv

Error Message:

curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html

Wget

wget https://customer-portal.ngl.com/abc

Error Message:

ERROR: The certificate of 'customer-portal.ngl.com' is not trusted.
ERROR: The certificate of 'customer-portal.ngl.com' doesn't have a known issuer.

DotNet

var httpClient = new HttpClient();
HttpResponseMessage response = await httpClient.GetAsync("https://customer-portal.ngl.com/abc");
Console.WriteLine(response);

dotnet script some.csx

Output:

StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
Connection: keep-alive
X-Powered-By: JSP/2.3
Date: Thu, 08 Jul 2021 09:25:51 GMT
Content-Type: text/html; charset=ISO-8859-1
Content-Length: 110
}

Details:
OS Docker Image From: mcr.microsoft.com/dotnet/core/sdk:3.1.100
Dotnet version: 3.1.100

Question we have here, is what is Dotnet doing special which other platforms are missing and is it a security threat in the my application?

Please let us know.
Thank you

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hello Team,

We are using DotNet to connect to one of our APIs which is using Let's Encrypt Certificate. There are some issues in the certificate and it is giving SSL issues when we try to connect using curl or java or wget but it works quite fine with DotNet HttpClient. Isn't this a security issue because it might be trusting a certificate which it should not?

Please find the code/command and errors from different tools below:

Java-11:

        HttpURLConnection connection = null;
        try {
            URL url = new URL(String.format("https://customer-portal.ngl.com/abc"));
            connection = (HttpURLConnection) url.openConnection();
            connection.setRequestMethod("GET");
            System.out.println(connection.getResponseCode());
        } finally {
            if (connection != null) {
                connection.disconnect();
            }
        }

Error Message:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Curl

curl https://customer-portal.ngl.com/abc -vv

Error Message:

curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html

Wget

wget https://customer-portal.ngl.com/abc

Error Message:

ERROR: The certificate of 'customer-portal.ngl.com' is not trusted.
ERROR: The certificate of 'customer-portal.ngl.com' doesn't have a known issuer.

DotNet

var httpClient = new HttpClient();
HttpResponseMessage response = await httpClient.GetAsync("https://customer-portal.ngl.com/abc");
Console.WriteLine(response);

dotnet script some.csx

Output:

StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
Connection: keep-alive
X-Powered-By: JSP/2.3
Date: Thu, 08 Jul 2021 09:25:51 GMT
Content-Type: text/html; charset=ISO-8859-1
Content-Length: 110
}

Details:
OS Docker Image From: mcr.microsoft.com/dotnet/core/sdk:3.1.100
Dotnet version: 3.1.100

Question we have here, is what is Dotnet doing special which other platforms are missing and is it a security threat in the my application?

Please let us know.
Thank you

Author:	mddubey
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[wfurt]

To verify the certificate, you need full chain to the trusted root. Server is supposed to send everything - 1. You should see that in ServerHello/Certificate.
.NET has code to cache intermediates as well as code to try to download them is missing (based on CA distribution point from the certificate).
I don't know about Java but Curl and base OpenSSL do not have code AFAIK to fetch the missing links.
You can register custom callback to see the chain as well you can inspect the handshake @mddubey to see if server is sending everything as it should.

[mddubey]

Hello @wfurt,

Thanks for your input. So I did some reading and have the understanding that there are 3 layers of certificates:

• CA root certificate
• Layer of intermediate certificates
• The End Server Certificate

Ideally a server sends all the certificates barring the CA Root Certificate because that is generally known to everyone. In case the intermediate certificates are not sent the DotNet platform fetches it over the internet from different repositories/proxies and stores locally. The other platforms(java/curl/wget) might not be that smart. So with that understanding I know why the above situation is happening.

However, we are facing one more issue. One of our envs this is not working and reason is because we don't have internet in that environment. Since it is a non-prod environment we were okay to turn the SSL verification off, but turned out even after ignoring SSL verification there is need of internet somewhere (like checking revoked certificates). We turned that off too, but still there is some problem while making SSL connection.

Below is the code we are using:

var httpClientHandler = new HttpClientHandler
            {
                ClientCertificateOptions = ClientCertificateOption.Manual,
                CheckCertificateRevocationList = false,
                ServerCertificateCustomValidationCallback = (httpRequestMessage, cert, cetChain, policyErrors) =>
                {
                    return true;
                }
            };
            client = new HttpClient(httpClientHandler);
            await client
                    .SendAsync(request)
                    .ConfigureAwait(false);

It times-out after 30 seconds or so with error Below:

>[15:15:36 ERR] Error Occured System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

Feels like it still requires internet for some check/reason. Is there a way to turn off SSL verification for a client which has no access to internet? What all settings should be turned off.

Thanks for all your help.

[wfurt]

You can try packet capture to se what is happening. The validation code will still try to get the missing parts and there is no good way how to avoid it at the moment. If set of sites you need to access is small, you can put all the missing intermediates on the system. See #55368 for some example.

[mddubey]

Hi @wfurt,

Thanks for the help. Ideally I would have liked to turn the whole thing off because these are let's encrypt certificate, and server will be renewing it every few months and we will run into similar problem. We don't have much control on the server.

I can think of two more ways if possible it will be super helpful to automate:

• Is there a way to get it from other envs where we have internet? Like if the dotnet is fetching the chain from internet and saving it somewhere then I could dump all these to a cer files and use it in the other envs. Probably the custom callback could be the way to go because X509Chain is passed in the callback. What is your opinion about this?

• If above doesn't work is there a way to do the same using openssl or something which could be automated?

Thank you for your help.

[wfurt]

.NET will save certificate cache on exit. So in theory one should be able to copy that to other machine(s). However that feels fragile and I would not recommend it. (it also depends on internal behavior that can possible change)
If anything, I would wrote small app that puts the intermediates to StoreName.CertificateAuthority.

There was some discussion and desire to provide better control over certificate validation but that is not going to happen in 6. (and would not meed bar for servicing anyway)

[wfurt]

I hope this is answered @mddubey.