SocketsHttpHandler does not take into acount new certificate authorities

[Tratcher]

From @busesorin94 on October 2, 2018 11:17

This issue only happens on .NET Core 2.1.X, when using DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=1.
Steps to reproduce:

1. Have a running application which calls a service using HTTPS for which the certificate authority is not trusted on the machine
2. Install the new authority on the machine and run update-ca-certificates
3. Call the service from the running .NET application

Expected behavior:
The application should take into account the newly installed certificate authority, without restarting the application

Actual behavior:
An exception is thrown:
System.Net.Http.HttpRequestException: The SSL connection could not be established

Workaround:

1. You can restart the application after you run the update-ca-certificates command
2. Add the environment variable DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0

Copied from original issue: aspnet/KestrelHttpServer#2972

[stephentoub]

cc: @bartonjs

[bartonjs]

This is known / intentional / by design. The root trust list changes very rarely (particularly since .NET has no programmatic access to it), so as a performance optimization the contents are only read at most once per process.

We could, of course, change that model, but it would make all cert chain operations need to do a whole lot more IO, which would slow down chain building.

[davidsh]

@karelz

[davidsh]

Workaround:
Add the environment variable DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0

@bartonjs Why would this work with WinHttpHandler (WinHTTP). Is it because they use a different algorithm for validating the certificates and trust roots etc.?

[bartonjs]

Why would this work with WinHttpHandler (WinHTTP)

I assume it's CurlHandler. And it's because we let curl tell us that the TLS backend said it was okay. Once curl isn't involved, it's just managed code, so it's our chain builder, and the cached trust list applies.

[davidsh]

I assume it's CurlHandler.

The repro didn't say explicitly whether it was Windows or Linux. By I guess this comment:

Install the new authority on the machine and run update-ca-certificates

means Linux?

[karelz]

I agree that it sounds very corner case and looks like by design.
@busesorin94 do you have any specific scenarios which depend on the original curl behavior?

[busesorin94]

I've reproduced this using a docker container hosted application (from microsoft/dotnet:2.1-aspnetcore-runtime).
My scenario is that we have an integration application which needs to deploy some custom authorities to docker services (E.g. after we set up a proxy for our system, we need to send the proxy certificate to docker services). We'd like to do that without having to kill all the running containers. That would imply having downtime for the system and also it would increase the time needed to deploy the certificates.

[karelz]

I think we have these options:

1. Check root trust list more often -- This has perf impact on all scenarios, therefore it is not desirable, unless these are more scenarios & users who need it
2. Subscribe to (file system?) changes and refresh it when it changes -- @bartonjs is that an option?
3. Let users refresh root trust list explicitly (e.g. new API or side effect of existing one) -- Adding a new API for corner-case scenario sounds like overkill
4. Keep it as it is and ask users to restart their app (e.g. detect the root trust list changes by themselves)

[bartonjs]

Option 2 is a no-go, we have decided to not depend on FileSystemWatcher within the shared framework; and it's only semi-reliable.

Option 3 is also not very likely, it's API that wouldn't have any effect on Windows or macOS, and that makes it hard to use.

That leaves it at 1 or 4.

@busesorin94 It seems like the custom trust should be known before the container starts, so it seems like it's part of container setup, no?

[busesorin94]

I'm not sure I understood your question but I'll try to come up with an answer by describing how we register new CAs.
Firstly, a folder containing custom trust is mounted to the container. The entrypoint of the container is "update-ca-certificates && dotnet run X.dll". This assured us that at every restart of the container is, the custom CA list is trusted by the container OS.
When we are dynamically updating the trusts, we are copying the CA into that folder and we expect that running "update-ca-certificates" will update the trust list system-wide and the .NET Core application should be aware of that change.
I can't be the only one that has this problem in a containerized/cloud-ready world.

[bartonjs]

How often are you updating your CA? Usually a CA is good for several years, and is usually managed as "start sending this CA as trusted now, and we'll start issuing certs from it in a month or so", letting normal product update deployments happen between those events.

[busesorin94]

I find this very used in a microservice based architecture. Think of a core application which's functionality is expanded with plugins. Every plugin could come with its own CA which should be trusted by other plugins or by the core itself.
We can wait a while to see if there are other people who want this feature. I can use DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER for now. Do you know if there any plans to remove the old HTTP handler soon?

[geoffkizer]

How about option 5: If the CA check fails, check if the files have been updated, and if so, re-read and re-check. This would only impact perf in failure cases.

[Tratcher]

As long as that check was rate limited to once per minute or hour.