dev-certificates on Linux -- how to get dotnet-to-dotnet comms to work?

[fluffynuts]

Description

I have an asp.net core application with a self-hosted IdentityServer component. I can convince a browser (Chrome and Firefox) to visit the site, but I'm having trouble getting the IdentityServer component to be able to talk back to localhost due to certificate errors, specifically:

fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
      Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:44303/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:44303/.well-known/openid-configuration'. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

I realise that certificate installation can differ from distro to distro, however, I've tried this first on my (more native) Gentoo, and also in an Ubuntu vm:

1. export the dev cert to a pfx with dotnet dev-certs https -ep /tmp/dnc.pfx -p SomePassword
2. extract the crt from the pfx with openssl pkcs12 -in /tmp/dnc.pfx -clcerts -nokeys -out /tmp/dnc.crt
3. copy to /usr/local/share/ca-certificates and run update-ca-certificates. In both cases (Gentoo and Ubuntu), see a message about 1 added certificate
   3.5 in the case of Gentoo, I was also able to use certutil to install the crt into my ~/.pki/nssdb; in the case of Ubuntu, this dir does not exist and I'm not sure how to bootstrap it
4. Fire up the asp.net site with angular frontend -- on Gentoo, where certutil worked, Chrome happily visits the site; on both (iirc) I had to add a security exception for Firefox and store, but could get to the site.
5. Attempts to log in fail because all auth requests go through the primary api, which then attempts to contact the same host/port (localhost:44303) for IdentityServer comms -- and that's where the failure occurs with the message above: I can get no further

To Reproduce

Set up an asp.net core project with self-hosted IdentityServer on Linux, such that the config for identity server points back to the same hosting application, using dev certs. Any route requiring authorization will result in a 500 error with the above logging in place. Using dotnet dev-certs https --trust on Windows, same code, no problem.

Expected behavior

I expect to be able to get dotnet-to-dotnet comms working with the dev cert. I'm probably missing something )':

Ubuntu dotnet info:
.NET Core SDK (reflecting any global.json):
Version: 2.2.103
Commit: 8edbc2570a

Runtime Environment:
OS Name: ubuntu
OS Version: 18.10
OS Platform: Linux
RID: ubuntu.18.10-x64
Base Path: /usr/share/dotnet/sdk/2.2.103/

Host (useful for support):
Version: 2.2.1
Commit: 878dd11e62

.NET Core SDKs installed:
2.2.103 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Gentoo dotnet info:
.NET Core SDK (reflecting any global.json):
Version: 2.2.103
Commit: 8edbc2570a

Runtime Environment:
OS Name: ubuntu
OS Version: 18.10
OS Platform: Linux
RID: ubuntu.18.10-x64
Base Path: /usr/share/dotnet/sdk/2.2.103/

Host (useful for support):
Version: 2.2.1
Commit: 878dd11e62

.NET Core SDKs installed:
2.2.103 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

I have also tried with dotnet binaries from https://dotnet.microsoft.com/download/thank-you/dotnet-sdk-2.2.103-linux-x64-binaries

[fluffynuts]

If it would be helpful, I can possibly set up a self-hosting IdentityServer project at some point, given a little time. I can't supply the full code I'm working on because it's for a client.

[maq1823]

@fluffynuts Were you able to resolve this in any way?
I have the same exact problem.

[fluffynuts]

@maq1823 no, I didn't )': I've read that this is an issue within dotnet itself (specifically to do with cert trusting on linux), but I don't know any more about it. I eventually had to stop development under linux and boot into windows as this was a show-stopper preventing any progress on the project.
AFAIK, the only solution right now would be to have a proper certificate issued from an authority and use a hosts file redirection for that domain.

[nickcoad]

I'm having the same issue - I can find a lot of details for how to enable host-to-container SSL trust, but not for container-to-container trust.

[sntnupl]

Facing the same issue. I am trying to setup ASP.NET Core 3.0 Web Api, with self-hosted OpenIddict middleware, for handling OAuth token generation.

In Windows, things work fine, even with dotnet generated TLS certificate.
But things fall apart in Linux environment - I tried with WSL (ubuntu 18.04), as well as in a Ubuntu 18.04 VM. I created local self-signed certificate and setup the system to trust it - but still the issue persists.

There is no issue while trying to access the application URL, Token URL etc.
Once I get the token, and then make a cURL call to one of the protected API endpoints, application tries to internally talk to https://localhost:44320/.well-known/openid-configuration to get the OAuth configuration, and BOOM, it fails estabilish TLS session, as it thinks that the remote certificate is inavlid.

fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
      Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:44320/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:44320/.well-known/openid-configuration'. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)

This is blocking development/deployment of my project in Linux environment. Would really appreciate some one looking/responding to it.

[poke]

Is this about development time or when deploying for production? The dev certs probably shouldn't be used for production purposes.

[sntnupl]

@poke development time. Unless the use cases work we can't be even thinking about prod.

[blowdart]

https://github.com/dotnet/corefx/issues/40725

You need to trust it in libcurl I believe. Regardless, it's a corefx issue not an aspnet one.

[sntnupl]

@blowdart I retried by asking libcurl to trust the self signed certificate:

curl -X --insecure --cacert path/to/self-signed.crt GET \
  https://localhost:44320/api/somesecureaction \
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer <token>'

But the issue still happens:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 --insecure https://localhost:44320/api/Account/UserData  
fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
      Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:44320/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:44320/.well-known/openid-configuration'. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()

The problem does not seem to related to libcUrl though, since the following works fine:

curl -X GET https://localhost:44320/.well-known/openid-configuration

The problem happens when JWT middleware internally tries to fetch config from OAuth endpoint, by making HTTPS call to: https://localhost:44320/.well-known/openid-configuration.
I have already setup ASPNET to use my self-signed cert:

       public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder => {
                    webBuilder.UseKestrel(options => {

                        var certificate = new X509Certificate2("localhost.pfx", "<password>");
                        options.Listen(IPAddress.Loopback, 44320, listenOptions => {
                            listenOptions.UseHttps(certificate);
                        });
                    });
                    webBuilder.UseStartup<Startup>();
                });

But somehow that does not stop JWT middleware to throw the error.

[blowdart]

@terrajobst this appears to be a corefx problem rather than aspnet core. Can someone explain how to get corefx to trust a self signed cert?

[terrajobst]

this appears to be a corefx problem rather than aspnet core

That's fair but before we move it let's first verify (because we can't transfer directly and we'll lose some context).

@davidsh @bartonjs could one of you take a look?

[davidsh]

@davidsh @bartonjs could one of you take a look?

I don't understand what the repro is exactly here. Can you attach the dev certificate so we can look at it?

Can someone explain how to get corefx to trust a self signed cert?

The problem is most likely not a "trust" issue. We have see that installing self-signed certificates into the trust list on Linux doesn't work because the certificate itself is usually missing some X509v3 attributes that Linux (due to OpenSssl) require but Windows doesn't. OpenSsl on Linux requires that self-signed certificates have the "Cert Signing" attribute on them. This is because a self-signed, trusted certificate has to be both a valid CA certificate (requires 'Cert Signing') as well as a "leaf node" certificate.

[sntnupl]

@davidsh please find attached the dev certificate file with which the issue happens.
dev-cert.zip

It would also be very helpful if you could point to any article where the proper step for generating self-signed certificates are provided, that would make linux/openssl happy.