libraries behave badly with difficulties in linux certificate store

[jrvjrv]

$ dotnet --version
2.1.302

I created a new mvc project, started Visual Studio Code and tried to run it with F5. I got the following exceptions:

dbug: HttpsConnectionAdapter[1]
Failed to authenticate HTTPS connection.
System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+Crypto+OpenSslCryptographicException: error:2006D080:BIO routines:BIO_new_file:no such file
at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet1 downloaded, HashSet1 systemTrusted, TimeSpan& remainingDownloadTime)
at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
at System.Net.Http.TLSCertificateExtensions.BuildNewChain(X509Certificate2 certificate, Boolean includeClientApplicationPolicy)
at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
--- End of inner exception stack trace ---
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.BeginAuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.b__51_0(SslServerAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1,TArg2](Func5 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsServerAsync(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionAdapter.InnerOnConnectionAsync(ConnectionAdapterContext context)
HttpsConnectionAdapter:Debug: Failed to authenticate HTTPS connection.

System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+Crypto+OpenSslCryptographicException: error:2006D080:BIO routines:BIO_new_file:no such file
at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet1 downloaded, HashSet1 systemTrusted, TimeSpan& remainingDownloadTime)
at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
at System.Net.Http.TLSCertificateExtensions.BuildNewChain(X509Certificate2 certificate, Boolean includeClientApplicationPolicy)
at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
--- End of inner exception stack trace ---
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.BeginAuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.b__51_0(SslServerAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1,TArg2](Func5 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsServerAsync(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionAdapter.InnerOnConnectionAsync(ConnectionAdapterContext context)
dbug: HttpsConnectionAdapter[1]
Failed to authenticate HTTPS connection.
System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+Crypto+OpenSslCryptographicException: error:2006D080:BIO routines:BIO_new_file:no such file
at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet1 downloaded, HashSet1 systemTrusted, TimeSpan& remainingDownloadTime)
at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
at System.Net.Http.TLSCertificateExtensions.BuildNewChain(X509Certificate2 certificate, Boolean includeClientApplicationPolicy)
at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
--- End of inner exception stack trace ---
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.BeginAuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.b__51_0(SslServerAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1,TArg2](Func5 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsServerAsync(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionAdapter.InnerOnConnectionAsync(ConnectionAdapterContext context)
HttpsConnectionAdapter:Debug: Failed to authenticate HTTPS connection.

System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+Crypto+OpenSslCryptographicException: error:2006D080:BIO routines:BIO_new_file:no such file
at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet1 downloaded, HashSet1 systemTrusted, TimeSpan& remainingDownloadTime)
at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
at System.Net.Http.TLSCertificateExtensions.BuildNewChain(X509Certificate2 certificate, Boolean includeClientApplicationPolicy)
at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
--- End of inner exception stack trace ---
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.BeginAuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.SslStream.<>c.b__51_0(SslServerAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1,TArg2](Func5 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func5 beginMethod, Action1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
at System.Net.Security.SslStream.AuthenticateAsServerAsync(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionAdapter.InnerOnConnectionAsync(ConnectionAdapterContext context)

analysis

The problem seemed to stem from "corruption" in the linux (ubuntu 16.04) ssl certificate store in /etc/ssl/certs. In particular there were several symbolic links where the underlying target did not exist. This did not seem to be caused by dotnet, but dotnet reacted to it very badly.

re-create

To re-create (ubuntu; I am assuming there is no existing files foo.pem or 00000001.0):

• sudo su
• cd /etc/ssl/certs
• touch foo.pem
• ln -s foo.pem 00000001.0
• rm foo.pem
• run website that has app.UseHttpsRedirection(); in Startup.Configure() on a machine with a devcert configured: dotnet dev-certs https

I speculate that the code is looping through the existing certificates and encounters the file error.

expected behavior

I believe the dotnet code should either log a warning when it has trouble reading a particular certificate or it should ignore it altogether. If it completes its looping without successfully finding whatever information it needed, it can throw an appropriate and more meaningful exception then.

determine if you have the same issue

I originally used this code https://github.com/dotnet/corefx/issues/29942#issuecomment-397207557 to diagnose the problem, but the issues were also obvious when I did a ls -l /etc/ssl/certs. The broken links show up in red in bash. One might also try "find /etc/ssl/certs -xtype l". This won't find permission issues as in other bugs, but it does find broken links.

possible related issues

I had what seems to be a related problem when I tried to add a package while investigating the problem:
dotnet add package Microsoft.AspNetCore.Server.Kestrel.Https
Writing /tmp/tmpxrF8CW.tmp
info : Adding PackageReference for package 'Microsoft.AspNetCore.Server.Kestrel.Https' into project '/home/jrv/Documents/projects/20180711 signalr/signalr/signalrlab/signalrlab.csproj'.
log : Restoring packages for /home/jrv/Documents/projects/20180711 signalr/signalr/signalrlab/signalrlab.csproj...
error: Unable to load the service index for source https://api.nuget.org/v3/index.json.
error: The SSL connection could not be established, see inner exception.
error: error:2006D080:BIO routines:BIO_new_file:no such file

There are several other bugs that seem somewhat similar and are also perhaps related:
https://github.com/dotnet/corefx/issues/30385
https://github.com/dotnet/corefx/issues/29942

solution

All of my issues went away when I fixed the links in /etc/ssl/certs.

[tmds]

This is fixed in the 2.1.3 servicing release (https://github.com/dotnet/corefx/issues/29942).

[jrvjrv]

@tmds

This is fixed in the 2.1.3 servicing release (#29942).

I saw that, but I was unsure if that fix was included in 2.1.302, which is what I was running. If it was, that fix did not solve the problem. If the fix was not included in 2.1.302, then I am guessing the problem is probably fixed by that code, but just from an eyeball inspection.

[tmds]

If the fix was not included in 2.1.302

The fix is not included. sdk version 2.1.302 is runtime version 2.1.2.

[jrvjrv]

Based on my read of the submitted fix, I think it will fix the problem. I am closing this issue.