Different behaviour during SSL handshake between Windows and Linux

[Lukasz736]

Hello everybody.
We faced a problem in our Client – Server (dotnet – dotnet) application connected with certificate validation. Client try to connect Server by SslStream with delivered .pfx certificate. Server checks client’s certificate and accepts or decline connection.
We use SslStream.AuthenticateAsServer() method with RemoteCertificateValidationCallback. If validation fails we return false and expect that client app will close the connection with proper error. But errors are different when Server app runs on Windows and Linux.

If server app runs on Windows Client gets error:
Exception (IOException) with HResult -2146232800 (0x80131620)
InnerException (Win32Exception) with HResult -2147467259 (0x80004005) and NativeErrorCode -2146893017 (0x80090327) which means „An unknown error occurred while processing the certificate” and this is OK.

If server app runs on Linux Client gets error:
Exception (IOException) with HResult -214623280 (0x80131620)
InnerException (SocketException) with HResult -2147467259 (0x80004005) and NativeErrorCode 10054 which means „An existing connection was forcibly closed by the remote host.” and this is not OK.

Client app runs on Windows in both cases.

After some investigations we found out that in our RemoteCertificateValidation() method, paramter X509Chain contains different ChainStatuses for Windows and Linux.

On Windows we have one chain status:
NotSignatureValid with description "The signature of the certificate cannot be verified."

On Linux we have two chain statuses:
PartialChain with description "unable to get local issuer certificate"
HasNotSupportedCriticalExtension with description "unhandled critical extension".

These ChainStatuses are used in .net method SecureChannel.CreateFatalHandshakeAlertToken() which generates alertMessage sent to the client in SslStream.SendAuthResetSignal(). AlertToken is generated on Windows and SentAuthResetSignal() executes line InnerStream.Write() and then throws an exception, but on Linux token is empty (message.Size == 0) and SendAuthResetSignal() only throws an exception (in if’s body).

Client platform:
Windows 10 Pro x64 21H2
.NET SDK 6.0.1 x64

Server platform:
Raspberry PI OS, kernel 5.10
.NET SDK 6.0.1 Arm32
OpenSsl 1.1.1k

Any ideas what can cause this difference?

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hello everybody.
We faced a problem in our Client – Server (dotnet – dotnet) application connected with certificate validation. Client try to connect Server by SslStream with delivered .pfx certificate. Server checks client’s certificate and accepts or decline connection.
We use SslStream.AuthenticateAsServer() method with RemoteCertificateValidationCallback. If validation fails we return false and expect that client app will close the connection with proper error. But errors are different when Server app runs on Windows and Linux.

If server app runs on Windows Client gets error:
Exception (IOException) with HResult -2146232800 (0x80131620)
InnerException (Win32Exception) with HResult -2147467259 (0x80004005) and NativeErrorCode -2146893017 (0x80090327) which means „An unknown error occurred while processing the certificate” and this is OK.

If server app runs on Linux Client gets error:
Exception (IOException) with HResult -214623280 (0x80131620)
InnerException (SocketException) with HResult -2147467259 (0x80004005) and NativeErrorCode 10054 which means „An existing connection was forcibly closed by the remote host.” and this is not OK.

Client app runs on Windows in both cases.

After some investigations we found out that in our RemoteCertificateValidation() method, paramter X509Chain contains different ChainStatuses for Windows and Linux.

On Windows we have one chain status:
NotSignatureValid with description "The signature of the certificate cannot be verified."

On Linux we have two chain statuses:
PartialChain with description "unable to get local issuer certificate"
HasNotSupportedCriticalExtension with description "unhandled critical extension".

These ChainStatuses are used in .net method SecureChannel.CreateFatalHandshakeAlertToken() which generates alertMessage sent to the client in SslStream.SendAuthResetSignal(). AlertToken is generated on Windows and SentAuthResetSignal() executes line InnerStream.Write() and then throws an exception, but on Linux token is empty (message.Size == 0) and SendAuthResetSignal() only throws an exception (in if’s body).

Client platform:
Windows 10 Pro x64 21H2
.NET SDK 6.0.1 x64

Server platform:
Raspberry PI OS, kernel 5.10
.NET SDK 6.0.1 Arm32
OpenSsl 1.1.1k

Any ideas what can cause this difference?

Author:	Lukasz736
Assignees:	-
Labels:	area-System.Net.Security, untriaged
Milestone:	-

[wfurt]

You can possibly use SslStream.ShutdownAsync to send cleaner closer but the error code on client will be still different.
From the error, it seems like you may setting up your trust different way on each OS? You can use X509Chain directly to verify that the chain can be built.
And do you know what the "unhandled critical extension" is?
It may be worth of posting your cert (without private key)

[Lukasz736]

We can't use SslStream.ShutdownAsync(), because SSL connection is not established yet, we decline client's certificate.
What do you mean by "setting up your trust"? How can we set up trust on Windows or Linux?
We build (after Reset() ) X509Chain in our RemoteCertificateValidation() method. Build() returns false with same statuses I wrote in previous post.

We dont know what is "unhandled critical extension", thought you will know :d
We just know that client's certificate contains such extensions and all of them are critical.

Certificate is in attachment. Zip archive has password 'a123a'

PfxCert.zip

[karelz]

Triage: It would be nice to have consistency across platforms. However, this is fairly corner case with low priority as it does not block larger number of customers.

[rzikm]

The behavioral differences come from differences between underlying platforms.

RE: An existing connection was forcibly closed by the remote host, this should get fixed once we implement #18837.

The differences between the validation errors reported between SChannel/CAPI vs OpenSSL, we do not plan to unify and prefer to keep the platform behavior.

Since the first item is already covered by the mentioned issue, there is no need to keep this issue open, so I am going to close this. Feel free to reopen if you see a need.