.Net 6/Win11 HttpClient HandshakeFailure TLS issue

[awdorrin]

Description

I recently updated my laptop to Windows 11 and while developing a ASP.Net Core application running on .Net 6, I ran into an issue with an HttpClient service that had been working fine on Windows 10.

It appears that Windows 11 prefers TLS 1.3 while the server (Pivotal Cloud Foundry/Tanzu) is limited to TLS 1.2.
Attempts to connect from the code running on Windows 11 results in a HandshakeFailure exception during the service's query via HttpClient.

I was able to force TLS1.2 with the following code:

            services.AddHttpClient<MyService>()
                .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
                {
                    SslProtocols = SslProtocols.Tls12
                });

I hate hard coding to TLS 1.2, because eventually the server environment will support TLS 1.3.
I did try setting the SslProtocols to: SslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13
however that brought back the HandshakeFailure exception.

Reproduction Steps

See descrption

Expected behavior

Windows 11 HttpClient should be able to down shift to TLS 1.2

Actual behavior

HandshakeFailure exception when HttpClient makes a request to a server limited to TLS 1.2

Regression?

This works on Windows 10/Linux clients.

Known Workarounds

Force SslProtocols.Tls12

Configuration

Windows 11, .Net 6.0.10

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I recently updated my laptop to Windows 11 and while developing a ASP.Net Core application running on .Net 6, I ran into an issue with an HttpClient service that had been working fine on Windows 10.

It appears that Windows 11 prefers TLS 1.3 while the server (Pivotal Cloud Foundry/Tanzu) is limited to TLS 1.2.
Attempts to connect from the code running on Windows 11 results in a HandshakeFailure exception during the service's query via HttpClient.

I was able to force TLS1.2 with the following code:

            services.AddHttpClient<MyService>()
                .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
                {
                    SslProtocols = SslProtocols.Tls12
                });

I hate hard coding to TLS 1.2, because eventually the server environment will support TLS 1.3.
I did try setting the SslProtocols to: SslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13
however that brought back the HandshakeFailure exception.

Reproduction Steps

See descrption

Expected behavior

Windows 11 HttpClient should be able to down shift to TLS 1.2

Actual behavior

HandshakeFailure exception when HttpClient makes a request to a server limited to TLS 1.2

Regression?

This works on Windows 10/Linux clients.

Known Workarounds

Force SslProtocols.Tls12

Configuration

Windows 11, .Net 6.0.10

Other information

No response

Author:	awdorrin
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I recently updated my laptop to Windows 11 and while developing a ASP.Net Core application running on .Net 6, I ran into an issue with an HttpClient service that had been working fine on Windows 10.

It appears that Windows 11 prefers TLS 1.3 while the server (Pivotal Cloud Foundry/Tanzu) is limited to TLS 1.2.
Attempts to connect from the code running on Windows 11 results in a HandshakeFailure exception during the service's query via HttpClient.

I was able to force TLS1.2 with the following code:

            services.AddHttpClient<MyService>()
                .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
                {
                    SslProtocols = SslProtocols.Tls12
                });

I hate hard coding to TLS 1.2, because eventually the server environment will support TLS 1.3.
I did try setting the SslProtocols to: SslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13
however that brought back the HandshakeFailure exception.

Reproduction Steps

See descrption

Expected behavior

Windows 11 HttpClient should be able to down shift to TLS 1.2

Actual behavior

HandshakeFailure exception when HttpClient makes a request to a server limited to TLS 1.2

Regression?

This works on Windows 10/Linux clients.

Known Workarounds

Force SslProtocols.Tls12

Configuration

Windows 11, .Net 6.0.10

Other information

No response

Author:	awdorrin
Assignees:	-
Labels:	area-System.Net.Security, untriaged
Milestone:	-

[wfurt]

Can you share packet captures @awdorrin? This should work as long as there is overlap. It is the server who picks the version and cipher suite.
https://learn.microsoft.com/en-us/windows/win32/secauthn/tls-cipher-suites-in-windows-11

[awdorrin]

Wireshark kept locking up on me but I believe I have the exchange captured.
I guess I was wrong about TLSv1.3 being the issue, as I only see TLSv1.2 protocols in the capture.
I am not familiar with the protocol, so hopefully you can make sense of the cause of the Handshake Failure. :-)
HandshakeFailure.zip

[wfurt]

Hello offers 4 versions in supported_versions extension and 22 cipher suites. I think you should collect capture that works and check if that cipher suited is offered in the ClientHello. (there are other ways but this is simplest one)
Among other things Windows 11 (And server 22) retired some sets deemed war from the default set. It is possible the problem lives there is the server only support old and weak options.
From Linux/macOS you can also try openssl s_client -connect <server:port> to see what cipher is negotiated and again check if that is offered in the ClientHello.

[awdorrin]

I captured packets for a successful request, after adding back the CofnigurePrimaryHttpMessageHandler code that specifies the SslProtocol as Tls12. In that exchange, it appears the server and client negotiate
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)

That same protocol is listed in the exchange where the handshake fails.

[wfurt]

Do you know what OS and SSL is running on the server? And if you can check, is there any error? The handshake fails immediately after client sends the ClientHello (and server response with fatal alert) This looks like general interop issue with Schannel. Since the failure is on the server, I don't think collecting Schannel traces on client client would help.

[awdorrin]

The server app is implemented by another team within my company. I may be able to find out more details, but all I know it is running within a Pivotal Cloud Foundry server instance. I believe it is a Linux buildpack.

If it were something in the server configuration, I just don't understand how setting the SslProtocol in the client would make it work cleaner?

What I did just notice, is in the second 'Client Hello' packet, which the package capture claims is TLSv1.2 protocol. That when you drill into the TLSv1.2 record, it still states that the Version is TLS 1.0 (0x0301)

This makes me think that .Net is not building the 'Client Hello' packet correctly when negotiating after the first TLSv1 ClientHello packet fails, as I would expect that internal version to read TLS 1.2 (0x0303)

v Transport Layer Security
  v TLSv1.2 Record Layer: Handshake Protocol: Client Hello
    Content Type: Handshake (22)
    Version: TLS 1.0 (0x0301)
    Length: 273

In the capture where I enable the SslProtocol.Tls12, that section of the packet reads:

v Transport Layer Security
  v TLSv1.2 Record Layer: Handshake Protocol: Client Hello
    Content Type: Handshake (22)
    Version: TLS 1.2 (0x0303)
    Length: 178

I'm thinking that those bytes (03 01) are wrong, and should be (03 03) but something isn't populating the field correctly in the second attempt.
Since it works when I set the SslProtocol.Tls12, it makes me think there is an issue in the renegotiation logic, when it moves from TLSv1 to TLSv1.2.
Of course I'm only guessing.

[wfurt]

The hello is not created by .NET. We only call Windows functions to make it happen. The versions are more complicated. There is version in the frame header, then the handshake as well as there is extension showing all supported versions. Typically all the versions would be set low as much as practical and server picks the stronger option.

I don't know how the posted capture correspond to the text. But the second hello is on different TCP connection. And according to the capture it happens before first one fails. It may be worth of creating simples possible repro with just HttpClient for the experiments and investigation. Unless you can reproduce it agains some generic Linux or public server this is really not actionable for us.

If the server is reachable from Internet you can try something like https://www.ssllabs.com/ssltest/ to gather more information about supported capabilities.

[awdorrin]

Here are two more capture files. One that works with SslProtocol.Tls12 set, and the other without that setting, that fails. It doesn't jump IPs (which I think may have been caused by a load balancer, not sure though...)

If .net is not building the packets being sent for the Hello, then I guess this may be a bug in the lower level windows functions that are building the packets.

HandshakeFailure_try2.zip
HandshakeGood.zip

[rzikm]

When I compare the "bad" capture with capture to e.g. https://microsoft.com, I get almost exactly the same ClientHello. The only difference seems to be server_name (obviously) and that the bad capture adds two more cipher suites (TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f) and TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)). But that should not make any difference.

As far as I can tell, this is a problem on the server side. When you explicitly specify TLS 1.2 only, the client does not send supported_versions extension, which is the only thing I can imagine makes difference here. When client offers TLS 1.3, the server panics and fails the handshake instead of picking the lower version.

[awdorrin]

To clarify, in the captures above, my laptop is 147.170.184.176 and the remote server is either 146.69.2.26 or 146.60.2.126.
While I'm not discounting that this is a server side issue, I'm not sure I understand why the same code works when run on Win10 or in the PCF Linux instance, but fails in Win11, unless the Tls12 flag is set.
It is too bad that the TLS protocol doesn't provide more of an explanation in the handshake failure packet.

[rzikm]

Win10 does not support TLS 1.3, so the client will not offer it, don't know about the PCF Linux.

As a test, you can try disabling TLS 1.3 via registry by setting HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client\Enabled key to 0

[wfurt]

as @rzikm mentioned, it should work ... but obviously it does not. You also mentioned load balancer. Is there anything in the middle inspecting traffic. You may run to something like https://knowledge.broadcom.com/external/article/185069/cannot-access-ssl-sites-using-tlsv3.html @awdorrin. While peers should not be overwhelmed by new ciphers and versions they sometimes do.

[awdorrin]

I created the registry key that @rzikm mentioned and removed the code to set the protocol to TLS1.2, and the TLS handshake worked from my Windows 11 system.

Looking at the packet captures, when things work, I don't see any exchange of 'supported_versions', it seems to already know that it needs to use TLS 1.2 - and I do not understand how that knowledge is being exchanged.
I'm guessing that there must be an encrypted request made from my laptop to another IP address, that contains the Http Get request. (I haven't used wireshark in a couple of years, so I'm a bit rusty.)

When I'm not forcing TLS 1.2 (either by not having that registry key, or by using the Ssl12 flag in code), I do see that the TLS 'client hello' that has the remote server IP as the source, is listing TLS 1.3 (even though I've been told that the remote servers do not yet support TLS 1.3...)

So, I am coming around to this being a server side issue. I just seem to be missing something before that first 'Client Hello' packet that must be giving a hint or information that it should default to TLS 1.2.

I do recall there is some means of getting the encryption key from your browser and providing that to Wireshark so that it can decode the HTTPS packets, but for some reason, Wireshark is incredibly unstable on my laptop, and quickly stops responding while I'm trying to capture these packets.

As thing happen, I just received a new laptop to replace my current one, so once I get up and running on the new system (which I'm not sure if it will be Win10 or 11 yet) I'll have to do some more testing.

Thanks

[rzikm]

Looking at the packet captures, when things work, I don't see any exchange of 'supported_versions',
...
I just seem to be missing something before that first 'Client Hello' packet that must be giving a hint or information that it should default to TLS 1.2.

The supported_versions extension has been introduced in TLS 1.3, it is not used in TLS 1.2. I briefly read the TLS 1.2 RFC and it seems that the client will simply start connection with version set to TLS 1.2, and if server does not implement TLS 1.2, it will reply with version set to TLS 1.1 (or lower), and if client is okay with that, the handshake continues... This is possible because the Client Hello between SSL 3.0 and TLS 1.2 are compatible.

I do recall there is some means of getting the encryption key from your browser and providing that to Wireshark so that it can decode the HTTPS packets,

Some browser (firefox, chrome) support exporting session keys by setting a SSLKEYLOGFILE env. variable to a path where it should be stored. Wireshark can then load the necessary keys from the file.

Since this seems not to be a .NET issue, I will close this issue.

[flower7434]

Thank you! This issue is still there. I can reproduce it in latest VS2022 with .NET 7. Windows 11 Pro 22000.1219. It also no longer work to do "ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;" instead of suggested solution.

Stacktrace:

   at System.Net.Http.ConnectHelper.<EstablishSslConnectionAsync>d__2.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<ConnectAsync>d__96.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<CreateHttp11ConnectionAsync>d__98.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<AddHttp11ConnectionAsync>d__73.MoveNext()
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.<WaitWithCancellationAsync>d__1.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<GetHttp11ConnectionAsync>d__75.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<SendWithVersionDetectionAndRetryAsync>d__83.MoveNext()
   at System.Net.Http.DiagnosticsHandler.<SendAsyncCore>d__8.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()
   at System.Net.Http.HttpClient.<<SendAsync>g__Core|83_0>d.MoveNext()

Messages:

The SSL connection could not be established, see inner exception.

The message received was unexpected or badly formatted.

Authentication failed because the remote party sent a TLS alert: 'HandshakeFailure'.

[wfurt]

ServicePointManager does nothing if you use HttpClient e.g. it is partially supported only for legacy HttpWebRequest.
If the client offers multiple choices but server sends alert back even if there is overlap the problem lives on server side (or anything in-between)

[flower7434]

ServicePointManager does nothing if you use HttpClient e.g. it is partially supported only for legacy HttpWebRequest. If the client offers multiple choices but server sends alert back even if there is overlap the problem lives on server side (or anything in-between)

I believe it depends on how the HttpClient is created in old .NET Framework.

[wfurt]

the comment about HttpClient is applicable to .NET Core. And yes, the way how APIs are stacked changed and it is somewhat confusing.