HttpClient sends invalid extension during TLS Client Hello causing failed connection to server

[Programatic]

I originally wrote this issue on the flutter issue tracker because I did not know where to put it. I do believe this is here, so I will just copy it over here; flutter/flutter#105381.

Steps to Reproduce

1. Execute dart run example.dart on the code sample
2. Press Button
3. See error in debug output

Expected results:
Request successfully goes through

Actual results:
Error fails with error

I/flutter (29677): Error HandshakeException: Handshake error in client (OS Error:
I/flutter (29677): 	TLSV1_ALERT_DECODE_ERROR(tls_record.cc:594))

Code sample

import 'dart:io';

void main() async {
  HttpClient client = HttpClient();
  Uri url = Uri.parse("https://10.0.0.224:8888/");
  try {
    await client.postUrl(url);
  } catch (error) {
    print("Error $error");
  }
}

Logs

Error HandshakeException: Handshake error in client (OS Error:
	TLSV1_ALERT_DECODE_ERROR(tls_record.cc:594))

Dart SDK version: 2.18.0-edge.09967d8c255e6d996bbc419fe3a2148aaff5e9e4 (be) (Sun Jun 5 03:11:37 2022 +0000) on "linux_x64"

Currently, my server is using axum-server with rustls in order to create a backend for my application. In doing so, I only have an ip to connect with. However, using wireshark we can decode the TLSv1.2 Client Hello packet that the flutter application is sending.

Specifically what we see here is that the HttpClient is sending a server_name extension. While this traditionally would serve to increase security, the way it is represented with the uri is a bug. 10.0.0.224 is clearly not a hostname, but instead an ip. Furthermore, according to the ietf rfc, an ip is a distinct thing from a hostname and "Literal IPv4 and IPv6 addresses are not permitted in 'HostName'". As a result, when the server goes to decode the host name, it fails to get a DnsNameRef. If we replay this packet with only the hostname changed to the hex of a string, such as 'google.com', then the connection succeeds.

EDIT: Changed quote from RFC to more proper RFC as well as simplifying code. Thanks @simolus3!

[Programatic]

Just to preemptively answer a question posed in the original flutter issues

Looking at the error TLSV1_ALERT_DECODE_ERROR, it could be related to your server using the latest TLSV version which Flutter may not be supporting.

This is actually not the case. It is only this extension causing the issue (I created a python script to replay this exact packet to my server dynamically. All the other fields are valid except this one, and prevents the connection), plus rustls supports 1.2 up. I also was able to confirm this by using the following curl command. This is also reinforced by my workaround: by adding a subdomain A record to my personal servers DNS records to point 10.0.0.224. Using this work around, it then works correctly.
curl -kX POST https://10.0.0.224:8888/ --tls-max 1.2

[brianquinlan]

Right now we call SSL_set_tlsext_host_name with a hostname that ultimately comes from InternetAddress.host.

InternetAddress.host can be an IP address if the client does not have a host associated with it.

[brianquinlan]

I'm closing this issue because certificate checks should succeed when the hostname is an IP address in Dart 3.1

[helgoboss]

I think this should be reopened or deserves a new issue.

#52118 fixes the call to X509_VERIFY_PARAM_set1_host (it uses X509_VERIFY_PARAM_set1_ip_asc instead, if we have an IP address). But it still calls SSL_set_tlsext_host_name, even if we have an IP address. This seems wrong. In fact, OpenSSL should reject this already on the client side, but because of this OpenSSL bug it doesn't. Instead, the server will be confronted with the invalid value. And depending on its standard conformance it will reject the request or accept it. Mine rejects it.

The SSL_set_tlsext_host_name problem was brought up and fixed already in PR #52707, but @mount33 couldn't seem to reproduce the practical issue later, therefore the PR was closed without merging.

This is not just of theoretical nature. I'm affected by this issue when trying to use TLS with an IP address. Here's what happens when I send a HTTP request using TLS with the host being an IP address (192.1678.178.182):

1. My server (using Rustls, apparently a quite strict TLS implementation) detects the error during the handshake and rejects the request: [2024-03-05T10:26:43Z WARN rustls::msgs::handshake] Illegal SNI hostname received "192.168.178.182". Because Dart has fed SSL_set_tlsext_host_name with an IP address. Here is the code in Rustls which rejects the hostname.
2. Flutter in return gives me below stacktrace.

I/flutter (23684): Connecting to URI https://192.168.178.182:39443...
E/flutter (23684): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: HandshakeException: Handshake error in client (OS Error: 
E/flutter (23684): 	TLSV1_ALERT_DECODE_ERROR(tls_record.cc:592))
E/flutter (23684): #0      _SecureFilterImpl._handshake (dart:io-patch/secure_socket_patch.dart:99:46)
E/flutter (23684): #1      _SecureFilterImpl.handshake (dart:io-patch/secure_socket_patch.dart:143:25)
E/flutter (23684): #2      _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:920:54)
E/flutter (23684): #3      _RawSecureSocket._closeHandler (dart:io/secure_socket.dart:913:15)
E/flutter (23684): #4      _RawSecureSocket._eventDispatcher (dart:io/secure_socket.dart:856:9)
E/flutter (23684): #5      _RootZone.runUnaryGuarded (dart:async/zone.dart:1594:10)
E/flutter (23684): #6      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)
E/flutter (23684): #7      _BufferingStreamSubscription._add (dart:async/stream_impl.dart:271:7)
E/flutter (23684): #8      _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:784:19)
E/flutter (23684): #9      _StreamController._add (dart:async/stream_controller.dart:658:7)
E/flutter (23684): #10     _StreamController.add (dart:async/stream_controller.dart:606:5)
E/flutter (23684): #11     new _RawSocket.<anonymous closure> (dart:io-patch/socket_patch.dart:1943:35)
E/flutter (23684): #12     _NativeSocket.issueReadEvent.issue (dart:io-patch/socket_patch.dart:1372:18)
E/flutter (23684): #13     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
E/flutter (23684): #14     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)

Should I open a new issue?