mbedTLS 3.6.0 breaks curl

[bagder]

I did this

#13646 tried to upgrade the mbedTLS CI jobs to use version 3.6.0

This caused numerous failed CI jobs.

I expected the following

green builds

curl/libcurl version

curl master

operating system

on Linux, but probably a universal problem

[jay]

for the failed tests it looks as though stunnel server is sending a tls alert of bad certificate. (it's not clear to me if it's sending or receiving the alert)

https://github.com/curl/curl/actions/runs/9086442477/job/24972131840#step:35:1170

 2024.05.14 21:33:45 LOG5[ui]: stunnel 5.63 on x86_64-pc-linux-gnu platform
 2024.05.14 21:33:45 LOG5[ui]: Compiled/running with OpenSSL 3.0.2 15 Mar 2022
 2024.05.14 21:33:45 LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,OCSP,PSK,SNI Auth:LIBWRAP
 2024.05.14 21:33:45 LOG5[ui]: Reading configuration from file /home/runner/work/curl/curl/tests/log/server/https_stunnel.conf
 2024.05.14 21:33:45 LOG5[ui]: UTF-8 byte order mark not detected
 2024.05.14 21:33:45 LOG5[ui]: FIPS mode disabled
 2024.05.14 21:33:45 LOG5[ui]: Configuration successful
 2024.05.14 21:33:45 LOG5[ui]: Binding service [curltest] to :::38897: Address already in use (98)
 2024.05.14 21:33:46 LOG5[0]: Service [curltest] accepted connection from 127.0.0.1:36328
 2024.05.14 21:33:46 LOG3[0]: SSL_accept: ../ssl/record/rec_layer_s3.c:1584: error:0A000412:SSL routines::sslv3 alert bad certificate
 2024.05.14 21:33:46 LOG5[0]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket

I can't reproduce in Linux however I did not copy the CI job configuration.

I built mbedtls like this:

curl -OL https://github.com/Mbed-TLS/mbedtls/releases/download/v3.6.0/mbedtls-3.6.0.tar.bz2
tar -xf mbedtls-3.6.0.tar.bz2
cd mbedtls-3.6.0/
LDFLAGS="-Wl,--enable-new-dtags -Wl,-rpath,/usr/local/lib" make SHARED=1
sudo make install

I built curl like LDFLAGS=... ./configure ... --enable-debug --without-openssl --with-mbedtls=/usr/local, make and then ran the full test suite, 100% ok.

[icing]

Build with 3.6.0. First test failing is 300, log shows:

ssl_handshake returned - mbedTLS: (-0x2700) X509 - Certificate verification failed, e.g. CRL, CA or signature check failed

Opened issue at mbedTLS: Mbed-TLS/mbedtls#9210

Update: this seems to only fail for connections with verifypeer == FALSE and looks like a bug in mbedTLS TLSv1.3 handling of such a setup.

Update2: "We don't support disabling server authentication in TLS 1.3." 🤓

[icing]

Made #13838 with workarounds to test fine locally. Let's see what CI says.

[jay]

Made #13838 with workarounds to test fine locally. Let's see what CI says.

Thanks for looking into this. IMO these are mbedtls bugs and I think we should wait and see what they are going to do about them instead of patching curl. In other words, if they intend to fix the problems then I don't think the workarounds should go in. Hats off to you though for figuring some out. Also- I don't understand why curl passes all tests for me with 3.6.0 if it was intended behavior on their part.