Problem with SMTP + STARTTLS sending using libcurl 8.8.0 that did not exist in 8.7.1

[tomy2105]

I did this

I have a small sample based on "official" TLS sample (code below) which send email using smtp + StartTLS.
Compiled on Windows, with libcurl 8.7.1 and OpenSSL 3.0.13 the sample worked OK.

However compiled on Windows, with libcurl 8.8.0 and same OpenSSL 3.0.13 the sample fails (see output below).

I've also examined Wireshark logs for both runs and it seems that libcurl 8.8.0 errors out in the place where 8.7.1 sends "Content Type: Change Cipher Spec (20)".

What causes it to fail on 8.8.0?
Can I somehow fix it with different options or....?

Another observation, if I enable smtps (different port) and use encryption from the beginning and not StartTLS, then both libcurl 8.7.1 and 8.8.0 work correctly.

Thank you in advance for help.

Wireshark screenshot for 8.7.1:

Wireshark screenshot for 8.8.0:

Complete Wireshark trace for 8.7.1: curl_8.7.1_mailjet.pcapng.gz
Complete Wireshark trace for 8.8.0: curl_8.8.0_mailjet.pcapng.gz

Output of failed run with 8.8.0:

* Host in-v3.mailjet.com:587 was resolved.
* IPv6: (none)
* IPv4: 104.199.96.85
*   Trying 104.199.96.85:587...
* Connected to in-v3.mailjet.com (104.199.96.85) port 587
< 220 in.mailjet.com ESMTP Mailjet
> EHLO HR-TOMPETRO-D
< 250-smtpin.mailjet.com
< 250-PIPELINING
< 250-SIZE 15728640
< 250-VRFY
< 250-ETRN
< 250-STARTTLS
< 250-AUTH PLAIN LOGIN DIGEST-MD5 CRAM-MD5
< 250-AUTH=PLAIN LOGIN DIGEST-MD5 CRAM-MD5
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250-SMTPUTF8
< 250 CHUNKING
> STARTTLS
< 220 2.0.0 Ready to start TLS
*  CAfile: .\Data\envox-curl-ca-bundle.crt
*  CApath: none
< ▬♥♥z☻v♥♥↓jő
< ľ˙▄▓+♂♠↨ÄU17úö;◄üE▀C■3ń˘_ăň š‼⌂VŹ►˝‼▓1»JvĎ╣T ⌂ľŕ\#öU,`nŁ■◄'Ž‼☻.+☻♥♦3$↔ <÷jJ$☼>ë¤ů↑íG;ťŕ▄¤+_$ EkD╗══i¶♥♥☺☺↨♥♥↨╣ŚÇć╩H╠Z/in%▒ś¨║┌YÔgŐ═╠↨♥♥♀☼|qNâ◄█↔ÖD˛+┤Ť÷ŃřV6/Ž¤$žą♠žÚ1Ć▀ęŢ'[×C☼║q▲
< ¶<Ă˙í[ÄĘ╣?đ♀┼└¸ć╩╠âĄ2ňxÍ{Yľúäăő{Á31;<A▓űëĆĂ
< ŚÁś┌♦$5ź┌█p=↕║?U╝╬(-Ĺ┤ňĐěIix_↕▄Š×w→ ►Úą1»ëKZ─Ëç♫ĐÜ ↓Aş╝▲qt¨-o43♂Hűť[:ŮZןŐ╣xűčšúX2ńňę↕ř>.ř(┘ÎŻod'♫˛░SNŢ'#ÉË.Zů↑$SMJ─`Ţę┌♦ENî˘Ů┴üĹ~ú♀☼ť(─ĚâĐŕ│'÷.ô6╔Ł▓Ăř∟╦Ř(šĘU╝
< ╩dGť-ŇW-ź+Ź╬*↓ŹXý└R▼ü!»ňé▄y~☼#ř'/-░↓/Y‼╝şH▓Ů1sűŽ)I
< ÇRży♀┬Ǹ☺ˇ}∟ôŮ╚∟'╗⌂ô&m░F¬Úđ‼@ľjHŠč├┼IËqdZĽ┘đ3ŞĹT☺P▀~ÎNŔoě┘Â┴NÝç♣Fmď▲╬Ď(ëÚ(─G─┌ˇBĽ¨┘­/■»♦%ÜżrMô
* OpenSSL/3.0.13: error:0A00010B:SSL routines::wrong version number
* Closing connection
curl_easy_perform() failed: SSL connect error

Source code of sample:

#include <stdio.h>
#include <string.h>
#include <curl/curl.h>

#define FROM_MAIL     "<--hidden-->"
#define TO_MAIL       "<--hidden-->"
#define SMTP_USER     "--hidden--"
#define SMTP_PASSWORD "--hidden--"

static const char* payload_text =
"Date: Mon, 29 Nov 2010 21:54:29 +1100\r\n"
"To: " TO_MAIL "\r\n"
"From: " FROM_MAIL "\r\n"
"Message-ID: <dcd7cb36-11db-487a-9f3a-e652a9458efd@"
"rfcpedant.example.org>\r\n"
"Subject: SMTP example message\r\n"
"\r\n" 
"The body of the message starts here.\r\n"
"\r\n"
"It could be a lot of lines, could be MIME encoded, whatever.\r\n"
"Check RFC 5322.\r\n";

struct upload_status {
    size_t bytes_read;
};

static size_t payload_source(char* ptr, size_t size, size_t nmemb, void* userp)
{
    struct upload_status* upload_ctx = (struct upload_status*)userp;
    const char* data;
    size_t room = size * nmemb;

    if ((size == 0) || (nmemb == 0) || ((size * nmemb) < 1)) {
        return 0;
    }

    data = &payload_text[upload_ctx->bytes_read];

    if (data) {
        size_t len = strlen(data);
        if (room < len)
            len = room;
        memcpy(ptr, data, len);
        upload_ctx->bytes_read += len;

        return len;
    }

    return 0;
}

int main(void)
{
    CURL* curl;
    CURLcode res = CURLE_OK;
    struct curl_slist* recipients = NULL;
    struct upload_status upload_ctx = { 0 };

    curl = curl_easy_init();
    if (curl) {
        curl_easy_setopt(curl, CURLOPT_USERNAME, SMTP_USER);
        curl_easy_setopt(curl, CURLOPT_PASSWORD, SMTP_PASSWORD);
        // works both with libcurl 8.8.0 and 8.7.1
        // curl_easy_setopt(curl, CURLOPT_URL, "smtps://in-v3.mailjet.com:465"); 
        // works only with libcurl 8.7.1 on 8.8.0 fails with "curl_easy_perform() failed: SSL connect error" "OpenSSL/3.0.13: error:0A00010B:SSL routines::wrong version number"
        curl_easy_setopt(curl, CURLOPT_URL, "smtp://in-v3.mailjet.com:587"); 
        curl_easy_setopt(curl, CURLOPT_USE_SSL, (long)CURLUSESSL_ALL);
        curl_easy_setopt(curl, CURLOPT_CAINFO, ".\\Data\\envox-curl-ca-bundle.crt");
        curl_easy_setopt(curl, CURLOPT_MAIL_FROM, FROM_MAIL);
        recipients = curl_slist_append(recipients, TO_MAIL);
        curl_easy_setopt(curl, CURLOPT_MAIL_RCPT, recipients);
        curl_easy_setopt(curl, CURLOPT_READFUNCTION, payload_source);
        curl_easy_setopt(curl, CURLOPT_READDATA, &upload_ctx);
        curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
        curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

        res = curl_easy_perform(curl);

        if (res != CURLE_OK)
            fprintf(stderr, "curl_easy_perform() failed: %s\n",
                curl_easy_strerror(res));

        curl_slist_free_all(recipients);
        curl_easy_cleanup(curl);
    }

    return (int)res;
}

I expected the following

Sample work both with libcurl 8.7.1 and libcurl 8.8.0.

curl/libcurl version

curl 8.8.0

operating system

Windows Server 2022 or Windows 10

[icing]

The garbage in your trace output look suspicious. Can you add the following to your client after curl_easy_init()?

curl_global_trace("all");

Thanks.

[tomy2105]

Thanks for helping....

New output after global trace all has been added to sample (below find also output when using 8.7.1 which runs ok)....

* Host in-v3.mailjet.com:587 was resolved.
* IPv6: (none)
* IPv4: 104.199.96.85
* [SETUP] added
* [HAPPY-EYEBALLS] created ipv4 (timeout 299500ms)
* [HAPPY-EYEBALLS] ipv4 starting (timeout=299500ms)
*   Trying 104.199.96.85:587...
* [TCP] cf_socket_open() -> 0, fd=404
* [TCP] local address 0.0.0.0 port 56644...
* [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=0
* [TCP] adjust_pollset(!connected) -> 1 socks
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset(!connected) -> 1 socks
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset(!connected) -> 1 socks
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] not connected yet
* [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=0
* [TCP] adjust_pollset(!connected) -> 1 socks
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset(!connected) -> 1 socks
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset(!connected) -> 1 socks
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] connected
* [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=1
* Connected to in-v3.mailjet.com (104.199.96.85) port 587
* [TCP] nw_in_read(len=900) -> 34, err=0
* [TCP] recv(len=900) -> 34, err=0
< 220 in.mailjet.com ESMTP Mailjet
* [TCP] send(len=20) -> 20, err=0
> EHLO HR-TOMPETRO-D
* [TCP] nw_in_read(len=900) -> -1, err=81
* [TCP] recv(len=900) -> -1, err=81
* [TCP] nw_in_read(len=900) -> 244, err=0
* [TCP] recv(len=900) -> 244, err=0
< 250-smtpin.mailjet.com
< 250-PIPELINING
< 250-SIZE 15728640
< 250-VRFY
< 250-ETRN
< 250-STARTTLS
< 250-AUTH PLAIN LOGIN DIGEST-MD5 CRAM-MD5
< 250-AUTH=PLAIN LOGIN DIGEST-MD5 CRAM-MD5
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250-SMTPUTF8
< 250 CHUNKING
* [TCP] send(len=10) -> 10, err=0
> STARTTLS
* [TCP] nw_in_read(len=900) -> -1, err=81
* [TCP] recv(len=900) -> -1, err=81
* [TCP] nw_in_read(len=900) -> 30, err=0
* [TCP] recv(len=900) -> 30, err=0
< 220 2.0.0 Ready to start TLS
* [SSL] added
* [SSL] cf_connect()
* [TCP] send(len=517) -> 517, err=0
* [SSL] ossl_bio_cf_out_write(len=517) -> 517, err=0
* [TCP] nw_in_read(len=5) -> -1, err=81
* [TCP] recv(len=5) -> -1, err=81
* [SSL] ossl_bio_cf_in_read(len=5) -> -1, err=81
* [SSL] populate_x509_store, path=.\Data\envox-curl-ca-bundle.crt, blob=0
*  CAfile: .\Data\envox-curl-ca-bundle.crt
*  CApath: none
* [SSL] cf_connect() -> 0, done=0
* [TCP] nw_in_read(len=900) -> 900, err=0
* [TCP] recv(len=900) -> 900, err=0
< ▬♥♥z☻v♥♥│»9Úč╦╚ĽŃüŻÖŹ Îh/↓d§]ŢúKÚbÎmxŁ}ü ú∟§↨OŚľ§ĄBŞť6O♠¶ŁnŞVÁ´_┼O9Ŕ▲Ç0‼☻.+☻♥♦3$↔ ↨ ąA*úD     Z┌┬ź↔ÄÔ-ID╣Č♂ö>I$¨┬♂┌h¶♥♥☺☺↨♥♥↨▄+X♀▒4o5\I║<q┐└╦Ź♂→?╬;╩Ć­ÝĐ
▬─ř┌ő█ĚţüômH└Ö░∟▒Í$ÜÝú49═Á`╬˙ÎgăA9█Žqď:nŮ↕J/Ý%╣oOo4Ń♦
<  ▒hNŞ´1Ńk9E«▲│░►ł$Ö'⌂;MťĂ5┐ħüMmôˇ­§ďŽ%)»ÜŁň¬J sĂÇľš#♀ĺÉ7╩╠úgßą♠♦˝pE└Ęh▼↨|[z☺Ł˛Eeô╔Źď♀♫ď~♠Ĺ%
* [SSL] cf_connect()
* [TCP] nw_in_read(len=5) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* OpenSSL/3.0.13: error:0A00010B:SSL routines::wrong version number
* [SSL] cf_connect() -> 35, done=0
* Closing connection
* [SSL] SSL shutdown, error: 'error:0A000197:SSL routines::shutdown while in init', errno 0
* [SETUP] close
* [HAPPY-EYEBALLS] close
* [TCP] cf_socket_close(404)
* [TCP] destroy
* [HAPPY-EYEBALLS] destroy
* [SETUP] destroy
curl_easy_perform() failed: SSL connect error

Output when succesfully using libcurl 8.7.1:

* Host in-v3.mailjet.com:587 was resolved.
* IPv6: (none)
* IPv4: 104.199.96.85
* [SETUP] added
* [HAPPY-EYEBALLS] created ipv4 (timeout 299950ms)
* [HAPPY-EYEBALLS] ipv4 starting (timeout=299950ms)
*   Trying 104.199.96.85:587...
* [TCP] cf_socket_open() -> 0, fd=404
* [TCP] local address 0.0.0.0 port 56661...
* [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=0
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] not connected yet
* [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=0
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] not connected yet
* [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=0
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] adjust_pollset, !connected, POLLOUT fd=404
* [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
* [TCP] connected
* [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=1
* Connected to in-v3.mailjet.com (104.199.96.85) port 587
* [TCP] nw_in_read(len=900, fd=404) -> 34, err=0
* [TCP] recv(len=900) -> 34, err=0
< 220 in.mailjet.com ESMTP Mailjet
* [TCP] send(len=20) -> 20, err=0
> EHLO HR-TOMPETRO-D
* [TCP] nw_in_read(len=900, fd=404) -> 244, err=0
* [TCP] recv(len=900) -> 244, err=0
< 250-smtpin.mailjet.com
< 250-PIPELINING
< 250-SIZE 15728640
< 250-VRFY
< 250-ETRN
< 250-STARTTLS
< 250-AUTH PLAIN LOGIN DIGEST-MD5 CRAM-MD5
< 250-AUTH=PLAIN LOGIN DIGEST-MD5 CRAM-MD5
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250-SMTPUTF8
< 250 CHUNKING
* [TCP] send(len=10) -> 10, err=0
> STARTTLS
* [TCP] nw_in_read(len=900, fd=404) -> 30, err=0
* [TCP] recv(len=900) -> 30, err=0
< 220 2.0.0 Ready to start TLS
* [SSL] added
* [SSL] cf_connect()
* [TCP] send(len=517) -> 517, err=0
* [SSL] ossl_bio_cf_out_write(len=517) -> 517, err=0
* [TCP] nw_in_read(len=5, fd=404) -> -1, err=81
* [TCP] recv(len=5) -> -1, err=81
* [SSL] ossl_bio_cf_in_read(len=5) -> -1, err=81
* [SSL] populate_x509_store, path=.\Data\envox-curl-ca-bundle.crt, blob=0
*  CAfile: .\Data\envox-curl-ca-bundle.crt
*  CApath: none
* [SSL] cf_connect() -> 0, done=0
* [SSL] adjust_pollset, POLLIN fd=404
* [SSL] adjust_pollset, POLLIN fd=404
* [SSL] adjust_pollset, POLLIN fd=404
* [SSL] cf_connect()
* [TCP] nw_in_read(len=5, fd=404) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* [TCP] nw_in_read(len=122, fd=404) -> 122, err=0
* [TCP] recv(len=122) -> 122, err=0
* [SSL] ossl_bio_cf_in_read(len=122) -> 122, err=0
* [TCP] nw_in_read(len=5, fd=404) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* [TCP] nw_in_read(len=1, fd=404) -> 1, err=0
* [TCP] recv(len=1) -> 1, err=0
* [SSL] ossl_bio_cf_in_read(len=1) -> 1, err=0
* [TCP] nw_in_read(len=5, fd=404) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* [TCP] nw_in_read(len=23, fd=404) -> 23, err=0
* [TCP] recv(len=23) -> 23, err=0
* [SSL] ossl_bio_cf_in_read(len=23) -> 23, err=0
* [TCP] nw_in_read(len=5, fd=404) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* [TCP] nw_in_read(len=3087, fd=404) -> 3087, err=0
* [TCP] recv(len=3087) -> 3087, err=0
* [SSL] ossl_bio_cf_in_read(len=3087) -> 3087, err=0
* [TCP] nw_in_read(len=5, fd=404) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* [TCP] nw_in_read(len=281, fd=404) -> 281, err=0
* [TCP] recv(len=281) -> 281, err=0
* [SSL] ossl_bio_cf_in_read(len=281) -> 281, err=0
* [TCP] nw_in_read(len=5, fd=404) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* [TCP] nw_in_read(len=69, fd=404) -> 69, err=0
* [TCP] recv(len=69) -> 69, err=0
* [SSL] ossl_bio_cf_in_read(len=69) -> 69, err=0
* [TCP] send(len=80) -> 80, err=0
* [SSL] ossl_bio_cf_out_write(len=80) -> 80, err=0
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* Server certificate:
*  subject: C=FR; L=Paris; O=MAILJET SAS; CN=mailjet.com
*  start date: Mar 12 00:00:00 2024 GMT
*  expire date: Apr  2 23:59:59 2025 GMT
*  subjectAltName: host "in-v3.mailjet.com" matched cert's "*.mailjet.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* [SSL] cf_connect() -> 0, done=1
* [TCP] send(len=42) -> 42, err=0
* [SSL] ossl_bio_cf_out_write(len=42) -> 42, err=0
> EHLO HR-TOMPETRO-D
* [TCP] nw_in_read(len=5, fd=404) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* [TCP] nw_in_read(len=250, fd=404) -> 250, err=0
* [TCP] recv(len=250) -> 250, err=0
* [SSL] ossl_bio_cf_in_read(len=250) -> 250, err=0
* [TCP] nw_in_read(len=5, fd=404) -> -1, err=81
* [TCP] recv(len=5) -> -1, err=81
* [SSL] ossl_bio_cf_in_read(len=5) -> -1, err=81
* [SSL] cf_recv(len=900) -> -1, 81
* [TCP] nw_in_read(len=5, fd=404) -> 5, err=0
* [TCP] recv(len=5) -> 5, err=0
* [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
* [TCP] nw_in_read(len=247, fd=404) -> 247, err=0
* [TCP] recv(len=247) -> 247, err=0
* [SSL] ossl_bio_cf_in_read(len=247) -> 247, err=0
* [SSL] cf_recv(len=900) -> 230, 0
< 250-smtpin.mailjet.com
< 250-PIPELINING
< 250-SIZE 15728640
< 250-VRFY
< 250-ETRN
< 250-AUTH PLAIN LOGIN DIGEST-MD5 CRAM-MD5
< 250-AUTH=PLAIN LOGIN DIGEST-MD5 CRAM-MD5
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250-SMTPUTF8
< 250 CHUNKING
* [TCP] send(len=39) -> 39, err=0
* [SSL] ossl_bio_cf_out_write(len=39) -> 39, err=0
> AUTH DIGEST-MD5

[icing]

Thanks, that helps.

@bagder: I can reproduce when I mess with smtp_statemachine() in the following way:

1. in smtp.c:1239 it reads the server response to STARTTLS, adds the SSL filter and goes into SMTP_UPGRADETLS
2. if the while loop at smtp.c:1273 continues, likely cause is Curl_pp_moredata(pp) returning TRUE...
3. then smtp.c:1214 will read more pingpong data and, as the SSL filter is not yet connected, will read 900 bytes from the socket which is the incoming ServerHello.
4. Now the connection is garbled. The next attempt to continue the SSL handshake fails miserably.

Question: what makes Curl_pp_moredata(pp) think there is more? I see no difference in the source between 8.7.1 and 8.8.0 here. Maybe I got something wrong?

[tomy2105]

@icing thanks for investigating.

I did a little debugging and step by step and this is what I found.
I can tell that in failing case smtp_statemachine is entered 4 times.
3 times it goes into while loop and exits OK.
1st time it goes from state 1 to 2 (Curl_pp_readresp invoked once).
2nd time it goes from state 2 to 4 (Curl_pp_readresp invoked 13 times I think)
3rd time it goes from state 4 to 5 (Curl_pp_readresp invoked 2 times I think)
4th time it doesn't enter do/while but goes into smtp_perform_upgrade_tls and exits with error 35 (CURLE_SSL_CONNECT_ERROR).

Don't know if it helps but digging a little deeper call stack (line numbers might be off because of me adding some debugging lines):
libcurl_scp.dll!ossl_connect_common(Curl_cfilter * cf, Curl_easy * data, bool nonblocking, bool * done) Line 4499 C
libcurl_scp.dll!ossl_connect_nonblocking(Curl_cfilter * cf, Curl_easy * data, bool * done) Line 4531 C
libcurl_scp.dll!ssl_cf_connect(Curl_cfilter * cf, Curl_easy * data, bool blocking, bool * done) Line 1669 C
libcurl_scp.dll!Curl_conn_connect(Curl_easy * data, int sockindex, bool blocking, bool * done) Line 345 C
libcurl_scp.dll!smtp_perform_upgrade_tls(Curl_easy * data) Line 406 C
libcurl_scp.dll!smtp_state_starttls_resp(Curl_easy * data, int smtpcode, smtpstate instate) Line 874 C
libcurl_scp.dll!smtp_statemachine(Curl_easy * data, connectdata * conn) Line 1249 C
libcurl_scp.dll!Curl_pp_statemach(Curl_easy * data, pingpong * pp, bool block, bool disconnecting) Line 137 C
libcurl_scp.dll!smtp_multi_statemach(Curl_easy * data, bool * done) Line 1307 C
libcurl_scp.dll!multi_runsingle(Curl_multi * multi, curltime * nowp, Curl_easy * data) Line 2189 C
libcurl_scp.dll!curl_multi_perform(Curl_multi * multi, int * running_handles) Line 2780 C
libcurl_scp.dll!easy_transfer(Curl_multi * multi) Line 675 C
libcurl_scp.dll!easy_perform(Curl_easy * data, bool events) Line 765 C
libcurl_scp.dll!curl_easy_perform(Curl_easy * data) Line 784 C
Inside ossl_connect_common the ossl_connect_step2 returned the error 35 (CURLE_SSL_CONNECT_ERROR).
In the ossl_connect_step2 there is line err = SSL_connect(backend->handle);, that one returned -1. However this is OpenSSL call and same openssl worked in libCurl 8.7.1 so I am lost here.

Hope it helps somehow, let me know if I can do anything else.

[icing]

I propose #14190 to fix this.

[tomy2105]

@icing thanks for the PR, I've taken the code from your branch, compiled and used it against my sample and it works OK now.

When will the new curl release with this fix be officially released?

[icing]

Thanks for confirming! I think we'll merge this into the 8.9 release on July 24th.

[tomy2105]

8.9 release by the end of the month works fine for me :), thank you very much for fixing this.