Issue with active FTPS connection after updating to libcurl 7.87.0/7.88.0

[SandakovMM]

Hello
I am experiencing issues with FTP connection after updating to libcurl 7.87.0 and 7.88.0. Unable to reproduce the issue on 7.86.0.

I did this

There is a simple C utility to reproduce the issue:

int debug_curl(CURL* handle, curl_infotype type, char* data, size_t size, void* userptr)
{
    switch (type) {
    case CURLINFO_TEXT:
        printf("Curl output: * %s\n", data);
        break;
    case CURLINFO_HEADER_IN:
        printf("Curl output: < %s\n", data);
        break;
    case CURLINFO_HEADER_OUT:
        printf("Curl output: > %s\n", data);
        break;
    case CURLINFO_DATA_IN:
    case CURLINFO_DATA_OUT:
    case CURLINFO_SSL_DATA_IN:
    case CURLINFO_SSL_DATA_OUT:
    case CURLINFO_END:
        break;
    }
    return 0;
}

int result(char *buf, size_t size, size_t nmemb, void *context)
{
    printf("result: %s\n", buf);
    return 0;
}

int main(int argc, char **argv)
{
    char *url_with_user = argv[1];
    char *password = argv[2];

    CURL *curl = curl_easy_init();
    struct curl_slist *m_headers = NULL;
    if (!curl) {
        printf("unable to init curl");
        return 1;
    }

    curl_easy_setopt(curl, CURLOPT_TIMEOUT, 300);
    curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 30);
    curl_easy_setopt(curl, CURLOPT_FTP_RESPONSE_TIMEOUT, 60);
    curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1);
    curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 60);
    curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 60);

    curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, debug_curl);
    curl_easy_setopt(curl, CURLOPT_DEBUGDATA, NULL);
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

    curl_easy_setopt(curl, CURLOPT_FTPPORT, "-");
    curl_easy_setopt(curl, CURLOPT_FTP_USE_EPRT, 1L);

    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0);
    curl_easy_setopt(curl, CURLOPT_FTP_SSL, CURLFTPSSL_TRY);
    curl_easy_setopt(curl, CURLOPT_SSLVERSION, CURL_SSLVERSION_MAX_TLSv1_3);

    curl_easy_setopt(curl, CURLOPT_URL, url_with_user);
    curl_easy_setopt(curl, CURLOPT_USERPWD, password);

    curl_easy_setopt(curl, CURLOPT_DIRLISTONLY, 0L);
    curl_easy_setopt(curl, CURLOPT_NOBODY, 0);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, result);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, NULL);
    curl_easy_perform(curl);
    return 0;
}

It works fine with 7.86.0, but when I build it with 7.87.0 or 7.88.0, the connection hangs for a wait timeout and closes without output.
There is the output of the program with libcurl 7.87.0:

Curl output: *   Trying 10.69.41.135:21...

Curl output: * Connected to 10.69.41.135 (10.69.41.135) port 21 (#0)

Curl output: < 220 ProFTPD Server (ProFTPD) [10.69.41.135]

Curl output: > AUTH SSL

Curl output: < 234 AUTH SSL successful
PD) [10.69.41.135]
Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Certificate (11):

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, CERT verify (15):

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Finished (20):

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Finished (20):

Curl output: * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384

Curl output: * Server certificate:

Curl output: *  subject: C=CH; L=Schaffhausen; O=Plesk; CN=Plesk; emailAddress=info@plesk.com

Curl output: *  start date: Feb 28 06:04:14 2023 GMT

Curl output: *  expire date: Feb 28 06:04:14 2024 GMT

Curl output: *  issuer: C=CH; L=Schaffhausen; O=Plesk; CN=Plesk; emailAddress=info@plesk.com

Curl output: *  SSL certificate verify result: self signed certificate (18), continuing anyway.

Curl output: > USER tftp

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

Curl output: * old SSL session ID is stale, removing

Curl output: < 331 Password required for tftp
.69.41.135]
Curl output: > PASS 1qazxsw2

Curl output: < 230 User tftp logged in
 tftp
Curl output: > PBSZ 0

Curl output: < 200 PBSZ 0 successful

Curl output: > PROT P

Curl output: < 200 Protection set to Private

Curl output: > PWD

Curl output: < 257 "/home/tftp" is the current directory

Curl output: * Entry path is '/home/tftp'

Curl output: * Request has same path as previous transfer

Curl output: > EPRT |1|10.69.41.80|38691|

Curl output: * ftp_perform ends with SECONDARY: 1

Curl output: < 200 EPRT command successful
nt directory
Curl output: * Connect data stream actively

Curl output: > TYPE A

Curl output: < 200 Type set to A
ccessful
Curl output: > LIST

Curl output: < 150 Opening ASCII mode data connection for file list

Curl output: * Maxdownload = -1

Curl output: * Preparing for accepting server on data port

Curl output: * Checking for server connect

Curl output: * Ready to accept data connection from server

Curl output: * Connection accepted from server

Curl output: * Operation timed out after 300001 milliseconds with 0 bytes received

Curl output: * Closing connection 0

Curl output: * [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS alert, close notify (256):

The issue disappears when I remove SSL parts of the code. So likely the problem somewhere in this part. Unfortunately, I can't see where exactly.

I expected the following

I expect to see the line:

result: drwxr-xr-x   2 tftp     root         4096 Feb 28 10:46 tftpboot

Received from the other side.
For example this is the utility output built with libcurl 7.86.0:

Curl output: *   Trying 10.69.41.135:21...

Curl output: * Connected to 10.69.41.135 (10.69.41.135) port 21 (#0)

Curl output: < 220 ProFTPD Server (ProFTPD) [10.69.41.135]

Curl output: > AUTH SSL

Curl output: < 234 AUTH SSL successful
PD) [10.69.41.135]
Curl output: * TLSv1.3 (OUT), TLS handshake, Client hello (1):

Curl output: * TLSv1.3 (IN), TLS handshake, Server hello (2):

Curl output: * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

Curl output: * TLSv1.3 (IN), TLS handshake, Certificate (11):

Curl output: * TLSv1.3 (IN), TLS handshake, CERT verify (15):

Curl output: * TLSv1.3 (IN), TLS handshake, Finished (20):

Curl output: * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

Curl output: * TLSv1.3 (OUT), TLS handshake, Finished (20):

Curl output: * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384

Curl output: * Server certificate:

Curl output: *  subject: C=CH; L=Schaffhausen; O=Plesk; CN=Plesk; emailAddress=info@plesk.com

Curl output: *  start date: Feb 28 06:04:14 2023 GMT

Curl output: *  expire date: Feb 28 06:04:14 2024 GMT

Curl output: *  issuer: C=CH; L=Schaffhausen; O=Plesk; CN=Plesk; emailAddress=info@plesk.com

Curl output: *  SSL certificate verify result: self signed certificate (18), continuing anyway.

Curl output: > USER tftp

Curl output: * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

Curl output: * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

Curl output: * old SSL session ID is stale, removing

Curl output: < 331 Password required for tftp
.69.41.135]
Curl output: > PASS 1qazxsw2

Curl output: < 230 User tftp logged in
 tftp
Curl output: > PBSZ 0

Curl output: < 200 PBSZ 0 successful

Curl output: > PROT P

Curl output: < 200 Protection set to Private

Curl output: > PWD

Curl output: < 257 "/home/tftp" is the current directory

Curl output: * Entry path is '/home/tftp'

Curl output: * Request has same path as previous transfer

Curl output: > EPRT |1|10.69.41.80|39219|

Curl output: * ftp_perform ends with SECONDARY: 1

Curl output: < 200 EPRT command successful
nt directory
Curl output: * Connect data stream actively

Curl output: > TYPE A

Curl output: < 200 Type set to A
ccessful
Curl output: > LIST

Curl output: < 150 Opening ASCII mode data connection for file list

Curl output: * Maxdownload = -1

Curl output: * Preparing for accepting server on data port

Curl output: * Checking for server connect

Curl output: * Ready to accept data connection from server

Curl output: * Connection accepted from server

Curl output: * Doing the SSL/TLS handshake on the data stream

Curl output: * SSL re-using session ID

Curl output: * TLSv1.3 (OUT), TLS handshake, Client hello (1):

Curl output: * TLSv1.3 (IN), TLS handshake, Server hello (2):

Curl output: * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

Curl output: * TLSv1.3 (IN), TLS handshake, Finished (20):

Curl output: * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

Curl output: * TLSv1.3 (OUT), TLS handshake, Finished (20):

Curl output: * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384

Curl output: * Server certificate:

Curl output: *  subject: C=CH; L=Schaffhausen; O=Plesk; CN=Plesk; emailAddress=info@plesk.com

Curl output: *  start date: Feb 28 06:04:14 2023 GMT

Curl output: *  expire date: Feb 28 06:04:14 2024 GMT

Curl output: *  issuer: C=CH; L=Schaffhausen; O=Plesk; CN=Plesk; emailAddress=info@plesk.com

Curl output: *  SSL certificate verify result: self signed certificate (18), continuing anyway.

Curl output: * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

Curl output: * old SSL session ID is stale, removing

result: drwxr-xr-x   2 tftp     root         4096 Feb 28 10:46 tftpboot <------------------------------- THE LINE

Curl output: * Failure writing output to destination

Curl output: * TLSv1.3 (IN), TLS alert, close notify (256):

Curl output: * TLSv1.3 (OUT), TLS alert, close notify (256):

Curl output: * Closing connection 0

curl/libcurl version

libcurl 7.87.0 and 7.88.0

operating system

Ubuntu 20

[icing]

Would you be able to check if #10669 resolves the issue? Thanks!

[SandakovMM]

Hi,
I've checked the patch, and it helps. Thank you very much, @icing.
Also, I've checked FTP in passive mode, and seems it doesn't work with this patch. To check it just remove curl_easy_setopt(curl, CURLOPT_FTPPORT, "-"); from code I mentioned earlier:

int debug_curl(CURL* handle, curl_infotype type, char* data, size_t size, void* userptr)
{
    switch (type) {
    case CURLINFO_TEXT:
        printf("Curl output: * %s\n", data);
        break;
    case CURLINFO_HEADER_IN:
        printf("Curl output: < %s\n", data);
        break;
    case CURLINFO_HEADER_OUT:
        printf("Curl output: > %s\n", data);
        break;
    case CURLINFO_DATA_IN:
    case CURLINFO_DATA_OUT:
    case CURLINFO_SSL_DATA_IN:
    case CURLINFO_SSL_DATA_OUT:
    case CURLINFO_END:
        break;
    }
    return 0;
}

int result(char *buf, size_t size, size_t nmemb, void *context)
{
    printf("result: %s\n", buf);
    return 0;
}

int main(int argc, char **argv)
{
    char *url_with_user = argv[1];
    char *password = argv[2];

    CURL *curl = curl_easy_init();
    struct curl_slist *m_headers = NULL;
    if (!curl) {
        printf("unable to init curl");
        return 1;
    }

    curl_easy_setopt(curl, CURLOPT_TIMEOUT, 300);
    curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 30);
    curl_easy_setopt(curl, CURLOPT_FTP_RESPONSE_TIMEOUT, 60);
    curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1);
    curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 60);
    curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 60);

    curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, debug_curl);
    curl_easy_setopt(curl, CURLOPT_DEBUGDATA, NULL);
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

    // Option was here <-------------------------------------------------------------------
    curl_easy_setopt(curl, CURLOPT_FTP_USE_EPRT, 1L);

    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0);
    curl_easy_setopt(curl, CURLOPT_FTP_SSL, CURLFTPSSL_TRY);
    curl_easy_setopt(curl, CURLOPT_SSLVERSION, CURL_SSLVERSION_MAX_TLSv1_3);

    curl_easy_setopt(curl, CURLOPT_URL, url_with_user);
    curl_easy_setopt(curl, CURLOPT_USERPWD, password);

    curl_easy_setopt(curl, CURLOPT_DIRLISTONLY, 0L);
    curl_easy_setopt(curl, CURLOPT_NOBODY, 0);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, result);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, NULL);
    curl_easy_perform(curl);
    return 0;
}

I believe we could disable your changes for FTP in passive mode:

  DEBUGF(infof(data, "ftp InitiateTransfer()"));
-  if(conn->bits.ftp_use_data_ssl &&
+  if(conn->bits.ftp_use_data_ssl && data->set.ftp_use_port &&
      !Curl_conn_is_ssl(conn, SECONDARYSOCKET)) {
    result = Curl_ssl_cfilter_add(data, conn, SECONDARYSOCKET);
    if(result)

It works for me, but there might be better solutions since I'm not familiar with curl source code good enough.

[icing]

@bagder: you understand the ftp flags better than me. Does the proposed fix sound good to you?

[bagder]

It seems like a good fix. The InitiateTransfer function gets called in both passive and active mode, so it needs to check data->set.ftp_use_port to make sure the condition only applies to active.

[icing]

Added this to #10669.