Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FTP download fails using libcurl when TCP_FASTOPEN enabled #6252

Closed
KlausC opened this issue Nov 26, 2020 · 4 comments
Closed

FTP download fails using libcurl when TCP_FASTOPEN enabled #6252

KlausC opened this issue Nov 26, 2020 · 4 comments
Labels

Comments

@KlausC
Copy link
Contributor

@KlausC KlausC commented Nov 26, 2020

I found, that the ftp example program, when modified by setting the TCP_FASTOPEN flag, fails.
My working environment is Fedora Linux release 33, but I think that is also for other Linux variants.

Linux klauspc 5.9.8-200.fc33.x86_64 #1 SMP Tue Nov 10 21:58:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[crusius@klauspc src]$ curl-config --version
libcurl 7.71.1

[crusius@klauspc src]$ cc curlftp.c -lcurl -o curlftp

[crusius@klauspc src]$ curlftp --tcp-fastopen "ftp://xmlsoft.org/libxslt/libxslt-1.1.33.tar.gz"
*   Trying 91.121.203.120:21...
* Connected to xmlsoft.org () port 21 (#0)
< 220 (vsFTPd 2.2.2)
> USER anonymous
< 331 Please specify the password.
> PASS ftp@example.com
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD libxslt
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||13558|).
* Could not resolve host: 
* Can't resolve new host :13558
* Closing connection 0
curl told us 15

# For comparison with unset flag:
[crusius@klauspc src]$ curlftp "ftp://xmlsoft.org/libxslt/libxslt-1.1.33.tar.gz"
*   Trying 91.121.203.120:21...
* Connected to xmlsoft.org (91.121.203.120) port 21 (#0)
< 220 (vsFTPd 2.2.2)
> USER anonymous
< 331 Please specify the password.
> PASS ftp@example.com
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD libxslt
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||53060|).
*   Trying 91.121.203.120:53060...
* Connecting to 91.121.203.120 (91.121.203.120) port 53060
* Connected to xmlsoft.org (91.121.203.120) port 21 (#0)
> TYPE I
< 200 Switching to Binary mode.
> SIZE libxslt-1.1.33.tar.gz
< 213 3444093
> RETR libxslt-1.1.33.tar.gz
< 150 Opening BINARY mode data connection for libxslt-1.1.33.tar.gz (3444093 bytes).
* Maxdownload = -1
* Getting file with size: 3444093
* Remembering we are in dir "libxslt/"
< 226 Transfer complete.
* Connection #0 to host xmlsoft.org left intact


[crusius@klauspc src]$ cat curlftp.c

/***************************************************************************
 *                                  _   _ ____  _
 *  Project                     ___| | | |  _ \| |
 *                             / __| | | | |_) | |
 *                            | (__| |_| |  _ <| |___
 *                             \___|\___/|_| \_\_____|
 *
 * Copyright (C) 1998 - 2020, Daniel Stenberg, <daniel@haxx.se>, et al.
 *
 * This software is licensed as described in the file COPYING, which
 * you should have received as part of this distribution. The terms
 * are also available at https://curl.se/docs/copyright.html.
 *
 * You may opt to use, copy, modify, merge, publish, distribute and/or sell
 * copies of the Software, and permit persons to whom the Software is
 * furnished to do so, under the terms of the COPYING file.
 *
 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
 * KIND, either express or implied.
 *
 ***************************************************************************/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <curl/curl.h>

/* <DESC>
 * Get a single file from an FTP server.
 * </DESC>
 */

struct FtpFile {
  const char *filename;
  FILE *stream;
};

static size_t my_fwrite(void *buffer, size_t size, size_t nmemb, void *stream)
{
  struct FtpFile *out = (struct FtpFile *)stream;
  if(!out->stream) {
    /* open file for writing */
    out->stream = fopen(out->filename, "wb");
    if(!out->stream)
      return -1; /* failure, can't open file to write */
  }
  return fwrite(buffer, size, nmemb, out->stream);
}


int main(int argc, char**argv)
{
  CURL *curl;
  CURLcode res;
  struct FtpFile ftpfile = {
    "curl.tar.gz", /* name to store the file as if successful */
    NULL
  };

  curl_global_init(CURL_GLOBAL_DEFAULT);

  long tcpfast = 0L;
  if( argc >= 3 && strcmp(argv[1], "--tcp-fastopen") == 0) {
    tcpfast = 1L;
    argc -= 1;
    argv += 1;
  }
  if (argc < 2) exit(1);

  curl = curl_easy_init();
  if(curl) {
    /*
     * You better replace the URL with one that works!
     */
    curl_easy_setopt(curl, CURLOPT_TCP_FASTOPEN, tcpfast);
    curl_easy_setopt(curl, CURLOPT_URL, argv[1]);

    /* Define our callback to get called when there's data to be written */
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, my_fwrite);
    /* Set a pointer to our struct to pass to the callback */
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, &ftpfile);

    /* Switch on full protocol/debug output */
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

    res = curl_easy_perform(curl);

    /* always cleanup */
    curl_easy_cleanup(curl);

    if(CURLE_OK != res) {
      /* we failed */
      fprintf(stderr, "curl told us %d\n", res);
    }
  }

  if(ftpfile.stream)
    fclose(ftpfile.stream); /* close the local file */

  curl_global_cleanup();

  return 0;
}
@bagder bagder changed the title FTP download fails using libcurl FTP download fails using libcurl when TCP_FASTOPEN enabled Nov 26, 2020
@bagder bagder added the FTP label Nov 26, 2020
@KlausC
Copy link
Contributor Author

@KlausC KlausC commented Nov 26, 2020

With the current version (7.74), compiled with -g -Og, the same run fails earlier:

 curlftp --tcp-fastopen "ftp://xmlsoft.org/libxslt/libxslt-1.1.33.tar.gz"
*   Trying 91.121.203.120:21...
* Connected to xmlsoft.org () port 21 (#0)
< 220 (vsFTPd 2.2.2)
* Send failure: Operation already in progress
* Closing connection 0
curl told us 55
@bagder
Copy link
Member

@bagder bagder commented Nov 26, 2020

I don't see how TCP Fast Open will help for an FTP transfer so I would recommend you don't enable that, as a rather safe work-around.

@KlausC
Copy link
Contributor Author

@KlausC KlausC commented Nov 26, 2020

I think we will disable that in our downstream library wrapper. See Julia: JuliaLang/Downloads.jl#76

KlausC added a commit to KlausC/curl that referenced this issue Nov 28, 2020
The HAVE_LINUX_TCP_H constant was not set by cmake.
The variable has been used in some C-files in lib without including
neither linux/tcp.h nor netinet/tcp.h
That lead to inconsistent calls wrt. TCP_FASTOPEN
Data field conn->ip_addr_str was not set in case of tcp_fastopen,
but tried to be used later.

Fixes curl#6252
KlausC added a commit to KlausC/curl that referenced this issue Nov 29, 2020
After this change, the PR no longer fixes curl#6252.
This original issue will be handled in a separate PR.
KlausC added a commit to KlausC/curl that referenced this issue Nov 29, 2020
In the case of TFO, the remote host name is not resolved at the
connetion time.
For FTP that has lead to missing hostname for the secondary connection.
Therefore the name resolution is done at the time, when FTP requires it.

Fixes curl#6252
@bagder bagder closed this in d6bfbfa Dec 1, 2020
@bagder
Copy link
Member

@bagder bagder commented Dec 1, 2020

That was wrong, the commit only fixed #6262, this should rather be fixed by #6265 ...

@bagder bagder reopened this Dec 1, 2020
KlausC added a commit to KlausC/curl that referenced this issue Dec 1, 2020
In the case of TFO, the remote host name is not resolved at the
connetion time.
For FTP that has lead to missing hostname for the secondary connection.
Therefore the name resolution is done at the time, when FTP requires it.

Fixes curl#6252
KlausC added a commit to KlausC/curl that referenced this issue Dec 2, 2020
In the case of TFO, the remote host name is not resolved at the
connetion time.
For FTP that has lead to missing hostname for the secondary connection.
Therefore the name resolution is done at the time, when FTP requires it.

Fixes curl#6252
bagder added a commit that referenced this issue Dec 5, 2020
In the case of TFO, the remote host name is not resolved at the
connetion time.

For FTP that has lead to missing hostname for the secondary connection.
Therefore the name resolution is done at the time, when FTP requires it.

Fixes #6252
Closes #6265
@bagder bagder closed this in 2c0d721 Dec 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants