Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

curl_easy_reset() does not reset text/binary option for FTP transfer #6577

Closed
bodob opened this issue Feb 8, 2021 · 9 comments
Closed

curl_easy_reset() does not reset text/binary option for FTP transfer #6577

bodob opened this issue Feb 8, 2021 · 9 comments
Labels

Comments

@bodob
Copy link

@bodob bodob commented Feb 8, 2021

Transfer after curl_easy_reset() uses incorrect options, thus still happening in ASCII mode, even when explicitly setting CURLOPT_TRANSFERTEXT to 0.
In addition, this also causes '\r' (carriage returns) to be written to the output on Unix, which is wrong for both ASCII and binary mode.

To reproduce

  1. Transfer (upload) uses ASCII mode by using FTP command "TYPE A" via CURLOPT_PREQUOTE.
  2. curl_easy_reset() to reset all options
  3. Next transfer (download)

Example C code

#include <curl/curl.h>
int main(int argc, char *argv[])
{
  CURLcode ret;
  CURL *hnd = NULL;
  struct curl_slist *cmdlist = NULL;

  hnd = curl_easy_init();
  curl_easy_setopt(hnd, CURLOPT_URL, "ftp://ftpserver.example.com//tmp/myfile.txt");
  curl_easy_setopt(hnd, CURLOPT_USERPWD, "anonymous:xyz");
  curl_easy_setopt(hnd, CURLOPT_UPLOAD, 1L);
  // Using CURLOPT_TRANSFERTEXT=1 would be easier, but we want to use/test CURLOPT_PREQUOTE
  cmdlist = curl_slist_append(cmdlist, "TYPE A"); // ASCII transfer
  curl_easy_setopt(hnd, CURLOPT_PREQUOTE, cmdlist);
  curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
  ret = curl_easy_perform(hnd);

  fprintf(stderr, "\nUpload done, now doing download\n\n.");

  // Resetting  all options before download
  curl_easy_reset(hnd);
  // Workaround (instead of the above call):
  // curl_easy_cleanup(hnd);
  // hnd = curl_easy_init();

  curl_easy_setopt(hnd, CURLOPT_URL, "ftp://ftpserver.example.com//tmp/myfile.txt");
  curl_easy_setopt(hnd, CURLOPT_USERPWD, "anonymous:xyz");
  curl_easy_setopt(hnd, CURLOPT_TRANSFERTEXT, 0L); // Trying explicitly set binary mode - does not work
  // Note: Even CURLOPT_TRANSFERTEXT=1 did not correctly download the file -  output still contains \r characters
  curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
  ret = curl_easy_perform(hnd);

  curl_easy_cleanup(hnd);
  hnd = NULL;

  return (int)ret;
}

Example usage

From bash (ftpdld is the name of the executable created from code above):

printf "Line1\nLine2" | ftpdld | od -cx

Workaround

Instead of curl_easy_reset() use a new curl handle for the download.

Expected output

Verbose output (stderr) should indicate binary mode (TYPE I) being used for download.
File downloaded only contains newline (no carriage return) as end-of-line characters.

Actual output

Verbose output (stderr) indicates that ASCII mode is used for download.
File downloaded contains carriage return + newline as end-of-line characters.

curl/libcurl version

libcurl/7.74.0

operating system

Linux xxxxx 3.10.0-1062.el7.x86_64 #1 SMP Thu Jul 18 20:25:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

@bagder bagder added the FTP label Feb 8, 2021
@bagder bagder changed the title curl_easy_reset() does not reset all options correctly, i.e. for FTP transfer curl_easy_reset() does not reset text/binary option for FTP transfer Feb 8, 2021
@bagder
Copy link
Member

@bagder bagder commented Feb 8, 2021

I renamed it to be more specific

@bagder
Copy link
Member

@bagder bagder commented Feb 8, 2021

I don't think your example case shows any bug. Your code inserts a TYPE A as a command there behind curl's back so it doesn't know you changed the mode and therefore it will also not change it back even if you set it to use binary, because as far as curl knows it already uses binary there.

If you instead do curl_easy_setopt(hnd, CURLOPT_TRANSFERTEXT, 1L); for the first transfer and then reset, you'll see that the second transfer will set it to TYPE I correctly.

@bodob
Copy link
Author

@bodob bodob commented Feb 8, 2021

Well, the "TYPE A" command is also issued via curl's CURLOPT_PREQUOTE option), so it's actually not behind curl's back.
And even when I tell curl afterwards (during the download), that this is now ASCII by using curl_easy_setopt(hnd, CURLOPT_TRANSFERTEXT, 1L);, then the output is still containing carriage returns (coming from the FTP protocol), which are not removed by curl, which is wrong on Unix.

At least it should be documented that commands executed by the CURLOPT_PREQUOTE and CURLOPT_POSTQUOTE options are executed unnoticed by curl and not unset by the curl_easy_reset() function, which can cause problems afterwards, thus the developer is responsible for unsetting anything changed by these commands.
The documentation for curl_easy_reset currently states:

This puts back the handle to the same state as it was in when it was just created with curl_easy_init.

This is obviously not the case.

Of course setting CURLOPT_TRANSFERTEXT=1 during upload the step would be the obvious cure,
but I was in the step of testing CURLOPT_PREQUOTE and CURLOPT_POSTQUOTE options (with curl_slist), and the "TYPE A" command was just the first one I tried - and immediately got into unexpected behaviour, so I am wondering if there are other problems to expect.

@jzakrzewski
Copy link
Contributor

@jzakrzewski jzakrzewski commented Feb 8, 2021

Well, the "TYPE A" command is also issued via curl's CURLOPT_PREQUOTE option), so it's actually not behind curl's back.

Yes, but libcurl does not parse your commands, so you're effectively changing state of the connection without informing libcurl about it.

And even when I tell curl afterwards (during the download), that this is now ASCII by using curl_easy_setopt(hnd, CURLOPT_TRANSFERTEXT, 1L);, then the output is still containing carriage returns (coming from the FTP protocol), which are not removed by curl, which is wrong on Unix.

Yep, and it's a known and documented limitation "that nobody has rectified" to quote the docs.

not unset by the curl_easy_reset() function

Is it really so or is the "TYPE A" simply a sticky command that is in effect until you actually issue "TYPE I"?

@bagder
Copy link
Member

@bagder bagder commented Feb 8, 2021

This puts back the handle to the same state as it was in when it was just created with curl_easy_init.
This is obviously not the case.

I still claim it is the case even if it is a bit of a word game. The handle knows what state it wants and the reset restores that "wanted state" to default. What happens isn't related to the handle's state, what happened was that the connection has been modified to be in a different state than libcurl knows about.

bagder added a commit that referenced this issue Feb 8, 2021
... so passed in commands may confuse libcurl's knowledge of state.

Reported-by: Bodo Bergmann
Fixes #6577
@bodob
Copy link
Author

@bodob bodob commented Feb 8, 2021

So, why is libcurl still producing carriage returns in the output on Unix when I explicitly tell it, that this is an ASCII transfer (so it should remove any incoming carriage returns)?

And I still claim, that it is at least a documentation issue - if not for curl_easy_reset(), then for the CURLOPT_PREQUOTE and CURLOPT_POSTQUOTE options.
Something mentioning that the developer is responsible for unsetting anything in the connection state set by these commands, i.e. for commands that change something the curl handle usually knows about.
IMHO, a "bug" doesn't always mean a bug in the product (which I admit it's probably not, apart from the strange behavior mentioned above), but could also mean a problem in the documentation, where users expect a certain behavior.
A little more explanation in the documentation could prevent such expectations.

@bagder
Copy link
Member

@bagder bagder commented Feb 8, 2021

why is libcurl still producing carriage returns in the output on Unix when I explicitly tell it, that this is an ASCII transfer

That's unrelated to this bug report though and is about curl not doing the necessary text-translations for FTP ASCII transfers. It is mentioned in the CURLOPT_TRANSFERTEXT option man page. This is a known flaw.

@bodob
Copy link
Author

@bodob bodob commented Feb 8, 2021

I do not actually expect a code change, I am fine with a clarifying documentation change for the CURLOPT_PREQUOTE and CURLOPT_POSTQUOTE options (not sure if other options are also affected) to help others to avoid problems like this, i.e. as the libcurl documentation is refered to by others (e.g. wrapper libraries for other languages).

BTW, thanks for the very quick response.

@bagder
Copy link
Member

@bagder bagder commented Feb 8, 2021

#6580 attempts to clarify the curl-not-parsing-quote-commands situation

@bagder bagder closed this in ff9ec4e Feb 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants