Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resume does not interact well with --fail #6740

Closed
jwatt opened this issue Mar 13, 2021 · 1 comment
Closed

Resume does not interact well with --fail #6740

jwatt opened this issue Mar 13, 2021 · 1 comment
Assignees
Labels

Comments

@jwatt
Copy link

@jwatt jwatt commented Mar 13, 2021

I have a script that is designed to allow the user to invoke it again in the event that something fails, while also trying to avoid redoing work that has already been completed when that happens. As part of that:

  1. when the script invokes curl to download a (very large) file, it use -O -C - to tell curl to resume the download if it has already been partially down loaded, or to skip it entirely if the whole thing has been downloaded;
  2. it passes -f/--fail since I do want curl to exit with an error if it was to encounter HTTP errors such as 404.

Unfortunately, when curl tries to resume a completed download, it gets a '416 Range Not Satisfiable' HTTP error, which causes curl to exit with an error instead of skipping the completed download.

Example

# Invoke this twice
curl -O -C - -f "https://github.com/robots.txt"
curl -O -C - -f "https://github.com/robots.txt"

Spitballing

I know that curl has code to specially handle 416, but I guess it's not handling this case correctly.

Perhaps curl could make a HEAD request before trying to resume a download, and check whether the returned Content-Length matches the current size of the file. It could then completely avoid making requests with Range: bytes=<completed-length>-.

Or if that's tricky, perhaps curl could trim a byte from the end of any pre-existing file before trying to resume it, which would again avoid the 416 in the case the file was complete.

curl/libcurl version

curl 7.71.1 (x86_64-redhat-linux-gnu) libcurl/7.71.1 OpenSSL/1.1.1j-fips zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.1 (+libidn2/2.3.0) libssh/0.9.5/openssl/zlib nghttp2/1.43.0
Release-Date: 2020-07-01
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

operating system

Fedora 33
Linux 5.10.21-200.fc33.x86_64 #1 SMP Mon Mar 8 00:24:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

@bagder bagder added the HTTP label Mar 13, 2021
@bagder bagder self-assigned this Mar 16, 2021
bagder added a commit that referenced this issue Mar 16, 2021
When asked to resume a download, libcurl will convert that to HTTP logic
and if then the entire file is already transferred it will result in a
416 response from the HTTP server. With CURLOPT_FAILONERRROR set in that
scenario, it should *not* lead to an error return.

Updated test 1156, added test 1273

Reported-by: Jonathan Watt
Fixes #6740
Closes #
@bagder bagder closed this in 6d176be Mar 17, 2021
@jwatt
Copy link
Author

@jwatt jwatt commented Mar 18, 2021

Thank you, @bagder! I really appreciate you fixing this, especially so quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants