Resume does not interact well with --fail

[jwatt]

I have a script that is designed to allow the user to invoke it again in the event that something fails, while also trying to avoid redoing work that has already been completed when that happens. As part of that:

1. when the script invokes curl to download a (very large) file, it use -O -C - to tell curl to resume the download if it has already been partially down loaded, or to skip it entirely if the whole thing has been downloaded;
2. it passes -f/--fail since I do want curl to exit with an error if it was to encounter HTTP errors such as 404.

Unfortunately, when curl tries to resume a completed download, it gets a '416 Range Not Satisfiable' HTTP error, which causes curl to exit with an error instead of skipping the completed download.

Example

# Invoke this twice
curl -O -C - -f "https://github.com/robots.txt"
curl -O -C - -f "https://github.com/robots.txt"

Spitballing

I know that curl has code to specially handle 416, but I guess it's not handling this case correctly.

Perhaps curl could make a HEAD request before trying to resume a download, and check whether the returned Content-Length matches the current size of the file. It could then completely avoid making requests with Range: bytes=<completed-length>-.

Or if that's tricky, perhaps curl could trim a byte from the end of any pre-existing file before trying to resume it, which would again avoid the 416 in the case the file was complete.

curl/libcurl version

curl 7.71.1 (x86_64-redhat-linux-gnu) libcurl/7.71.1 OpenSSL/1.1.1j-fips zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.1 (+libidn2/2.3.0) libssh/0.9.5/openssl/zlib nghttp2/1.43.0
Release-Date: 2020-07-01
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

operating system

Fedora 33
Linux 5.10.21-200.fc33.x86_64 #1 SMP Mon Mar 8 00:24:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[jwatt]

Thank you, @bagder! I really appreciate you fixing this, especially so quickly.