Download size does not match requests with multiple byte ranges #16139
Labels
Comments
|
The output is of type multipart/byteranges and the difference in expected vs.
actual sizes is the overhead inherent in multipart/byteranges format. You'll
need a program to decode multipart/byteranges to extract the range parts in
order to get the individual parts of the file.
|
jay
added a commit
to jay/curl
that referenced
this issue
Feb 1, 2025
- Warn that curl does not support decoding a server response containing multiple byte ranges. If a user requests multiple byte ranges (eg --range 1-2,3-4) and the server supports it then it replies with a multipart/byteranges response that is not decoded by curl. Prior to this change there was no notice and users could be caught unaware that the server response contained meta information in addition to the requested bytes. Reported-by: Ralf A. Timmermann Fixes curl#16139 Closes #xxxxx
jay
added a commit
to jay/curl
that referenced
this issue
Feb 1, 2025
- Warn during --range parsing that curl does not support decoding a server response containing multiple byte ranges. - Warn in CURLOPT_RANGE doc that multipart range responses are not decoded by libcurl. If a user requests multiple byte ranges (eg --range 1-2,3-4) and the server supports that then it replies with a multipart/byteranges response that is not decoded by curl. There is already a doc warning for --range about this, however prior to this change there was no usage warning for the tool and a less explicit doc warning for CURLOPT_RANGE. Reported-by: Ralf A. Timmermann Fixes curl#16139 Closes #xxxxx
|
curl does not support decoding multiple byte range responses and that is noted in the --range doc: (*) = NOTE that these make the server reply with a multipart response, which is returned as-is by curl. Parsing or otherwise transforming this response is the responsibility of the caller.
edit: I'm going with #16150 instead which adds some clarification to the doc but does not add a usage warning. |
jay
added a commit
to jay/curl
that referenced
this issue
Feb 1, 2025
- Better explain that if the requested range (--range or CURLOPT_RANGE) contains multiple ranges then the response contains meta information in addition to the requested bytes. Prior to this change it was noted that a multiple part response was returned as-is but not what that meant. In particular, meta information is returned in addition to the requested bytes and that may have been unexpected. Reported-by: Ralf A. Timmermann Fixes curl#16139 Closes #xxxx
|
Sounds good. Thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I did this
curl --max-time 60 --retry 2 -f -s -r 101503120-106075451,108837960-111876728 https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20250131/06/atmos/gfs.t06z.sfluxgrbf000.grib2 --output data0.grib2 -w '%{size_download}'
I expected the following
The downloaded size was 7611277, whilst the sum of byte ranges was 7611099
curl/libcurl version
$ curl -V
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
operating system
$ uname -a
Linux ralf-Latitude-5500 5.4.0-205-generic #225-Ubuntu SMP Fri Jan 10 22:23:35 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
The text was updated successfully, but these errors were encountered: