curl appends error messages for 416 status code from server to the downloaded file with option "-C -" #1163

Closed
ithubg opened this Issue Dec 13, 2016 · 7 comments

Comments

Projects
None yet
6 participants
@ithubg

ithubg commented Dec 13, 2016

I did this

There is a file of size 100 bytes on HTTP server(nginx).

kohji_@comp-ppl-fs03:~$ curl -C - -o 100b.bin 'http://ppl-fs01:80/aos/static/100b.bin'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100 100 100 0 0 16917 0 --:--:-- --:--:-- --:--:-- 20000
kohji_@comp-ppl-fs03:~$ ls -l 100b.bin
-rw-rw-r-- 1 kohji_ kohji_ 100 Dec 13 12:04 100b.bin
kohji_@comp-ppl-fs03:~$ curl -C - -o 100b.bin 'http://ppl-fs01:80/aos/static/100b.bin'
** Resuming transfer from byte position 100
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 222 100 222 0 0 127k 0 --:--:-- --:--:-- --:--:-- 216k
kohji_@comp-ppl-fs03:~$ ls -l 100b.bin
-rw-rw-r-- 1 kohji_ kohji_ 322 Dec 13 12:04 100b.bin

File size grew to 322 bytes!

kohji_@comp-ppl-fs03:~$ od -c 100b.bin
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0000140 \0 \0 \0 \0 < h t m l > \r \n < h e a
0000160 d > < t i t l e > 4 1 6 R e q
0000200 u e s t e d R a n g e N o t
0000220 S a t i s f i a b l e < / t i
0000240 t l e > < / h e a d > \r \n < b o
0000260 d y b g c o l o r = " w h i t
0000300 e " > \r \n < c e n t e r > < h 1
0000320 > 4 1 6 R e q u e s t e d R
0000340 a n g e N o t S a t i s f i
0000360 a b l e < / h 1 > < / c e n t e
0000400 r > \r \n < h r > < c e n t e r >
0000420 n g i n x / 1 . 1 0 . 0 ( U b
0000440 u n t u ) < / c e n t e r > \r \n
0000460 < / b o d y > \r \n < / h t m l >
0000500 \r \n
0000502

I expected the following

I expects that curl ignores body of HTTP response when its status code is 416 as wget behaves

curl/libcurl version

curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

operating system

Linux comp-ppl-fs03 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

@jay

This comment has been minimized.

Show comment
Hide comment
@jay

jay Dec 13, 2016

Member

The only way I think that could happen is if the 416 returns a Content-Range. I don't recall offhand if that's allowed. Run curl in verbose mode to confirm.

* Rebuilt URL to: http://localhost:8000/
*   Trying ::1...
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> Range: bytes=3-
> User-Agent: curl/7.47.0-DEV
> Accept: */*
>
< HTTP/1.1 416 Requested Range Not Satisfiable
< Accept-Ranges: bytes
< Content-Range: bytes 3-4/5
< Content-Length: 2
<
{ [2 bytes data]
* Connection #0 to host localhost left intact

12/13/2016  02:26 AM                 5 100b.bin

A server could also return a 404 not found as a 216 partial , I think, so in that case you would also end up with bad appended data in your file. I don't know the best solution for that (or any solution, actually..), but in the 416 case I think you should use --fail.

Member

jay commented Dec 13, 2016

The only way I think that could happen is if the 416 returns a Content-Range. I don't recall offhand if that's allowed. Run curl in verbose mode to confirm.

* Rebuilt URL to: http://localhost:8000/
*   Trying ::1...
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> Range: bytes=3-
> User-Agent: curl/7.47.0-DEV
> Accept: */*
>
< HTTP/1.1 416 Requested Range Not Satisfiable
< Accept-Ranges: bytes
< Content-Range: bytes 3-4/5
< Content-Length: 2
<
{ [2 bytes data]
* Connection #0 to host localhost left intact

12/13/2016  02:26 AM                 5 100b.bin

A server could also return a 404 not found as a 216 partial , I think, so in that case you would also end up with bad appended data in your file. I don't know the best solution for that (or any solution, actually..), but in the 416 case I think you should use --fail.

@ithubg

This comment has been minimized.

Show comment
Hide comment
@ithubg

ithubg Dec 13, 2016

Thanks for you comment.

According to RFC 7233, Content-Range should be added to the response with 416.
[https://tools.ietf.org/html/rfc7233#section-4.4]

And at least nginx does:

kohji_@comp-ppl-fs03:~$ curl -v -C - -o 100b.bin  'http://ppl-fs01:80/aos/static/100b.bin'
** Resuming transfer from byte position 100
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.1.100...
* Connected to ppl-fs01 (192.168.1.100) port 80 (#0)
> GET /aos/static/100b.bin HTTP/1.1
> Host: ppl-fs01
> Range: bytes=100-
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 416 Requested Range Not Satisfiable
< Server: nginx/1.10.0 (Ubuntu)
< Date: Tue, 13 Dec 2016 08:01:35 GMT
< Content-Type: text/html
< Content-Length: 222
< Connection: keep-alive
< Content-Range: bytes */100
< 
{ [222 bytes data]

I tried --fail option and it worked. But the option makes it harder for caller of curl to distinguish 416 from 404 or other possible error.

I think curl should handle 416 with Content-Range: */100 as OK and discard body part for the request with Range: 100-.

ithubg commented Dec 13, 2016

Thanks for you comment.

According to RFC 7233, Content-Range should be added to the response with 416.
[https://tools.ietf.org/html/rfc7233#section-4.4]

And at least nginx does:

kohji_@comp-ppl-fs03:~$ curl -v -C - -o 100b.bin  'http://ppl-fs01:80/aos/static/100b.bin'
** Resuming transfer from byte position 100
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.1.100...
* Connected to ppl-fs01 (192.168.1.100) port 80 (#0)
> GET /aos/static/100b.bin HTTP/1.1
> Host: ppl-fs01
> Range: bytes=100-
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 416 Requested Range Not Satisfiable
< Server: nginx/1.10.0 (Ubuntu)
< Date: Tue, 13 Dec 2016 08:01:35 GMT
< Content-Type: text/html
< Content-Length: 222
< Connection: keep-alive
< Content-Range: bytes */100
< 
{ [222 bytes data]

I tried --fail option and it worked. But the option makes it harder for caller of curl to distinguish 416 from 404 or other possible error.

I think curl should handle 416 with Content-Range: */100 as OK and discard body part for the request with Range: 100-.

@bagder bagder added the HTTP label Dec 14, 2016

@krayon

This comment has been minimized.

Show comment
Hide comment
@krayon

krayon Apr 11, 2017

I agree, the only sane action seems to be, at least to me, handling the 416 by NOT writing the output to the file and then perhaps returning an error code. Is there any use case where the current action is valid? Any circumstance where a 416 would be returned with the remaining content? I wouldn't have thought so.

krayon commented Apr 11, 2017

I agree, the only sane action seems to be, at least to me, handling the 416 by NOT writing the output to the file and then perhaps returning an error code. Is there any use case where the current action is valid? Any circumstance where a 416 would be returned with the remaining content? I wouldn't have thought so.

krayon added a commit to krayon/packtdl that referenced this issue Apr 11, 2017

@GermanG

This comment has been minimized.

Show comment
Hide comment
@GermanG

GermanG Apr 14, 2017

The 416 is omitted on purpose, looking at the code:

 if(data->state.resume_from &&
            (data->set.httpreq==HTTPREQ_GET) &&
            (k->httpcode == 416)) {
           /* "Requested Range Not Satisfiable", just proceed and
              pretend this is no error */
         }
         else {
           /* serious error, go home! */
           print_http_error(data);
           return CURLE_HTTP_RETURNED_ERROR;
         }

GermanG commented Apr 14, 2017

The 416 is omitted on purpose, looking at the code:

 if(data->state.resume_from &&
            (data->set.httpreq==HTTPREQ_GET) &&
            (k->httpcode == 416)) {
           /* "Requested Range Not Satisfiable", just proceed and
              pretend this is no error */
         }
         else {
           /* serious error, go home! */
           print_http_error(data);
           return CURLE_HTTP_RETURNED_ERROR;
         }
@stale

This comment has been minimized.

Show comment
Hide comment
@stale

stale bot Oct 11, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale bot commented Oct 11, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Oct 11, 2017

@krayon

This comment has been minimized.

Show comment
Hide comment
@krayon

krayon Oct 11, 2017

If it's not on anyone's radar I will try to submit a PR soon (that probably means a month or so). I personally can't really use curl for anything whilst this bug exists, it's a fairly high sev for me.

krayon commented Oct 11, 2017

If it's not on anyone's radar I will try to submit a PR soon (that probably means a month or so). I personally can't really use curl for anything whilst this bug exists, it's a fairly high sev for me.

@stale stale bot removed the stale label Oct 11, 2017

@jay jay added the help wanted label Nov 21, 2017

@strowger

This comment has been minimized.

Show comment
Hide comment
@strowger

strowger Feb 3, 2018

I am experiencing this issue as well.

It affects the use of curl with the webserver embedded in the Goluk T series of dashcams, which is nginx 1.9.6 and some sort of (unknown/undocumented) custom CGI.

The first attempt:

$ curl -C - http://DASHCAM-IP/api/video?id=WND_20180129162526 -o WND_20180129162526.mp4

Successfully downloads the file.

Every subsequent attempt with the same commandline results in the following being appended to the file:

<head><title>416 Requested Range Not Satisfiable</title></head>
<body bgcolor="white">
<center><h1>416 Requested Range Not Satisfiable</h1></center>
<hr><center>nginx/1.9.6</center>
</body>
</html>

strowger commented Feb 3, 2018

I am experiencing this issue as well.

It affects the use of curl with the webserver embedded in the Goluk T series of dashcams, which is nginx 1.9.6 and some sort of (unknown/undocumented) custom CGI.

The first attempt:

$ curl -C - http://DASHCAM-IP/api/video?id=WND_20180129162526 -o WND_20180129162526.mp4

Successfully downloads the file.

Every subsequent attempt with the same commandline results in the following being appended to the file:

<head><title>416 Requested Range Not Satisfiable</title></head>
<body bgcolor="white">
<center><h1>416 Requested Range Not Satisfiable</h1></center>
<hr><center>nginx/1.9.6</center>
</body>
</html>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment