Can not send 475020 byte file over chunked transfer encoding.

[moohoorama]

I did this

Whenever sending 475020 byte file over chunked transfer encoding, a curl doesn't send last contents chunk. As far as I know, the curl sends 29 chunks of 16372 byte, and sends last contents chunk of 232 byte, finally, sends a zero chunk, which means "transferred all contents". But a curl doesn't send last contents chunk of 232 byte. So, a HTTP Servers wait to recv a zero chunk, Both client and server waits for each other, This request fail.
475018, 475019, 475021, 475022, All okay.

ex)

1. 475020 byte

• Client

$ truncate -s 475020 test.file
$ curl    -H "transfer-encoding:chunked" http://localhost:44444/ -T test.file -vvv -H 'Expect: '
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 44444 (#0)
> PUT /test.file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:44444
> Accept: */*
> transfer-encoding:chunked
> 
* We are completely uploaded and fine

• Server

PUT /test.file HTTP/1.1
User-Agent: curl/7.35.0
Host: localhost:44444
Accept: */*
transfer-encoding:chunked


3ff4
XXXXXXXX
..
..
..
3ff4
XXXXXXXX

2. 475021 byte

• Client

$ truncate -s 475021 test.file
curl    -H "transfer-encoding:chunked" http://localhost:44444/ -T test.file -vvv -H 'Expect: '
...
> Accept: */*
> transfer-encoding:chunked
*
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BaseHTTP/0.3 Python/2.7.6
< Date: Fri, 20 Oct 2017 05:47:18 GMT
* we are done reading and this is set to close, stop send
* Closing connection 0

• Server

PUT /test.file HTTP/1.1
...

3ff4
XXXXXXXX
..
..
..
3ff4
XXXXX...
e9
XXXXX...

0

2. 475019 byte

• Client

$ truncate -s 475021 test.file
curl    -H "transfer-encoding:chunked" http://localhost:44444/ -T test.file -vvv -H 'Expect: '
...
> Accept: */*
> transfer-encoding:chunked
*
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BaseHTTP/0.3 Python/2.7.6
< Date: Fri, 20 Oct 2017 05:47:18 GMT
* we are done reading and this is set to close, stop send
* Closing connection 0

• Server

PUT /test.file HTTP/1.1
...

3ff4
XXXXXXXX
..
..
..
3ff4
XXXXX...
e7
XXXXX...

0

I expected the following

A curl send last 232 bytes chunk, and zero chunk.

curl/libcurl version

curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3

operating system

Ubuntu 14.04.5 LTS
Kernel 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[jay]

I can reproduce this in the latest curl. There is code that does not seem to account for chunked encoding, for example in the fail case it detects that the file has finished uploading when it hasn't:

curl/lib/transfer.c

Lines 1047 to 1053 in 3ea7679

	k->writebytecount += bytes_written;
	if(k->writebytecount == data->state.infilesize) {
	/* we have sent all data we were supposed to */
	k->upload_done = TRUE;
	infof(data, "We are completely uploaded and fine\n");
	}

Yet in the success case (I used 475021) that's never hit (shouldn't be anyway in my opinion), it just keeps uploading:

bytes: 16384
fillcount: 16380
k->writebytecount = k->writebytecount + bytes_written (442260 + 16380)
k->writebytecount: 458640
bytes: 16384
fillcount: 16380
k->writebytecount = k->writebytecount + bytes_written (458640 + 16380)
k->writebytecount: 475020
bytes: 16384
fillcount: 239
k->writebytecount = k->writebytecount + bytes_written (475020 + 239)
k->writebytecount: 475259
bytes: 16384
fillcount: 5
k->writebytecount = k->writebytecount + bytes_written (475259 + 5)
k->writebytecount: 475264

bytes is what's going in to fillreadbuffer and fillcount is what comes out. it just happens to work because k->writebytecount is never 475021 so it never caps it

so we could check chunked there and ignore infilesize in that case, or keep track of the whole thing? but if it's chunked we may not know the filesize. fillreadbuffer has some logic to deal with chunks

curl/lib/transfer.c

Lines 130 to 134 in 3ea7679

	if(data->req.upload_chunky) {
	/* if chunked Transfer-Encoding */
	buffersize -= (8 + 2 + 2); /* 32bit hex + CRLF + CRLF */
	data->req.upload_fromhere += (8 + 2); /* 32bit hex + CRLF */
	}

[jay]

I can reproduce this in the latest curl. There is code that does not seem to account for chunked encoding, for example in the fail case it detects that the file has finished uploading when it hasn't:

curl/lib/transfer.c

Lines 1047 to 1053 in 3ea7679

	k->writebytecount += bytes_written;
	if(k->writebytecount == data->state.infilesize) {
	/* we have sent all data we were supposed to */
	k->upload_done = TRUE;
	infof(data, "We are completely uploaded and fine\n");
	}

Yet in the success case (I used 475021) that's never hit (shouldn't be anyway in my opinion), it just keeps uploading:

bytes: 16384
fillcount: 16380
k->writebytecount = k->writebytecount + bytes_written (442260 + 16380)
k->writebytecount: 458640
bytes: 16384
fillcount: 16380
k->writebytecount = k->writebytecount + bytes_written (458640 + 16380)
k->writebytecount: 475020
bytes: 16384
fillcount: 239
k->writebytecount = k->writebytecount + bytes_written (475020 + 239)
k->writebytecount: 475259
bytes: 16384
fillcount: 5
k->writebytecount = k->writebytecount + bytes_written (475259 + 5)
k->writebytecount: 475264

bytes is what's going in to fillreadbuffer and fillcount is what comes out. it just happens to work because k->writebytecount is never 475021 so it never caps it

so we could check chunked there and ignore infilesize in that case, or keep track of the whole thing? but if it's chunked we may not know the filesize. fillreadbuffer has some logic to deal with chunks

curl/lib/transfer.c

Lines 130 to 134 in 3ea7679

	if(data->req.upload_chunky) {
	/* if chunked Transfer-Encoding */
	buffersize -= (8 + 2 + 2); /* 32bit hex + CRLF + CRLF */
	data->req.upload_fromhere += (8 + 2); /* 32bit hex + CRLF */
	}

[bagder]

transfer.c:1049 (if(k->writebytecount == data->state.infilesize)) should indeed not be checked if chunked encoding is used, that looks like a blatant bug to me!

[bagder]

transfer.c:1049 (if(k->writebytecount == data->state.infilesize)) should indeed not be checked if chunked encoding is used, that looks like a blatant bug to me!

[jay]

how about

diff --git a/lib/transfer.c b/lib/transfer.c
index 8e66d0d..63d673e 100644
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -1046,7 +1046,8 @@ static CURLcode readwrite_upload(struct Curl_easy *data,
 
     k->writebytecount += bytes_written;
 
-    if(k->writebytecount == data->state.infilesize) {
+    if((!data->req.upload_chunky || data->req.forbidchunk) &&
+       (k->writebytecount == data->state.infilesize)) {
       /* we have sent all data we were supposed to */
       k->upload_done = TRUE;
       infof(data, "We are completely uploaded and fine\n");

in the case of chunked afaics readbuffer sets the upload_done even though the data is just read in to the buffer, and the upload is not necessarily done.. seems kind of misleading

curl/lib/transfer.c

Lines 241 to 243 in 3ea7679

	if((nread - hexlen) == 0)
	/* mark this as done once this chunk is transferred */
	data->req.upload_done = TRUE;

[jay]

how about

diff --git a/lib/transfer.c b/lib/transfer.c
index 8e66d0d..63d673e 100644
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -1046,7 +1046,8 @@ static CURLcode readwrite_upload(struct Curl_easy *data,
 
     k->writebytecount += bytes_written;
 
-    if(k->writebytecount == data->state.infilesize) {
+    if((!data->req.upload_chunky || data->req.forbidchunk) &&
+       (k->writebytecount == data->state.infilesize)) {
       /* we have sent all data we were supposed to */
       k->upload_done = TRUE;
       infof(data, "We are completely uploaded and fine\n");

in the case of chunked afaics readbuffer sets the upload_done even though the data is just read in to the buffer, and the upload is not necessarily done.. seems kind of misleading

curl/lib/transfer.c

Lines 241 to 243 in 3ea7679

	if((nread - hexlen) == 0)
	/* mark this as done once this chunk is transferred */
	data->req.upload_done = TRUE;

[jay]

seems kind of misleading

In fairness I found it noted in the comments.

curl/lib/urldata.h

Lines 583 to 584 in 3ea7679

	bool upload_done; /* set to TRUE when doing chunked transfer-encoding upload
	and we're uploading the last chunk */

[jay]

seems kind of misleading

In fairness I found it noted in the comments.

curl/lib/urldata.h

Lines 583 to 584 in 3ea7679

	bool upload_done; /* set to TRUE when doing chunked transfer-encoding upload
	and we're uploading the last chunk */

[bagder]

how about

Yes, seems appropriate. Make it a PR and let's see that nothing obvious breaks from it!

[bagder]

how about

Yes, seems appropriate. Make it a PR and let's see that nothing obvious breaks from it!

[jay]

Fixed in 979d287, thanks

[jay]

Fixed in 979d287, thanks