net/http/cgi: allow chunked body in request

[gopherbot]

by cgmurray:

What steps will reproduce the problem?
Why are requests with chunked transfer encoding not permitted in http/cgi? Is this
according to the cgi specification? Git sends data (push) with chunked transfer encoding
and I believe that it's supported in Apache.

Side-note: By disabling the error-check in ServeHTTP chunked requests seems work in
general but when sending large data, >= 1MB,  in combination with writing the
"Content-Type" header the request data seems to be corrupted. Any other
header, e.g. "_Content-Type" works however.

Which version are you using?  (run 'go version')
go version go1.1 darwin/amd64

[bradfitz]

Comment 1:

I think the CGI "specification" predates HTTP/1.1, so it's likely it doesn't say
anything specifically about it.
Can you confirm that Apache or some other popular CGI host supports this? If so, include
the environment variables that a child CGI process under Apache sees.

Status changed to WaitingForReply.

[bradfitz]

Comment 2:

Labels changed: removed priority-triage.

[gopherbot]

Comment 3 by cgmurray:

Hi,
Tested with the following cgi-script:
-----------------------------------
#!/bin/sh
echo "Content-Type: application/octet-stream"
echo ""
set
md5
-----------------------------------
md5 /tmp/large_file
MD5 (/tmp/large_file) = 768c49f96473c4aa1d9efc5aaa094c1f
curl  -X POST --data-binary @/tmp/large_file http://localhost:9090/md5/
...
768c49f96473c4aa1d9efc5aaa094c1f
Differences between chunked and regular:
curl -X POST --data-binary @/tmp/large_file http://localhost:9090/md5/ -v > /tmp/a
curl -H "Transfer-Encoding: chunked" -X POST --data-binary @/tmp/large_file
http://localhost:9090/md5/ -v > /tmp/b
diff /tmp/a /tmp/b
8d7
< CONTENT_LENGTH=1024000
21a21
> HTTP_TRANSFER_ENCODING=chunked
37c37
< REMOTE_PORT=59660
---
> REMOTE_PORT=59661
Note that the data is "de-chunked" as the md5-checksum is correct when chunked encoding
is on.
I can create a complete go-example to demonstrate the strange behavior when
"Content-Type" and only "Content-Type" is set as described in first post if you like.

[gopherbot]

Comment 4 by cgmurray:

Forgot to mention that the web server is Apache/2.2.22

[gopherbot]

Comment 5 by cgmurray:

Hi,
The comments are out of order as I forget to fill in the captcha for the first one.
Tested with the following cgi-script:
----------------------------------------------
#!/bin/sh
echo "Content-Type: application/octet-stream"
echo ""
set
md5
----------------------------------------------
md5 /tmp/large_file
MD5 (/tmp/large_file) = 768c49f96473c4aa1d9efc5aaa094c1f
curl -H "Transfer-Encoding: chunked" -X POST --data-binary @/tmp/large_file
http://localhost:9090/md5/
...
768c49f96473c4aa1d9efc5aaa094c1f
Differences between chunked and regular:
curl -X POST --data-binary @/tmp/large_file http://localhost:9090/md5/ -v > /tmp/a
curl -H "Transfer-Encoding: chunked" -X POST --data-binary @/tmp/large_file
http://localhost:9090/md5/ -v > /tmp/b
diff /tmp/a /tmp/b
8d7
< CONTENT_LENGTH=1024000
21a21
> HTTP_TRANSFER_ENCODING=chunked
37c37
< REMOTE_PORT=59668
---
> REMOTE_PORT=59669
Note that the data is "de-chunked" as the md5 is correct.
I can create a complete go-example demonstrating the strange behavior when
"Content-Type" and only "Content-Type" is set as described briefly in the first post if
you like.
Thanks!

[gopherbot]

Comment 6 by cgmurray:

I've tried to post a complete example twice. Perhaps caught in the spam-filter due to
links?

[gopherbot]

Comment 7 by cgmurray:

Here is the post a tried to send as an attachment instead.

Attachments:

1. chunked.txt (1050 bytes)

[gopherbot]

Comment 8 by cgmurray:

Any news on this issue? See also issue #5660 for a potentially related problem.

[rsc]

Comment 9:

Labels changed: added go1.2maybe, priority-later.

Status changed to Accepted.

[rsc]

Comment 10:

Labels changed: added feature.

[robpike]

Comment 11:

Not for 1.2.

Labels changed: removed go1.2maybe.

[rsc]

Comment 12:

Labels changed: added go1.3maybe.

[rsc]

Comment 13:

Labels changed: removed feature.

[rsc]

Comment 14:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 15:

Labels changed: added repo-main.