net/http: support "gzip" as a Transfer Encoding

[rv-dtomaz]

What version of Go are you using (go version)?

go version go1.10.3 windows/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\xxxx\AppData\Local\go-build
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\xxxx\go
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\xxxx\AppData\Local\Temp\go-build557436246=/tmp/go-build -gno-record-gcc-switches

What did you do?

I am requesting a URL with code below:

package main

import (
	"fmt"
	"net/http"
	"time"
)

func main() {

	
	urlReq2 := "https://www.azulseguros.com.br/azul-cms/wp-content/themes/azul2016/integracao-azul-leve.php?timestamp=NGJYHX7MXNS"
	req, err := http.NewRequest("GET", urlReq2, nil)
	netClient := &http.Client{
		Timeout: time.Second * 10,
	}
	_, err = netClient.Do(req)
	if err != nil {
		fmt.Println(err)
	}

	

}

I got this error:
net/http: HTTP/1.x transport connection broken: unsupported transfer encoding "gzip"

What did you expect to see?

I expect to be foward to other page and get HTML.
In other languages like nodejs and c#, the same request is OK.
Running cURL also bring me the result.

What did you see instead?

Error on netClient.Do():
net/http: HTTP/1.x transport connection broken: unsupported transfer encoding "gzip"

[odeke-em]

Thank you for filing this issue @rv-dtomaz and welcome to the Go project!

So I believe we only support "identity" and "chunked" transfer encodings. However, I've retitled this
as a feature request and unfortunately we are in the Go1.12 so this will have to wait until Go1.13.

@bradfitz if I may indulge you, perhaps we should document that we only support "identity" and "chunked" transfer encodings, for this cycle? In the next cycle we can do a net/http hackathon where we examine all supported transfer encodings and perhaps add them.

[odeke-em]

And here is a standalone repro that doesn't need to make the network call, obtained by the results of calling that server https://play.golang.org/p/ND1AgZHXoDZ or inlined below

package main

import (
	"bufio"
	"io"
	"io/ioutil"
	"log"
	"net/http"
	"strings"
)

func main() {
	res, err := http.ReadResponse(bufio.NewReader(strings.NewReader(
		`HTTP/1.1 302 Found
Date: Sun, 09 Dec 2018 21:00:11 GMT
Server: Apache
X-Powered-By: PHP/5.6.25
Set-Cookie: PHPSESSID=0pn7kf6i398bml60vnjr28m1o7; path=/
Expires: Sat, 01 Jan 2017 01:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: https://as.azulseguros.com.br/AzulLeve/index.jsf?token=b75c366dbccb65be524d215aa9a1a55ff4f813230b3f18795a2c1edda4509339
Transfer-Encoding: gzip
Vary: Accept-Encoding
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8
Set-Cookie: AzulSeguros-WWW_HTTPS=EBABABAK; Expires=Sun, 09-Dec-2018 23:00:11 GMT; Path=/

`)), nil)
	if err != nil {
		log.Fatalf("Failed to parse response: %v", err)
	}
	io.Copy(ioutil.Discard, res.Body)
	_ = res.Body.Close()
}

Failed to parse response: unsupported transfer encoding "gzip"

[rv-dtomaz]

Hi @odeke-em ,thanks a lot for your quick suport.
I suspected that this was the problem.
Thanks a lot.
If I can help in solution, I am here.

best

[rv-dtomaz]

@odeke-em Is there any workaround about this ? Something that we could do temporary.
I would really apreciate.

best

[bradfitz]

Good overview of Content-Encoding gzip vs Transfer-Encoding gzip: https://stackoverflow.com/questions/11641923/transfer-encoding-gzip-vs-content-encoding-gzip

[odeke-em]

@odeke-em Is there any workaround about this ? Something that we could do temporary.
I would really apreciate.

Great question @rv-dtomaz! My apologies for the late reply, I just saw this on waking up early.

So currently the work around that I can think of is a bit of a hackasauraus but it'll produce the identical response to the last response from curl -i -L $URL
It basically involves trying the request, if it spits out the error you've encountered, then:
a) Make a connection the server
b) Speak HTTP/1.1 to it
c) If it returns "Transfer-Encoding":"gzip", turn that into "Content-Encoding":"gzip"
d) Rewire the entire request's body and invoke http.ReadResponse

and here it is https://gist.github.com/odeke-em/39c8ecb7522edf80f1d91df71e415340 or inlined below

package main

import (
	"bufio"
	"bytes"
	"crypto/tls"
	"fmt"
	"io"
	"log"
	"net"
	"net/http"
	"net/http/httputil"
	"net/textproto"
	"strings"
	"time"
)

func main() {
	uri := "https://www.azulseguros.com.br/azul-cms/wp-content/themes/azul2016/integracao-azul-leve.php?timestamp=NGJYHX7MXNS"
	req, err := http.NewRequest("GET", uri, nil)
	netClient := &http.Client{
		Timeout:   time.Second * 10,
		Transport: &transferEncodingRedactor{base: http.DefaultTransport.(*http.Transport)},
	}
	res, err := netClient.Do(req)
	if err != nil {
		log.Fatalf("Failed to make request: %v", err)
	}
	blob, _ := httputil.DumpResponse(res, true)
	fmt.Printf("%s\n", blob)
	_ = res.Body.Close()
}

type transferEncodingRedactor struct {
	base *http.Transport
}

func (rt *transferEncodingRedactor) RoundTrip(req *http.Request) (*http.Response, error) {
	// Optimistic route i.e common case.
	res, err := rt.base.RoundTrip(req)
	if err == nil || res != nil {
		return res, err
	}

	// An error occured here but now time ot examine the contents
	if !strings.Contains(err.Error(), "unsupported transfer encoding") {
		return res, err
	}

	u := req.URL
	var conn net.Conn
	switch req.URL.Scheme {
	case "https":
		tr := rt.base
		if tr == nil {
			tr = http.DefaultTransport.(*http.Transport)
		}
		conn, err = tls.Dial("tcp", "www.azulseguros.com.br:443", tr.TLSClientConfig)
	default:
		conn, err = net.Dial("tcp", "www.azulseguros.com.br:80")
	}

	if err != nil {
		return nil, fmt.Errorf("Dial error: %v", err)
	}

	// Ensure we close the connection after.
	// Perhaps you might want to reuse it per host?
	defer conn.Close()
	br := bufio.NewReader(conn)
	bw := bufio.NewWriter(conn)

	s := u.Path
	if q := u.RawQuery; len(q) > 0 {
		s += "?" + q
	}

	intro := fmt.Sprintf("%s %s HTTP/1.1\r\nHost: %s\r\n\r\n", req.Method, s, u.Host)
	fmt.Fprint(bw, intro)
	bw.Flush()

	// And now we've got the case of an unsupported transfer
	// encoding time for a raw fetch and header rewrite.
	tp := textproto.NewReader(br)

	protoLine, err := tp.ReadLine()
	if err != nil {
		return nil, fmt.Errorf("ReadLine error: %v", err)
	}
	// Time to parse the headers
	rhdr, err := tp.ReadMIMEHeader()
	if err != nil {
		return nil, fmt.Errorf("ReadMIMEHeader: %v", err)
	}
	hdr := http.Header(rhdr)
	// Now rewrite the header
	if te := hdr.Get("Transfer-Encoding"); te == "gzip" {
		hdr.Set("Content-Encoding", "gzip")
		delete(hdr, "Transfer-Encoding")
	}

	// After this rewrite, reassemble the already read parts back
	// and then get them to ReadResponse
	hdrBuf := new(bytes.Buffer)
	if err := hdr.Write(hdrBuf); err != nil {
		return nil, fmt.Errorf("Rewriting header to wire: %v", err)
	}

	reconstructed := io.MultiReader(strings.NewReader(protoLine+"\r\n"), hdrBuf, strings.NewReader("\r\n"), br)
	return http.ReadResponse(bufio.NewReader(reconstructed), req)
}

Response

HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Date: Mon, 10 Dec 2018 16:52:57 GMT
Server: WildFly/10
Set-Cookie: JSESSIONID=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2; path=/AzulLeve
Set-Cookie: ASAzul-Zafira_SSL=FDABABAK; Expires=Mon, 10-Dec-2018 18:52:57 GMT; Path=/
Vary: Accept-Encoding
X-Powered-By: Undertow/1

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"><head><link type="text/css" rel="stylesheet" href="/AzulLeve/javax.faces.resource/theme.css.jsf?ln=primefaces-redmond" /><link type="text/css" rel="stylesheet" href="/AzulLeve/javax.faces.resource/primefaces.css.jsf;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?ln=primefaces&amp;v=5.3&amp;v=5.3&amp;v=5.3" /><script type="text/javascript" src="/AzulLeve/javax.faces.resource/jquery/jquery.js.jsf;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?ln=primefaces&amp;v=5.3&amp;v=5.3&amp;v=5.3"></script><script type="text/javascript" src="/AzulLeve/javax.faces.resource/jquery/jquery-plugins.js.jsf;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?ln=primefaces&amp;v=5.3&amp;v=5.3&amp;v=5.3"></script><script type="text/javascript" src="/AzulLeve/javax.faces.resource/primefaces.js.jsf;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?ln=primefaces&amp;v=5.3&amp;v=5.3&amp;v=5.3"></script><script type="text/javascript">if(window.PrimeFaces){PrimeFaces.settings.projectStage='Development';}</script>

	<title>
		AZUL AUTO LEVE / POPULAR	
	</title>
	<meta http-equiv="expires" content="0" />
	<meta http-equiv="pragma" content="no-cache" />
	<meta http-equiv="cache-control" content="no-cache" />
	<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
	<meta http-equiv="X-UA-Compatible" content="IE=9" />
	<meta http-equiv="X-UA-Compatible" content="IE=8" />
	<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
	<link rel="stylesheet" href="/AzulLeve/resources/css/application.css" type="text/css" />
				<script type="text/javascript" src="/AzulLeve/resources/js/AzulLeve.js">
		</script></head><body>
		<div name="mensagens"><div id="messages" class="ui-messages ui-widget" aria-live="polite"></div>			
		</div>	
		
		<main class="content">

		<p class="labelazul">AZUL AUTO LEVE</p><div id="j_idt8" class="ui-panel ui-widget ui-widget-content ui-corner-all" data-widget="widget_j_idt8"><div id="j_idt8_content" class="ui-panel-content ui-widget-content">
			<h3 class="errotitulo">Sessão expirada.</h3><img id="j_idt10" src="/AzulLeve/resources/img/fim_sessao.png;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?pfdrid_c=true" alt="" class="erroimg" width="76" height="72" /></div></div><script id="j_idt8_s" type="text/javascript">PrimeFaces.cw("Panel","widget_j_idt8",{id:"j_idt8"});</script> 
		</main></body>
</html>

@bradfitz might cringe at me :)

[rv-dtomaz]

Hi @odeke-em ...
Don`t worry, this will help me a lot...
Thanks

[monis0395]

Hello, @odeke-em

will this be supported soon? I am encountering the error

In the next cycle we can do a net/http hackathon where we examine all supported transfer encodings and perhaps add them.

Thanks!

[odeke-em]

@monis0395 thanks for the ping! Did you see my suggested workaround in #29162 (comment)?

For sure, I'll prioritize this and roll out a CL to fix it in the next 4 days.

[monis0395]

Thank for the quick reply, sorry could respond earlier

yes I tried but it didn't work, just the error message got shorter 😛
from net/http: HTTP/1.x transport connection broken: unsupported transfer encoding "gzip"
to unsupported transfer encoding "gzip"

it's great to hear that it would be fix so soon 🙌🙌🙌

[monis0395]

Hi @odeke-em

I tried debugging it more and found that at line 86 the values are

protoLine: HTTP/1.1 500 Internal Server Error
err: nil

The endpoint which I am hitting is a http2 and content-type is text/event-stream is it because of that?

I dont if this helps but when I use fasthttp for that endpoint it works but I wanna avoid using fasthttp

Thanks

[gopherbot]

Change https://golang.org/cl/166517 mentions this issue: net/http: support gzip as a Transfer-Encoding

[monis0395]

Hi @odeke-em

I tried your code from https://golang.org/cl/166517, I copied tranfer.go to my local and checked my endpoint

Now I am getting 200 OK status 👍 but the contents are empty when I read them from response

any clue?

Thanks

[monis0395]

Hi @odeke-em

Yesterday I spent more time on this issue. I found the issue why the work around wasn't working from the suggested in #29162 (comment).

Instead of this

	intro := fmt.Sprintf("%s %s HTTP/1.1\r\nHost: %s\r\n\r\n", req.Method, s, u.Host)
	fmt.Fprint(bw, intro)
	bw.Flush()

	// And now we've got the case of an unsupported transfer
	// encoding time for a raw fetch and header rewrite.
        tp := textproto.NewReader(br)

       protoLine, err := tp.ReadLine()

I changed it to this

        intro := fmt.Sprintf("%s /%s HTTP/1.1\r\nHost:%s\r\n\r\n", req.Method, s, u.Host)
	fmt.Fprint(conn, intro)

	br := bufio.NewReader(conn)
	// And now we've got the case of an unsupported transfer
	// encoding time for a raw fetch and header rewrite.
        tp := textproto.NewReader(br)

       protoLine, err := tp.ReadLine()

Here is the full gist https://gist.github.com/monis0395/0202b2129e943989e83a828553077cde

But it still did not work when I am hitting to my server. I get protoLine = HTTP/1.1 500. I think some problem with the way I am connect to the server. I am completely new with textproto, can you please suggest where can i read more about it?

Also I tried the change u made in transfer.go it is working now. This time I added each of your change manually, last time I had download the file transfer.go. I don't know why it didn't work last time 😕

Good news is that It works now 😁

I want to use your change in my production code, so is it possible to use it without changing in the go/src/net/http/tranfer.go? Any bright ideas? Because I want to avoid changing the golang source code.

Thanks!

[odeke-em]

Sorry for the late reply @monis0395, I was quite swamped in the past quarters but I've made some small updates on CL https://go-review.googlesource.com/c/go/+/166517 and I shall kindly ping @bradfitz to take a look so that perhaps we can include it for Go1.14.

[odeke-em]

I want to use your change in my production code, so is it possible to use it without changing in the go/src/net/http/tranfer.go? Any bright ideas? Because I want to avoid changing the golang source code.

Unfortunately you'd need to deploy a custom fork otherwise we'd have to write a custom server which gets tricky ;)

[odeke-em]

@monis0395 if you get gotip or what will become Go1.14 (whose release candidates will be made soon) you'll have this in production. Cheers and thank you for the feature request!

[gopherbot]

Change https://golang.org/cl/215757 mentions this issue: Revert "net/http: support gzip, x-gzip Transfer-Encodings"

[FiloSottile]

Unfortunately the implementation looks broken for a Transfer-Encoding of "gzip" (without "chunked"), so I opened CL 215757 to roll it back from Go 1.14. The code in the original issue report fails with the following error:

net/http: HTTP/1.x transport connection broken: http: failed to gunzip body: unexpected EOF

Transfer-Encoding is one of the most delicate security surfaces of HTTP, and we've had a number of issues in the past with it, including request smuggling vulnerabilities, so I'd like to push back on implementing this, also considering most clients and servers use Content-Encoding instead.

[odeke-em]

Since this was reverted for Go1.14, we'll take a look at it for the next cycle for Go1.15 and I'll move the milestone.

[FiloSottile]

Transfer-Encoding is a very security sensitive surface. For example, that CL had almost certainly introduced a request smuggling vulnerability because an upstream Go server could be made to disagree with a proxy on chunking. I'd like to see more justification for why we need this feature, which AFAICT is not at all widely implemented.

[mark-kubacki]

Transfer-Encoding is theoretically interesting in the context of uploads, i. e. to have a smaller representation on the wire, but can be used to transmit zip bombs to exploit unbounded buffers.