Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: support "gzip" as a Transfer Encoding #29162

Open
rv-dtomaz opened this issue Dec 9, 2018 · 14 comments

Comments

@rv-dtomaz
Copy link

commented Dec 9, 2018

What version of Go are you using (go version)?

go version go1.10.3 windows/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\xxxx\AppData\Local\go-build
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\xxxx\go
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\xxxx\AppData\Local\Temp\go-build557436246=/tmp/go-build -gno-record-gcc-switches

What did you do?

I am requesting a URL with code below:

package main

import (
	"fmt"
	"net/http"
	"time"
)

func main() {

	
	urlReq2 := "https://www.azulseguros.com.br/azul-cms/wp-content/themes/azul2016/integracao-azul-leve.php?timestamp=NGJYHX7MXNS"
	req, err := http.NewRequest("GET", urlReq2, nil)
	netClient := &http.Client{
		Timeout: time.Second * 10,
	}
	_, err = netClient.Do(req)
	if err != nil {
		fmt.Println(err)
	}

	

}

I got this error:
net/http: HTTP/1.x transport connection broken: unsupported transfer encoding "gzip"

What did you expect to see?

I expect to be foward to other page and get HTML.
In other languages like nodejs and c#, the same request is OK.
Running cURL also bring me the result.

What did you see instead?

Error on netClient.Do():
net/http: HTTP/1.x transport connection broken: unsupported transfer encoding "gzip"

@odeke-em odeke-em changed the title Error on Request: Transport connection broken: unsupported transfer encoding "gzip" net/http: support "gzip" as a Transfer Encoding Dec 9, 2018

@odeke-em

This comment has been minimized.

Copy link
Member

commented Dec 9, 2018

Thank you for filing this issue @rv-dtomaz and welcome to the Go project!

So I believe we only support "identity" and "chunked" transfer encodings. However, I've retitled this
as a feature request and unfortunately we are in the Go1.12 so this will have to wait until Go1.13.

@bradfitz if I may indulge you, perhaps we should document that we only support "identity" and "chunked" transfer encodings, for this cycle? In the next cycle we can do a net/http hackathon where we examine all supported transfer encodings and perhaps add them.

@odeke-em odeke-em added this to the Go1.13 milestone Dec 9, 2018

@odeke-em

This comment has been minimized.

Copy link
Member

commented Dec 9, 2018

And here is a standalone repro that doesn't need to make the network call, obtained by the results of calling that server https://play.golang.org/p/ND1AgZHXoDZ or inlined below

package main

import (
	"bufio"
	"io"
	"io/ioutil"
	"log"
	"net/http"
	"strings"
)

func main() {
	res, err := http.ReadResponse(bufio.NewReader(strings.NewReader(
		`HTTP/1.1 302 Found
Date: Sun, 09 Dec 2018 21:00:11 GMT
Server: Apache
X-Powered-By: PHP/5.6.25
Set-Cookie: PHPSESSID=0pn7kf6i398bml60vnjr28m1o7; path=/
Expires: Sat, 01 Jan 2017 01:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: https://as.azulseguros.com.br/AzulLeve/index.jsf?token=b75c366dbccb65be524d215aa9a1a55ff4f813230b3f18795a2c1edda4509339
Transfer-Encoding: gzip
Vary: Accept-Encoding
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8
Set-Cookie: AzulSeguros-WWW_HTTPS=EBABABAK; Expires=Sun, 09-Dec-2018 23:00:11 GMT; Path=/

`)), nil)
	if err != nil {
		log.Fatalf("Failed to parse response: %v", err)
	}
	io.Copy(ioutil.Discard, res.Body)
	_ = res.Body.Close()
}
Failed to parse response: unsupported transfer encoding "gzip"

@odeke-em odeke-em added the help wanted label Dec 9, 2018

@rv-dtomaz

This comment has been minimized.

Copy link
Author

commented Dec 9, 2018

Hi @odeke-em ,thanks a lot for your quick suport.
I suspected that this was the problem.
Thanks a lot.
If I can help in solution, I am here.

best

@rv-dtomaz

This comment has been minimized.

Copy link
Author

commented Dec 9, 2018

@odeke-em Is there any workaround about this ? Something that we could do temporary.
I would really apreciate.

best

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

Good overview of Content-Encoding gzip vs Transfer-Encoding gzip: https://stackoverflow.com/questions/11641923/transfer-encoding-gzip-vs-content-encoding-gzip

@odeke-em

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

@odeke-em Is there any workaround about this ? Something that we could do temporary.
I would really apreciate.

Great question @rv-dtomaz! My apologies for the late reply, I just saw this on waking up early.

So currently the work around that I can think of is a bit of a hackasauraus but it'll produce the identical response to the last response from curl -i -L $URL
It basically involves trying the request, if it spits out the error you've encountered, then:
a) Make a connection the server
b) Speak HTTP/1.1 to it
c) If it returns "Transfer-Encoding":"gzip", turn that into "Content-Encoding":"gzip"
d) Rewire the entire request's body and invoke http.ReadResponse

and here it is https://gist.github.com/odeke-em/39c8ecb7522edf80f1d91df71e415340 or inlined below

package main

import (
	"bufio"
	"bytes"
	"crypto/tls"
	"fmt"
	"io"
	"log"
	"net"
	"net/http"
	"net/http/httputil"
	"net/textproto"
	"strings"
	"time"
)

func main() {
	uri := "https://www.azulseguros.com.br/azul-cms/wp-content/themes/azul2016/integracao-azul-leve.php?timestamp=NGJYHX7MXNS"
	req, err := http.NewRequest("GET", uri, nil)
	netClient := &http.Client{
		Timeout:   time.Second * 10,
		Transport: &transferEncodingRedactor{base: http.DefaultTransport.(*http.Transport)},
	}
	res, err := netClient.Do(req)
	if err != nil {
		log.Fatalf("Failed to make request: %v", err)
	}
	blob, _ := httputil.DumpResponse(res, true)
	fmt.Printf("%s\n", blob)
	_ = res.Body.Close()
}

type transferEncodingRedactor struct {
	base *http.Transport
}

func (rt *transferEncodingRedactor) RoundTrip(req *http.Request) (*http.Response, error) {
	// Optimistic route i.e common case.
	res, err := rt.base.RoundTrip(req)
	if err == nil || res != nil {
		return res, err
	}

	// An error occured here but now time ot examine the contents
	if !strings.Contains(err.Error(), "unsupported transfer encoding") {
		return res, err
	}

	u := req.URL
	var conn net.Conn
	switch req.URL.Scheme {
	case "https":
		tr := rt.base
		if tr == nil {
			tr = http.DefaultTransport.(*http.Transport)
		}
		conn, err = tls.Dial("tcp", "www.azulseguros.com.br:443", tr.TLSClientConfig)
	default:
		conn, err = net.Dial("tcp", "www.azulseguros.com.br:80")
	}

	if err != nil {
		return nil, fmt.Errorf("Dial error: %v", err)
	}

	// Ensure we close the connection after.
	// Perhaps you might want to reuse it per host?
	defer conn.Close()
	br := bufio.NewReader(conn)
	bw := bufio.NewWriter(conn)

	s := u.Path
	if q := u.RawQuery; len(q) > 0 {
		s += "?" + q
	}

	intro := fmt.Sprintf("%s %s HTTP/1.1\r\nHost: %s\r\n\r\n", req.Method, s, u.Host)
	fmt.Fprint(bw, intro)
	bw.Flush()

	// And now we've got the case of an unsupported transfer
	// encoding time for a raw fetch and header rewrite.
	tp := textproto.NewReader(br)

	protoLine, err := tp.ReadLine()
	if err != nil {
		return nil, fmt.Errorf("ReadLine error: %v", err)
	}
	// Time to parse the headers
	rhdr, err := tp.ReadMIMEHeader()
	if err != nil {
		return nil, fmt.Errorf("ReadMIMEHeader: %v", err)
	}
	hdr := http.Header(rhdr)
	// Now rewrite the header
	if te := hdr.Get("Transfer-Encoding"); te == "gzip" {
		hdr.Set("Content-Encoding", "gzip")
		delete(hdr, "Transfer-Encoding")
	}

	// After this rewrite, reassemble the already read parts back
	// and then get them to ReadResponse
	hdrBuf := new(bytes.Buffer)
	if err := hdr.Write(hdrBuf); err != nil {
		return nil, fmt.Errorf("Rewriting header to wire: %v", err)
	}

	reconstructed := io.MultiReader(strings.NewReader(protoLine+"\r\n"), hdrBuf, strings.NewReader("\r\n"), br)
	return http.ReadResponse(bufio.NewReader(reconstructed), req)
}

Response

HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Date: Mon, 10 Dec 2018 16:52:57 GMT
Server: WildFly/10
Set-Cookie: JSESSIONID=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2; path=/AzulLeve
Set-Cookie: ASAzul-Zafira_SSL=FDABABAK; Expires=Mon, 10-Dec-2018 18:52:57 GMT; Path=/
Vary: Accept-Encoding
X-Powered-By: Undertow/1

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"><head><link type="text/css" rel="stylesheet" href="/AzulLeve/javax.faces.resource/theme.css.jsf?ln=primefaces-redmond" /><link type="text/css" rel="stylesheet" href="/AzulLeve/javax.faces.resource/primefaces.css.jsf;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?ln=primefaces&amp;v=5.3&amp;v=5.3&amp;v=5.3" /><script type="text/javascript" src="/AzulLeve/javax.faces.resource/jquery/jquery.js.jsf;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?ln=primefaces&amp;v=5.3&amp;v=5.3&amp;v=5.3"></script><script type="text/javascript" src="/AzulLeve/javax.faces.resource/jquery/jquery-plugins.js.jsf;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?ln=primefaces&amp;v=5.3&amp;v=5.3&amp;v=5.3"></script><script type="text/javascript" src="/AzulLeve/javax.faces.resource/primefaces.js.jsf;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?ln=primefaces&amp;v=5.3&amp;v=5.3&amp;v=5.3"></script><script type="text/javascript">if(window.PrimeFaces){PrimeFaces.settings.projectStage='Development';}</script>

	<title>
		AZUL AUTO LEVE / POPULAR	
	</title>
	<meta http-equiv="expires" content="0" />
	<meta http-equiv="pragma" content="no-cache" />
	<meta http-equiv="cache-control" content="no-cache" />
	<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
	<meta http-equiv="X-UA-Compatible" content="IE=9" />
	<meta http-equiv="X-UA-Compatible" content="IE=8" />
	<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
	<link rel="stylesheet" href="/AzulLeve/resources/css/application.css" type="text/css" />
				<script type="text/javascript" src="/AzulLeve/resources/js/AzulLeve.js">
		</script></head><body>
		<div name="mensagens"><div id="messages" class="ui-messages ui-widget" aria-live="polite"></div>			
		</div>	
		
		<main class="content">

		<p class="labelazul">AZUL AUTO LEVE</p><div id="j_idt8" class="ui-panel ui-widget ui-widget-content ui-corner-all" data-widget="widget_j_idt8"><div id="j_idt8_content" class="ui-panel-content ui-widget-content">
			<h3 class="errotitulo">Sessão expirada.</h3><img id="j_idt10" src="/AzulLeve/resources/img/fim_sessao.png;jsessionid=4pq22naRmHI-EuX1FrYq5PUml_z1WOD4LPsdpp3K.sentraprdlb6:site-emissao-inst2?pfdrid_c=true" alt="" class="erroimg" width="76" height="72" /></div></div><script id="j_idt8_s" type="text/javascript">PrimeFaces.cw("Panel","widget_j_idt8",{id:"j_idt8"});</script> 
		</main></body>
</html>

@bradfitz might cringe at me :)

@rv-dtomaz

This comment has been minimized.

Copy link
Author

commented Dec 11, 2018

Hi @odeke-em ...
Don`t worry, this will help me a lot...
Thanks

@monis0395

This comment has been minimized.

Copy link

commented Mar 7, 2019

Hello, @odeke-em

will this be supported soon? I am encountering the error

In the next cycle we can do a net/http hackathon where we examine all supported transfer encodings and perhaps add them.

Thanks!

@odeke-em

This comment has been minimized.

Copy link
Member

commented Mar 7, 2019

@monis0395 thanks for the ping! Did you see my suggested workaround in #29162 (comment)?

For sure, I'll prioritize this and roll out a CL to fix it in the next 4 days.

@odeke-em odeke-em self-assigned this Mar 7, 2019

@monis0395

This comment has been minimized.

Copy link

commented Mar 7, 2019

Thank for the quick reply, sorry could respond earlier

yes I tried but it didn't work, just the error message got shorter 😛
from net/http: HTTP/1.x transport connection broken: unsupported transfer encoding "gzip"
to unsupported transfer encoding "gzip"

it's great to hear that it would be fix so soon 🙌🙌🙌

@monis0395

This comment has been minimized.

Copy link

commented Mar 8, 2019

Hi @odeke-em

I tried debugging it more and found that at line 86 the values are

protoLine: HTTP/1.1 500 Internal Server Error
err: nil

The endpoint which I am hitting is a http2 and content-type is text/event-stream is it because of that?

I dont if this helps but when I use fasthttp for that endpoint it works but I wanna avoid using fasthttp

Thanks

@gopherbot

This comment has been minimized.

Copy link

commented Mar 10, 2019

Change https://golang.org/cl/166517 mentions this issue: net/http: support gzip as a Transfer-Encoding

@monis0395

This comment has been minimized.

Copy link

commented Mar 25, 2019

Hi @odeke-em

I tried your code from https://golang.org/cl/166517, I copied tranfer.go to my local and checked my endpoint

Now I am getting 200 OK status 👍 but the contents are empty when I read them from response

any clue?

Thanks

@monis0395

This comment has been minimized.

Copy link

commented Jul 3, 2019

Hi @odeke-em

Yesterday I spent more time on this issue. I found the issue why the work around wasn't working from the suggested in #29162 (comment).

Instead of this

	intro := fmt.Sprintf("%s %s HTTP/1.1\r\nHost: %s\r\n\r\n", req.Method, s, u.Host)
	fmt.Fprint(bw, intro)
	bw.Flush()

	// And now we've got the case of an unsupported transfer
	// encoding time for a raw fetch and header rewrite.
        tp := textproto.NewReader(br)

       protoLine, err := tp.ReadLine()

I changed it to this

        intro := fmt.Sprintf("%s /%s HTTP/1.1\r\nHost:%s\r\n\r\n", req.Method, s, u.Host)
	fmt.Fprint(conn, intro)

	br := bufio.NewReader(conn)
	// And now we've got the case of an unsupported transfer
	// encoding time for a raw fetch and header rewrite.
        tp := textproto.NewReader(br)

       protoLine, err := tp.ReadLine()

Here is the full gist https://gist.github.com/monis0395/0202b2129e943989e83a828553077cde

But it still did not work when I am hitting to my server. I get protoLine = HTTP/1.1 500. I think some problem with the way I am connect to the server. I am completely new with textproto, can you please suggest where can i read more about it?

Also I tried the change u made in transfer.go it is working now. This time I added each of your change manually, last time I had download the file transfer.go. I don't know why it didn't work last time 😕

Good news is that It works now 😁

I want to use your change in my production code, so is it possible to use it without changing in the go/src/net/http/tranfer.go? Any bright ideas? Because I want to avoid changing the golang source code.

Thanks!

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.