Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/net/http2: Client POST request fails against golang http/2 server if client request.Body isn't completely consumed. #15425

Closed
nemosupremo opened this issue Apr 24, 2016 · 33 comments
Assignees
Milestone

Comments

@nemosupremo
Copy link

@nemosupremo nemosupremo commented Apr 24, 2016

1. What version of Go are you using (go version)? go version go1.6 darwin/amd64

2. What operating system and processor architecture are you using (go env)? OS X El Capitan 10.11.1

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/nimi/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.6/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.6/libexec/pkg/tool/darwin_amd64"
GO15VENDOREXPERIMENT="1"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
CXX="clang++"
CGO_ENABLED="1"

3. What did you do?

Initiate a POST request with a large body. Ignore the request (and/or also Close the request body). The golang http/2 server will not send the response body back to the client. In cases where the request Body is closed, Chrome and other browsers get an error in the connection (net::ERR_SPDY_PROTOCOL_ERROR)

See https://gist.github.com/nemothekid/c20921999d1cce1b3c982d035d6b09b5

This program will server a javascript application that will try to issue a POST request with some data to the server. When run, the POST request will fail.

4. What did you expect to see?

POST request should succeed, the browser should be able to retrieve the server's message body.

5. What did you see instead?

The POST request fails, and the body is never sent by the server. The works fine in HTTP/1.1 (Tested with GODEBUG=http2server=0). I discovered this issue while trying to debug caddyserver/caddy#782, where an upstream proxy server was rejecting the body. Also this issue would occur on servers that decide to refuse a payload above a specific size and need to respond with a sensible error message.

@uhthomas

This comment has been minimized.

Copy link

@uhthomas uhthomas commented Apr 24, 2016

To expand on this, I made a repo containing code which is able to reproduce this issue over at https://github.com/6f7262/go-http2-bug including an equivalent nodejs sever.

I'd also like to point out that during testing, some clients which were using HTTP/1.1 still had similar effects, except that consequent requests were still fine and the error returned from chrome's inspector was net::ERR_CONNECTION_CLOSED rather than net::ERR_SPDY_PROTOCOL_ERROR. Further, nodejs also exhibited similar behaviors when having even larger bodies, but instead of chrome reporting an error, the request would hang indefinitely. CURL has worked completely fine throughout all of this, which is strange to say the least.

@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Apr 25, 2016

Hmm, I've tried digging into this with go version

go version devel +758431f Mon Apr 25 02:13:58 2016 +0000 darwin/amd64

, because about 2 months ago, I encountered this same issue on production servers and worked around it by the same work around io.Copy(ioutil.Discard, res.Body). However, I cannot reproduce your issue. Maybe something am missing, maybe tip fixed it? Does it run with code refresh from the latest/tip?

I've updated your original code @nemothekid spinning a client and a server. The code is at https://github.com/odeke-em/bugs/tree/master/golang/15425 or here below inlined

  • client.go
package main

import (
    "crypto/tls"
    "io"
    "io/ioutil"
    "log"
    "net/http"
    "os"
    "strings"

    "golang.org/x/net/http2"
)

func main() {
    serverURL := "https://localhost:1333"
    if envServURL := os.Getenv("SERVER_URL"); envServURL != "" {
        serverURL = envServURL
    }

    certs, err := tls.LoadX509KeyPair("key.crt", "key.key")
    if err != nil {
        log.Fatal(err)
    }
    tr := &http.Transport{
        TLSClientConfig: &tls.Config{
            Certificates:       []tls.Certificate{certs},
            InsecureSkipVerify: true,
        },
    }
    http2.ConfigureTransport(tr)

    client := &http.Client{
        Transport: tr,
    }

    // Make a GigaByte of data
    raw := strings.Repeat("A", 1024*1024*1024)
    log.Printf("len(data)=%v\n", len(raw))
    sr := strings.NewReader(raw)
    req, err := http.NewRequest("POST", serverURL, sr)
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("client.Do next\n")
    res, err := client.Do(req)
    log.Printf("client.Do done\n")
    if err != nil {
        log.Fatal(err)
    }
    n, err := io.Copy(ioutil.Discard, res.Body)
    log.Printf("done Reading the server response's body n=(%v) err=%v\nres.headers: %v\n", n, err, res.Header)
    _ = res.Body.Close()
}
  • server.go
package main

import (
    "io"
    "io/ioutil"
    "log"
    "net/http"
    "strings"
    "time"
)

func handler(w http.ResponseWriter, r *http.Request) {
    if r.Method == "POST" {
        log.Printf("Received post request whose Headers are %v", r.Header)
        if false {
            io.Copy(ioutil.Discard, r.Body)
        }
        err := r.Body.Close()
        log.Printf("req.Body.Close() here, err=%v\n", err)
        if false {
            time.Sleep(700 * time.Millisecond)
        }
        //http.Error(w, error, code)
        w.WriteHeader(413)

        b, err := io.Copy(w, strings.NewReader(strings.Repeat("XXXXXX", 4192)))
        log.Printf("Wrote %d bytes. Error: %v", b, err)
        return
    }
    w.Header().Set("Content-Type", "text/html")
    w.Write([]byte(`
    <body>
    <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.0.0-beta1/jquery.min.js"></script>
    <script>
      var reqs = 1;
      $('body').text('Sending ' + reqs + ' requests...');
      function send(i) {
  var el = $('<div class="req">Waiting...</div>').attr('id', i).appendTo('body');
  var data = new FormData();
  data.append('file', new File([new Blob([new Array(5 * 1024 * 1024).join('0')], {type: 'image/jpeg'})], 'test.jpg'));
  var done = false;
  var req = new XMLHttpRequest();
  req.upload.onprogress = function(e) {
    if (!e.lengthComputable || done) return;
    el.text((e.loaded / e.total * 100) + '%');
  }
  req.onreadystatechange = function(e) {
    if (done || req.readyState !== 3) return;
    done = true;
    el.text('Done. Got ' + req.status);
  }
  req.open('POST', '/', true);
  req.send(data);
      }
      for (var i = 0; i < reqs; i++)
  send(i);
    </script>
    </body>
  `))
}

func main() {
    http.HandleFunc("/", handler)
    if err := http.ListenAndServeTLS(":1333", "key.crt", "key.key", nil); err != nil {
        log.Fatal(err)
    }
}

However, running it on my local network and debugging http2 gives

  • server
$ GODEBUG=http2debug=1 go run server.go 
2016/04/24 22:50:30 http2: server connection from [::1]:53481 on 0xc820088200
2016/04/24 22:50:30 http2: server: client [::1]:53481 said hello
2016/04/24 22:50:30 http2: server read frame SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2016/04/24 22:50:30 http2: server processing setting [ENABLE_PUSH = 0]
2016/04/24 22:50:30 http2: server processing setting [INITIAL_WINDOW_SIZE = 4194304]
2016/04/24 22:50:30 http2: server processing setting [MAX_HEADER_LIST_SIZE = 10485760]
2016/04/24 22:50:30 http2: server read frame WINDOW_UPDATE len=4 (conn) incr=1073741824
2016/04/24 22:50:30 http2: server read frame SETTINGS flags=ACK len=0
2016/04/24 22:50:30 http2: server read frame HEADERS flags=END_HEADERS stream=1 len=44
2016/04/24 22:50:30 Received post request whose Headers are map[Content-Length:[1073741824] Accept-Encoding:[gzip] User-Agent:[Go-http-client/2.0]]
2016/04/24 22:50:30 req.Body.Close() here, err=<nil>
2016/04/24 22:50:30 http2: server encoding header ":status" = "413"
2016/04/24 22:50:30 http2: server encoding header "content-type" = "text/plain; charset=utf-8"
2016/04/24 22:50:30 http2: server encoding header "date" = "Mon, 25 Apr 2016 05:50:30 GMT"
2016/04/24 22:50:30 Wrote 25152 bytes. Error: <nil>
2016/04/24 22:50:30 http2: server read frame DATA stream=1 len=65535 data="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" (65279 bytes omitted)
2016/04/24 22:50:30 http2: server read frame WINDOW_UPDATE stream=1 len=4 incr=12288
2016/04/24 22:50:30 http2: server read frame WINDOW_UPDATE stream=1 len=4 incr=12864

and the client when run

$ go run client.go 
2016/04/24 22:56:28 len(data)=1073741824
2016/04/24 22:56:28 client.Do next
2016/04/24 22:56:28 client.Do done
2016/04/24 22:56:28 done Reading the server response's body n=(25152) err=<nil>
res.headers: map[Content-Type:[text/plain; charset=utf-8] Date:[Mon, 25 Apr 2016 05:56:28 GMT]]

only the first 65535 of the client's Body are consumed in the HTTP2 server, res.Body.Close is successful and it writes out the response to the client asap.

Maybe am missing something?

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented Apr 25, 2016

In Go 1.6, I tested your client and server code and it works, but doesn't exhibit the same behavior the Chrome & Firefox seem to have.

AirNemo-2:Desktop nimi$ GODEBUG=http2debug=1 go run server.go 
2016/04/24 23:25:06 http2: server connection from 127.0.0.1:58511 on 0xc820016100
2016/04/24 23:25:06 http2: server: client 127.0.0.1:58511 said hello
2016/04/24 23:25:06 http2: server read frame SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2016/04/24 23:25:06 http2: server processing setting [ENABLE_PUSH = 0]
2016/04/24 23:25:06 http2: server processing setting [INITIAL_WINDOW_SIZE = 4194304]
2016/04/24 23:25:06 http2: server processing setting [MAX_HEADER_LIST_SIZE = 10485760]
2016/04/24 23:25:06 http2: server read frame WINDOW_UPDATE len=4 (conn) incr=1073741824
2016/04/24 23:25:06 http2: server read frame SETTINGS flags=ACK len=0
2016/04/24 23:25:06 http2: server read frame HEADERS flags=END_HEADERS stream=1 len=56
2016/04/24 23:25:06 http2: server decoded header field ":authority" = "localhost.channelmeter.com:1333"
2016/04/24 23:25:06 http2: server decoded header field ":method" = "POST"
2016/04/24 23:25:06 http2: server decoded header field ":path" = "/"
2016/04/24 23:25:06 http2: server decoded header field ":scheme" = "https"
2016/04/24 23:25:06 http2: server decoded header field "content-length" = "1073741824"
2016/04/24 23:25:06 http2: server decoded header field "accept-encoding" = "gzip"
2016/04/24 23:25:06 http2: server decoded header field "user-agent" = "Go-http-client/2.0"
2016/04/24 23:25:06 http2: server read frame DATA stream=1 len=65535 data="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" (65279 bytes omitted)
2016/04/24 23:25:06 Received HTTP/2.0 post request whose Headers are map[Content-Length:[1073741824] Accept-Encoding:[gzip] User-Agent:[Go-http-client/2.0]]
2016/04/24 23:25:06 req.Body.Close() here, err=<nil>
2016/04/24 23:25:06 http2: server encoding header ":status" = "413"
2016/04/24 23:25:06 http2: server encoding header "content-type" = "text/plain; charset=utf-8"
2016/04/24 23:25:06 http2: server encoding header "date" = "Mon, 25 Apr 2016 06:25:06 GMT"
2016/04/24 23:25:06 Wrote 25152 bytes. Error: <nil>
2016/04/24 23:25:06 http2: server read frame WINDOW_UPDATE stream=1 len=4 incr=24576
AirNemo-2:Desktop nimi$ go run ./client.go 
2016/04/24 23:25:06 len(data)=1073741824
2016/04/24 23:25:06 client.Do next
2016/04/24 23:25:06 client.Do done
2016/04/24 23:25:06 done Reading the server response's body n=(25152) err=<nil>
res.headers: map[Content-Type:[text/plain; charset=utf-8] Date:[Mon, 25 Apr 2016 06:25:06 GMT]]

However, Chrome 49.0.2623.112 still exhibits the erroneous behavior when Chrome makes the post request (via JS). Here is the output of server.go, from a Chrome client (go 1.6):

AirNemo-2:Desktop nimi$ GODEBUG=http2debug=1 go run server.go 
2016/04/24 23:28:49 http2: server connection from 127.0.0.1:58657 on 0xc820016100
2016/04/24 23:28:49 http2: server: client 127.0.0.1:58657 said hello
2016/04/24 23:28:49 http2: server read frame SETTINGS len=12, settings: MAX_CONCURRENT_STREAMS=1000, INITIAL_WINDOW_SIZE=6291456
2016/04/24 23:28:49 http2: server processing setting [MAX_CONCURRENT_STREAMS = 1000]
2016/04/24 23:28:49 http2: server processing setting [INITIAL_WINDOW_SIZE = 6291456]
2016/04/24 23:28:49 http2: server read frame WINDOW_UPDATE len=4 (conn) incr=15663105
2016/04/24 23:28:49 http2: server read frame SETTINGS flags=ACK len=0
2016/04/24 23:28:49 http2: server read frame HEADERS flags=END_STREAM|END_HEADERS|PRIORITY stream=1 len=537
2016/04/24 23:28:49 http2: server decoded header field ":method" = "GET"
2016/04/24 23:28:49 http2: server decoded header field ":authority" = "localhost.channelmeter.com:1333"
2016/04/24 23:28:49 http2: server decoded header field ":scheme" = "https"
2016/04/24 23:28:49 http2: server decoded header field ":path" = "/"
2016/04/24 23:28:49 http2: server decoded header field "cache-control" = "max-age=0"
2016/04/24 23:28:49 http2: server decoded header field "accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
2016/04/24 23:28:49 http2: server decoded header field "upgrade-insecure-requests" = "1"
2016/04/24 23:28:49 http2: server decoded header field "user-agent" = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36"
2016/04/24 23:28:49 http2: server decoded header field "accept-encoding" = "gzip, deflate, sdch"
2016/04/24 23:28:49 http2: server decoded header field "accept-language" = "en-US,en;q=0.8"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "rxx=9buf2utu0r8.znr7b5&v=1"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "_ga=GA1.2.1575566159.1439230335"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__cfduid=dc0421da8b1019e0b4589a6fa93642d341454224609"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utma=95374414.1575566159.1439230335.1461360906.1461366552.994"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utmc=95374414"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utmz=95374414.1460603659.954.25.utmcsr=channelmeter.atlassian.net|utmccn=(referral)|utmcmd=referral|utmcct=/browse/CMCOM-458"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "_hp2_id.3440866521=6666510306088045.3504969456630825.0651466615070784"
2016/04/24 23:28:49 http2: server encoding header ":status" = "200"
2016/04/24 23:28:49 http2: server encoding header "content-type" = "text/html"
2016/04/24 23:28:49 http2: server encoding header "content-length" = "952"
2016/04/24 23:28:49 http2: server encoding header "date" = "Mon, 25 Apr 2016 06:28:49 GMT"
2016/04/24 23:28:49 http2: server read frame HEADERS flags=END_HEADERS|PRIORITY stream=3 len=150
2016/04/24 23:28:49 http2: server decoded header field ":method" = "POST"
2016/04/24 23:28:49 http2: server decoded header field ":authority" = "localhost.channelmeter.com:1333"
2016/04/24 23:28:49 http2: server decoded header field ":scheme" = "https"
2016/04/24 23:28:49 http2: server decoded header field ":path" = "/"
2016/04/24 23:28:49 http2: server decoded header field "content-length" = "5243075"
2016/04/24 23:28:49 http2: server decoded header field "origin" = "https://localhost.channelmeter.com:1333"
2016/04/24 23:28:49 http2: server decoded header field "user-agent" = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36"
2016/04/24 23:28:49 http2: server decoded header field "content-type" = "multipart/form-data; boundary=----WebKitFormBoundarympkz8giGks318oJr"
2016/04/24 23:28:49 http2: server decoded header field "accept" = "*/*"
2016/04/24 23:28:49 http2: server decoded header field "referer" = "https://localhost.channelmeter.com:1333/"
2016/04/24 23:28:49 http2: server decoded header field "accept-encoding" = "gzip, deflate"
2016/04/24 23:28:49 http2: server decoded header field "accept-language" = "en-US,en;q=0.8"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "rxx=9buf2utu0r8.znr7b5&v=1"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "_ga=GA1.2.1575566159.1439230335"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__cfduid=dc0421da8b1019e0b4589a6fa93642d341454224609"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utma=95374414.1575566159.1439230335.1461360906.1461366552.994"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utmc=95374414"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utmz=95374414.1460603659.954.25.utmcsr=channelmeter.atlassian.net|utmccn=(referral)|utmcmd=referral|utmcct=/browse/CMCOM-458"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "_hp2_id.3440866521=6666510306088045.3504969456630825.0651466615070784"
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="------WebKitFormBoundarympkz8giGks318oJr\r\nContent-Disposition: form-data; name=\"file\"; filename=\"test.jpg\"\r\nContent-Type: application/octet-stream\r\n\r\n0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 Received HTTP/2.0 post request whose Headers are map[Referer:[https://localhost.channelmeter.com:1333/] Accept-Encoding:[gzip, deflate] Cookie:[rxx=9buf2utu0r8.znr7b5&v=1; _ga=GA1.2.1575566159.1439230335; __cfduid=dc0421da8b1019e0b4589a6fa93642d341454224609; __utma=95374414.1575566159.1439230335.1461360906.1461366552.994; __utmc=95374414; __utmz=95374414.1460603659.954.25.utmcsr=channelmeter.atlassian.net|utmccn=(referral)|utmcmd=referral|utmcct=/browse/CMCOM-458; _hp2_id.3440866521=6666510306088045.3504969456630825.0651466615070784] Origin:[https://localhost.channelmeter.com:1333] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36] Accept:[*/*] Content-Length:[5243075] Content-Type:[multipart/form-data; boundary=----WebKitFormBoundarympkz8giGks318oJr] Accept-Language:[en-US,en;q=0.8]]
2016/04/24 23:28:49 req.Body.Close() here, err=<nil>
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 Wrote 4096 bytes. Error: http2: stream closed
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2852 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2596 bytes omitted)
2016/04/24 23:28:49 http2: server read frame DATA stream=3 len=2791 data="0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" (2535 bytes omitted)
2016/04/24 23:28:49 http2: server read frame HEADERS flags=END_STREAM|END_HEADERS|PRIORITY stream=5 len=36
2016/04/24 23:28:49 http2: server decoded header field ":method" = "GET"
2016/04/24 23:28:49 http2: server decoded header field ":authority" = "localhost.channelmeter.com:1333"
2016/04/24 23:28:49 http2: server decoded header field ":scheme" = "https"
2016/04/24 23:28:49 http2: server decoded header field ":path" = "/favicon.ico"
2016/04/24 23:28:49 http2: server decoded header field "user-agent" = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36"
2016/04/24 23:28:49 http2: server decoded header field "accept" = "*/*"
2016/04/24 23:28:49 http2: server decoded header field "referer" = "https://localhost.channelmeter.com:1333/"
2016/04/24 23:28:49 http2: server decoded header field "accept-encoding" = "gzip, deflate, sdch"
2016/04/24 23:28:49 http2: server decoded header field "accept-language" = "en-US,en;q=0.8"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "rxx=9buf2utu0r8.znr7b5&v=1"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "_ga=GA1.2.1575566159.1439230335"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__cfduid=dc0421da8b1019e0b4589a6fa93642d341454224609"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utma=95374414.1575566159.1439230335.1461360906.1461366552.994"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utmc=95374414"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "__utmz=95374414.1460603659.954.25.utmcsr=channelmeter.atlassian.net|utmccn=(referral)|utmcmd=referral|utmcct=/browse/CMCOM-458"
2016/04/24 23:28:49 http2: server decoded header field "cookie" = "_hp2_id.3440866521=6666510306088045.3504969456630825.0651466615070784"
2016/04/24 23:28:49 http2: server encoding header ":status" = "200"
2016/04/24 23:28:49 http2: server encoding header "content-type" = "text/html"
2016/04/24 23:28:49 http2: server encoding header "content-length" = "952"
2016/04/24 23:28:49 http2: server encoding header "date" = "Mon, 25 Apr 2016 06:28:49 GMT"


Does the request succeed with Chrome, when the script is compiled with go tip? I can try building tip and testing the code again.

@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Apr 25, 2016

@nemothekid that's because in your Javascript you are trying to use an incompatible feature

if (!e.lengthComputable || done) return;

ProgressEvent.lengthComputable of which ProgressEvents are not supported in Chrome
See https://developer.mozilla.org/en-US/docs/Web/API/ProgressEvent/lengthComputable.
screen shot 2016-04-25 at 12 06 25 am
so the code will never terminate because e.lengthComputable will always return false in Chrome.

Also a quick examination of the console in Chrome gives
screen shot 2016-04-25 at 12 05 14 am

It however works in Firefox which supports ProgressEvents
screen shot 2016-04-25 at 12 05 51 am

This isn't a bug in Go, but a case of browser incompatibility.

@uhthomas

This comment has been minimized.

Copy link

@uhthomas uhthomas commented Apr 25, 2016

I don't think this has anything to do with a progress event in JavaScript... And I'm certain that MDN is wrong. Especially as it says it's supported in the Chrome documentation

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented Apr 25, 2016

e.lengthComputable is always false for me in Chrome 49.0.2623.112 - I'm not sure its supported.

req.upload.onprogress = function(e) {
    console.log("" + (!e.lengthComputable || done) + " ("+ !e.lengthComputable + " || " + done + ")")
    if (!e.lengthComputable || done) return;
    el.text((e.loaded / e.total * 100) + '%');
  }

Secondly, I just tried it for myself, and Im' not sure that part of the JS is relevant. If you remove the entire onprogress handler (including the unsupported code), the request still fails.

screen shot 2016-04-25 at 12 28 38 am

It should fail with status code 413, entity too large. Chrome doesn't get any data back. ERR_SPDY_PROTOCOL_ERROR isn't a valid error for a Javascript logic error. It could be a bug with Chrome.

@uhthomas

This comment has been minimized.

Copy link

@uhthomas uhthomas commented Apr 25, 2016

^this. Even if it wasn't supported then it wouldn't return an error like that.

@mholt

This comment has been minimized.

Copy link

@mholt mholt commented Apr 25, 2016

If it was a problem with lengthComputable then we wouldn't have seen results like this in Chrome:

Sending 3 requests...
1.2510978767231062%
0.05439555985752635%
0.05439555985752635%

As all the numbers would be 0%, and it still doesn't explain the connection hanging in both Chrome and Firefox.

But there definitely is something fishy going on with the browsers, too, I agree. Just strange that more than Chrome has had the issue. Interestingly, in my testing, using HTTPS vs. HTTP affected the results.

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented Apr 26, 2016

By studying the output of the Chrome's data transmission on the server side, I was able to narrow down the issue to the PRIORITY info that Chrome sends as an optimization. You can read more about it here.

I modified the go's http2 client code to mimic this behavior here - nemosupremo/net@aeb45a1

I modified the values based on what Chrome is doing which I uncovered with the article I referenced above.

I have modified the client code like so (to use my http2 code): https://gist.github.com/nemothekid/4edf9c5ded1e17d335e1d959718e93e8

With the new code, I'm seeing the golang client.go crash much like Chrome:

AirNemo:Desktop nimi$ go run ./client.go 
2016/04/25 18:01:02 len(data)=1073741824
2016/04/25 18:01:02 client.Do next
2016/04/25 18:01:02 client.Do done
2016/04/25 18:01:02 Post https://localhost.channelmeter.com:1333: stream error: stream ID 1; STREAM_CLOSED
exit status 1

Server.go

AirNemo:Desktop nimi$ GODEBUG=http2debug=1 go run bb.go 
2016/04/25 18:01:02 http2: server connection from 127.0.0.1:53464 on 0xc820016100
2016/04/25 18:01:02 http2: server: client 127.0.0.1:53464 said hello
2016/04/25 18:01:02 http2: server read frame SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2016/04/25 18:01:02 http2: server processing setting [ENABLE_PUSH = 0]
2016/04/25 18:01:02 http2: server processing setting [INITIAL_WINDOW_SIZE = 4194304]
2016/04/25 18:01:02 http2: server processing setting [MAX_HEADER_LIST_SIZE = 10485760]
2016/04/25 18:01:02 http2: server read frame WINDOW_UPDATE len=4 (conn) incr=1073741824
2016/04/25 18:01:02 http2: server read frame SETTINGS flags=ACK len=0
2016/04/25 18:01:02 http2: server read frame DATA stream=1 len=65535 data="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" (65279 bytes omitted)

I haven't investigated yet, but this is a path to look into - how is golang dealing with prioritized streams?

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented Apr 26, 2016

Furthermore, it looks like the golang server loses the HEADERS frame when StreamDep is set to 0. When its set to 1, it works as normal. FWIW, 0 is a valid stream dependency.

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented Apr 26, 2016

Ignore my last 2 posts, I just ended up discovering #15444. When I fix the WriteHeaderFrame function to account for StreamDep=0, the code works.


It looks more like Chrome is sending the DATA frames before it sends the END_HEADER frame.

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented Apr 26, 2016

I think in this case, my original assessment at caddyserver/caddy#782 (comment) may have been right.

I'm pretty sure its data race, as I'm having a really hard to time writing a script that will set the conditions just right (even though it always occurs in Chrome). Using https://github.com/golang/net/tree/master/http2 @ b797637b7aeeed133049c7281bfa31dcc9ca42d6

  1. Chrome makes a POST request
  2. net/http2 calls our Handler
  3. Our Handler closes req.Body
  4. The write at server.go#L1307 fails with write on closed buffer.
  5. Instead of finishing the request, net/http2 sends a CLOSE_STREAM

Kinda sort have a diagram:

 Chrome         net/http2           Handler
    |              |                    |
 POST req -------->|                    |
    |              |                    |
    | <-----------ACK                   |
    |              | ---------------> Handler(w http.ResponseWriter, r *http.Request)
  DATA -------> Body.write(data)        |
  DATA -------> Body.write(data)        |
    |              |                 Body.Close()
  DATA -------> Body.write(data)        |
    |       err = writeOnClosedBuff     |
    | <--------  CLOSE_STREAM           |

It looks like the issue here is the Handler shouldn't be allowed to Close the request body until net/http2 is sure it has finished writing to it.

@uhthomas

This comment has been minimized.

Copy link

@uhthomas uhthomas commented Apr 26, 2016

Great work! The results are really interesting but, where do we go from here?

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented Apr 26, 2016

I was able to craft a script that mimics Chrome's behavior - https://gist.github.com/nemothekid/753ca2db95faaa6fd0395fe857b36c21

EDIT: The script is racy :/ sometimes it reliably fails, sometimes it succeeds :/ (or it might just succeed the first time, and fail every other time).

Now its just fixing the http2 code in Go.

@bradfitz bradfitz added this to the Go1.7 milestone Apr 26, 2016
@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented Apr 26, 2016

So I was able to conceptualize a quick hack/fix for this issue that fixes the issue in my script here nemosupremo/net@ad5c348 that just pretty much ignores Close() and allows net/http2 to continue writing into the buffer until the WINDOW is full (at which point the client should stop sending DATA).

While this fixes the script, it doesn't fix the request in Chrome.

ChromeWiresharkout_key.zip

This is a wireshark output (decode port 1333 as SSL and use the session keys provided, see https://blog.cloudflare.com/tools-for-debugging-testing-and-using-http-2/). AFAICT nothing in the communication violates the spec, so I'm guessing Chrome's internal state machine is fucked?

I've opened an issue with Chromium here - https://bugs.chromium.org/p/chromium/issues/detail?id=606990

@gaillard

This comment has been minimized.

Copy link

@gaillard gaillard commented May 19, 2016

Is there any other workaround than consuming the request body in the handler using ioutil.Discard? In error cases and a large body this has an undesirable effect, doable until the fix though if there is no workaround. Thanks!

@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented May 19, 2016

@nemothekid, thanks for all the analysis so far, but I'm a bit overwhelmed with all of it (and with other Go 1.7 bugs in general).

What is the summary at this point? If there's something to fix on Go's side, I have approximately 24 hours to do so before I disappear for a month, and at that point fixing it for Go 1.7 will almost certainly be too late.

Is your comment at #15425 (comment) still the latest summary? It's not clear what's that's doing, or why. And is there a server component that's necessary to reproduce it?

Can you make a stand-alone repro using httptest.NewServer?

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented May 20, 2016

Sorry for leaving this bug report in such a sorry state. I looked into submitting a patch to net/http2, and got a bit overwhelmed/intimidated ¯_(ツ)_/¯.

  1. There is an issue with the go http2 server that prevents #15425 (comment) from being served correctly.
  2. There is also an issue with Chrome (reported before me at 603182 in their tracker) that prevents the Chrome from treating RST_STREAM correctly.

Point 2. can only be fixed in Chrome (the issue was brought up by the nginx team). Here is a script using httptest.NewServer that reproduces point 1. https://gist.github.com/nemothekid/35a3edcf332a20225c725696c0e40e70 Note that the request should succeed, but currently fails.

A possible work around I developed as to never close the underlying pipe in requestBody (nemosupremo/net@ad5c348) but I'm not sure what else this breaks/leaks or if this is an "optimal" solution.

@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented May 20, 2016

What is that repro (https://gist.github.com/nemothekid/35a3edcf332a20225c725696c0e40e70) trying to show?

There's a bunch of weird stuff in it:

  • it has its own LimitedReader instead of using io.LimitReader. Is that a relevant detail?
  • the client generates a 1GB of memory to only send 2KB of it. Is that a relevant detail?

Here's a modified version of your repro, if it still shows what you were trying to show: ?

package main

import (
    "crypto/tls"
    "io"
    "io/ioutil"
    "log"
    "net/http"
    "net/http/httptest"

    "golang.org/x/net/http2"
)

type neverEnding byte

func (b neverEnding) Read(p []byte) (int, error) {
    for i := range p {
        p[i] = byte(b)
    }
    return len(p), nil
}

func main() {

    ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        r.Body.Close()
        log.Printf("Http: %s", r.Proto)
        io.Copy(w, io.LimitReader(neverEnding('A'), 1024*1024*1024))
    }))
    http2.ConfigureServer(ts.Config, &http2.Server{})
    ts.TLS = ts.Config.TLSConfig
    ts.StartTLS()
    defer ts.Close()

    tr := &http.Transport{TLSClientConfig: &tls.Config{
        InsecureSkipVerify: true,
    }}
    http2.ConfigureTransport(tr)
    client := &http.Client{
        Transport: tr,
    }

    req, err := http.NewRequest("POST", ts.URL, struct{ io.Reader }{io.LimitReader(neverEnding('A'), 2048)})
    if err != nil {
        log.Fatal(err)
    }

    for i := 0; i < 2; i++ { // racy?
        log.Printf("client.Do next\n")
        res, err := client.Do(req)
        log.Printf("client.Do done\n")
        if err != nil {
            log.Fatalf("i=%d, Do = %v", i, err)
        }
        n, err := io.Copy(ioutil.Discard, res.Body)
        log.Printf("done Reading the server response's body n=(%v) err=%v\nres.headers: %v\n", n, err, res.Header)
        _ = res.Body.Close()
    }
}

Is this what you were seeing:

2016/05/20 16:41:01 client.Do next
2016/05/20 16:41:01 Http: HTTP/2.0
2016/05/20 16:41:01 client.Do done
2016/05/20 16:41:01 i=0, Do = Post https://127.0.0.1:33500: stream error: stream ID 1; STREAM_CLOSED
exit status 1

?

@bradfitz bradfitz self-assigned this May 20, 2016
@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented May 20, 2016

I have a fix,

diff --git a/http2/server.go b/http2/server.go
index 57c8276..4e07a20 100644
--- a/http2/server.go
+++ b/http2/server.go
@@ -1833,7 +1833,7 @@ type requestBody struct {

 func (b *requestBody) Close() error {
        if b.pipe != nil {
-               b.pipe.CloseWithError(errClosedBody)
+               b.pipe.BreakWithError(errClosedBody)
        }
        b.closed = true
        return nil

... but I notice that while it fixes it, the Transport is observed to be leaking goroutines, blocked on flow control tokens from the server, even after the stream is dead:

goroutine 14449 [semacquire]:
sync.runtime_notifyListWait(0xc42001a750, 0x3)
        /home/bradfitz/go/src/runtime/sema.go:267 +0x113
sync.(*Cond).Wait(0xc42001a740)
        /home/bradfitz/go/src/sync/cond.go:57 +0x72
golang.org/x/net/http2.(*clientStream).awaitFlowControl(0xc42008a640, 0x1, 0x0, 0x0, 0x0)
        /home/bradfitz/src/golang.org/x/net/http2/transport.go:953 +0x112
golang.org/x/net/http2.(*clientStream).writeRequestBody(0xc42008a640, 0x7b2aa0, 0xc420106480, 0x7fe9ad959000, 0xc42000db20, 0x0, 0x0)
        /home/bradfitz/src/golang.org/x/net/http2/transport.go:868 +0x22b
golang.org/x/net/http2.(*Transport).getBodyWriterState.func1()
        /home/bradfitz/src/golang.org/x/net/http2/transport.go:1759 +0x97
created by golang.org/x/net/http2.bodyWriterState.scheduleBodyWrite
        /home/bradfitz/src/golang.org/x/net/http2/transport.go:1806 +0x7b

So I need to fix that first, and also add a test.

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented May 20, 2016

  1. Brad, that is the proper response, the request fails because of "stream error: stream ID 1; STREAM_CLOSED". If it helps, you can think of the servers handler as function that sees the request is way to large and returns 413 Payload Too Large
  2. In your rewritten example, the response from the server doesn't need to be infinite (I believe only large enough that it isn't entirely served by a single Write call.
  3. The LimitReader I crafted for the post request just breaks up the Read call into 2kb chunks. The client only ends up sending 2kb only because the http2 server will send RST_STREAM when it tries to deliver the response (when r.Body.Close() is called). If I didn't do this, net/http2 would read enough data to fill it's window. While its window is filled, when RST_STREAM (when r.Body.Close() is called) is sent, we don't get the racy condition of trying to fill the window some more (and eventually trying to read on a closed stream).

To expand on point 3, I think the issue is (and referring back to my diagram) is that while the window can still be filled, net/http2 will try to write to that window. Simultaneously we can call Close() on that stream, and the next time we try to write to that window, we get an error.

@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented May 20, 2016

In my code, nothing is infinite. I wrapped all the infinite readers in LimitReaders. But this way the program doesn't consume gigabytes of RAM and crash due to my dev VM's small size.

I'm a little confused by your analysis still. What would be most helpful if I'm going to be able to fix this in the next few hours is if you can provide a version of your repro which doesn't allocate 2GB of memory and still demonstrates the problem.

Make it loop more than 2 times, and have it crash if it gets an unexpected result, making it clear in the failure message or code what it's expecting.

That is, I want to see the high level use case fail. I can diagnose the HTTP/2-level problems.

But it's hard for me to work the other way, with speculation on the HTTP/2-level problem and trying to guess the use case.

If you can comment on whether my patch above makes this better or worse (or completely fixes your case), that would be great too.

Thanks.

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented May 20, 2016

Sorry for the terrible state of this bug report. It's one of those things that while trying to find the bug I modified a bunch of different values - but didn't work backwards to really isolate what caused the issue.

This script uses much less memory, but reliably (on my system) fails. (Smaller values can succeed sometimes). https://gist.github.com/nemothekid/fad888179a061a0e3cd80d191a407b9f

The reason I have it loop is because I think the issue is racy. I'm going to continue to see if I can further isolate the issue.

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented May 20, 2016

Even the values I provided are a bit racy - I run the program here twice and get two different results (one where it succeeds twice, but then fails the first time).

NemoPro:Desktop nimi$ go run ./httptest2.go 
2016/05/20 11:07:14 len(data)=256
2016/05/20 11:07:14 client.Do next
2016/05/20 11:07:14 Http: HTTP/2.0
2016/05/20 11:07:14 client.Do done
2016/05/20 11:07:14 done Reading the server response's body n=(4) err=<nil>
res.headers: map[Content-Type:[text/plain; charset=utf-8] Content-Length:[4] Date:[Fri, 20 May 2016 18:07:14 GMT]]
2016/05/20 11:07:14 client.Do next
2016/05/20 11:07:14 Http: HTTP/2.0
2016/05/20 11:07:14 client.Do done
2016/05/20 11:07:14 done Reading the server response's body n=(4) err=<nil>
res.headers: map[Content-Type:[text/plain; charset=utf-8] Content-Length:[4] Date:[Fri, 20 May 2016 18:07:14 GMT]]

NemoPro:Desktop nimi$ go run ./httptest2.go 
2016/05/20 11:07:16 len(data)=256
2016/05/20 11:07:16 client.Do next
2016/05/20 11:07:16 Http: HTTP/2.0
2016/05/20 11:07:16 client.Do done
2016/05/20 11:07:16 Post https://127.0.0.1:61011: stream error: stream ID 1; STREAM_CLOSED
exit status 1
@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented May 20, 2016

This script uses much less memory, but reliably (on my system) fails. (Smaller values can succeed sometimes). https://gist.github.com/nemothekid/fad888179a061a0e3cd80d191a407b9f

I think you pasted the wrong file. That has tiny values and seems to work all the time.

Please provide a script which reliably fails. Make the loop iteration count high enough to make it eventually fail (not 2, more like 1000 or whatever). And make the script fail with log.Fatal when the behavior isn't as expected. That is, check the result of io.Copy, and don't just log it.

@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented May 20, 2016

Your patch looks fixes the issue (it no longer occurs on the small & large memory case).

I've also modified the test to better reflect what the real use case is - https://gist.github.com/nemothekid/be66f5b4521fbe48cf775a18783ea7d7 - this is when the server sees the request is too large, and refuses to read it (again, it passes it with your patch, but not with CloseWithError).

This uses values large enough to fail reliably for me, but not use a huge amount of memory. Note that where I have the log.Fatalf is the test, I don't really care about the response, only that the request shouldn't fail with an internal error.

Also, with that same script I am using:

  • go version go1.6.2 darwin/amd64
  • golang.org/x/net/http2 at the current master.
@nemosupremo

This comment has been minimized.

Copy link
Author

@nemosupremo nemosupremo commented May 20, 2016

I thought it may have been the fact that I was on a mac as well, and it seems I can't reproduce the issue on single core machines. I tried on a couple of our linux boxes:

NemoPro:Desktop nimi$ GOOS=linux go build ./httptest2.go 

NemoPro:Desktop nimi$ scp httptest2 redis1: && scp httptest2 dev1: 
httptest2                                                                                              100% 8587KB   8.4MB/s   00:01    
httptest2                                                                                              100% 8587KB   4.2MB/s   00:02    

NemoPro:Desktop nimi$ ssh redis1
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.13.0-44-generic x86_64)
root@redis1:~# nproc
16
root@redis1:~# ./httptest2 
2016/05/20 18:35:29 len(data)=65537
2016/05/20 18:35:29 client.Do next
2016/05/20 18:35:29 Http: HTTP/2.0
2016/05/20 18:35:29 client.Do done
2016/05/20 18:35:29 Expected Status Code 413, Got status code: 0, error: Post https://127.0.0.1:60051: stream error: stream ID 1; STREAM_CLOSED
root@redis1:~# exit
logout
Connection to redis1 closed.

NemoPro:Desktop nimi$ ssh dev1
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.13.0-44-generic x86_64)
nimi@dev1:~$ nproc
1
nimi@dev1:~$ ./httptest2
2016/05/20 18:35:36 len(data)=65537
2016/05/20 18:35:36 client.Do next
2016/05/20 18:35:36 Http: HTTP/2.0
2016/05/20 18:35:36 client.Do done
2016/05/20 18:35:36 done Reading the server response's body n=(4) err=<nil>
res.headers: map[Date:[Fri, 20 May 2016 18:35:36 GMT] Content-Type:[text/plain; charset=utf-8] Content-Length:[4]]
2016/05/20 18:35:36 client.Do next
2016/05/20 18:35:36 Http: HTTP/2.0
2016/05/20 18:35:36 client.Do done
2016/05/20 18:35:36 done Reading the server response's body n=(4) err=<nil>
res.headers: map[Content-Type:[text/plain; charset=utf-8] Content-Length:[4] Date:[Fri, 20 May 2016 18:35:36 GMT]]
@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented May 20, 2016

I've sent a fix for review: https://golang.org/cl/23287

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented May 20, 2016

CL https://golang.org/cl/23287 mentions this issue.

gopherbot pushed a commit to golang/net that referenced this issue May 20, 2016
Also, fix a Transport goroutine leak if the transport is still trying
to write its body after the stream has completed.

Updates golang/go#15425 (fixes after bundle into std)

Change-Id: Id14d9360d012f53a963ec1999ef88fc592978b80
Reviewed-on: https://go-review.googlesource.com/23287
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented May 20, 2016

CL https://golang.org/cl/23301 mentions this issue.

@gaillard

This comment has been minimized.

Copy link

@gaillard gaillard commented Jun 15, 2016

Am I correct that to get this fix we must use 1.7?

If I am doing something like

hs := &http.Server{Handler: thehandler}
if err := http2.ConfigureServer(hs, &http2.Server{}); err != nil {
    //handle
}
if err := hs.Serve(existingTLSListener); err != nil {
    //handle
}

and have vendored in the http2 package, I still won't get it right?

It would be useful to have in the docs a blurb about how using any of the http2 package directly and the bundled http2 in the http package works.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 15, 2016

@gaillard Please open a separate issue for the doc request.

Unfortunately I do not know the answer myself.

@gaillard

This comment has been minimized.

Copy link

@gaillard gaillard commented Jul 18, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.