net/http: nginx seems to require that entire request be read before writing response

[cloudaice]

go version go1.6.2 linux/amd64

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/data01/home/xiangchao01/code"
GORACE=""
GOROOT="/data01/home/xiangchao01/bin/go"
GOTOOLDIR="/data01/home/xiangchao01/bin/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Sorry i can't provide runnable program, but I think this is a common problem, I find this error in many projects。

Nginx  -> (My Go HTTP server)

Nginx use upstream, total QPS is about 3000，and about 1 second, I see an error log in my nginx error log, the log like this:

 2016/05/22 18:54:27 [error] 192493#0: *23511789339 readv() failed (104: Connection reset by peer) while reading upstream,.....

if I use go1.3.3 to build this project , there is no error log.

I catch the TCP pocket

[ianlancetaylor]

I doubt there is anything we can do if we don't have a way to recreate the problem. You will need to do some more debugging yourself to understand why and when the connection is being closed on the client side.

If this is a common problem, as you suggest, then I hope that somebody can provide a program that demonstrates it.

[cloudaice]

OK，I will try to provide a program.

[cloudaice]

I can provide program now.

The client code

package main

import (
        "bytes"
        "encoding/json"
        "fmt"
        "net/http"
)

func main() {
        for {
                req()
        }
}

func ret() string {
        s := "k"
        for i := 0; i < 12; i++ {
                s = s + s
        }
        return s
}

func req() {
        fmt.Println("start req")
        val := struct {
                Data string `json:"data"`
        }{Data: ret()}
        data, _ := json.Marshal(val)
        r, err := http.Post("http://127.0.0.1:9989", "application/json", bytes.NewBuffer(data))
        if err != nil {
                fmt.Println(err)
        } else {
                fmt.Println("Status Code: ", r.Status)
        }
        r.Body.Close()
}

The server code

package main

import (
        "compress/gzip"
        "encoding/json"
        "io"
        "io/ioutil"
        "net/http"
)

func main() {
        mux := http.NewServeMux()
        mux.HandleFunc("/", HelloHandler)
        server := &http.Server{Addr: ":8998", Handler: mux}
        server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
        reader, err := gzip.NewReader(r.Body) // I make this mistake Purposely.
        if err == nil {
                io.Copy(ioutil.Discard, reader)
        }
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusOK)
        val := struct {
                Message string `json:"message"`
        }{
                Message: "Hello World!",
        }
        body, _ := json.Marshal(val)
        w.Write(body)
}

The nginx config

worker_processes  1;

events {
    worker_connections  111024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;

    upstream myapp {
        server 127.0.0.1:8998;
    }

    server {
        listen       9989;
        server_name  localhost;

        location / {
            proxy_pass http://myapp;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

The nginx error log

2016/05/24 15:50:53 [error] 53509#0: *394941 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *394955 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395009 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395029 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395049 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395069 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395077 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395129 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395169 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395319 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"

when I build server code with go1.3.3 version, there is no "reset by peer" error in nginx error log,
but I build server code with go1.6.2 version, the "reset by peer" error appear in nginx error log.

In the server code, I use gzip reader to read body, I know this will return error, I just want to make this case.

nginx version is 1.9.15

@ianlancetaylor

[davecheney]

Does the error occur if you remove nginx from the scenario ?

On Tue, May 24, 2016 at 6:06 PM, 项超 notifications@github.com wrote:

I can provide program now.

The client code

package main

import (
"bytes"
"encoding/json"
"fmt"
"net/http"
)

func main() {
for {
req()
}
}

func ret() string {
s := "k"
for i := 0; i < 12; i++ {
s = s + s
}
return s
}

func req() {
fmt.Println("start req")
val := struct {
Data string json:"data"
}{Data: ret()}
data, _ := json.Marshal(val)
r, err := http.Post("http://127.0.0.1:9989", "application/json", bytes.NewBuffer(data))
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Status Code: ", r.Status)
}
r.Body.Close()
}

The server code

package main

import (
"compress/gzip"
"encoding/json"
"io"
"io/ioutil"
"net/http"
)

func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", HelloHandler)
server := &http.Server{Addr: ":8998", Handler: mux}
server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
reader, err := gzip.NewReader(r.Body) // I make this mistake Purposely.
if err == nil {
io.Copy(ioutil.Discard, reader)
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
val := struct {
Message string json:"message"
}{
Message: "Hello World!",
}
body, _ := json.Marshal(val)
w.Write(body)
}

The nginx config

worker_processes 1;

events {
worker_connections 111024;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;

upstream myapp {
    server 127.0.0.1:8998;
}

server {
    listen       9989;
    server_name  localhost;

    location / {
        proxy_pass http://myapp;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
}

}

The nginx error log

2016/05/24 15:50:53 [error] 53509#0: *394941 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *394955 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395009 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395029 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395049 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395069 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395077 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395129 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395169 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395319 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"

when I build server code with go1.3.3 version, there is no "reset by peer"
error in nginx error log,
but I build server code with go1.6.2 version, the "reset by peer" error
appear in nginx error log.

In the server code, I use gzip reader to read body, I know this will
return error, I just want to make this case.

@ianlancetaylor https://github.com/ianlancetaylor

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#15789 (comment)

[cloudaice]

If I remove nginx, there is no error .

[cloudaice]

I update the service code like this

package main

import (
        "compress/gzip"
        "encoding/json"
        "io"
        "io/ioutil"
        "net/http"
)

func main() {
        mux := http.NewServeMux()
        mux.HandleFunc("/", HelloHandler)
        server := &http.Server{Addr: ":8998", Handler: mux}
        server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
        reader, err := gzip.NewReader(r.Body)
        if err == nil {
                io.Copy(ioutil.Discard, reader)
        } else {
                io.Copy(ioutil.Discard, r.Body) // this is diffrent
        }
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusOK)
        val := struct {
                Message string `json:"message"`
        }{
                Message: "Hello World!",
        }
        body, _ := json.Marshal(val)
        w.Write(body)
}

There is no errors when I use the follow code

        if err == nil {
                io.Copy(ioutil.Discard, reader)
        } else {
                io.Copy(ioutil.Discard, r.Body) // this is diffrent
        }

It seems that I should read all of request data in body before write response.

[ianlancetaylor]

So it sounds like the problem is that when using nginx, the entire request must be read before the response is written. I'll leave this for analysis during the 1.8 release.

[bradfitz]

Yeah, this is a common situation with HTTP/1 implementations. I wouldn't even say "problem" because it's extremely ill-defined on whether you can read & write at the same time.

https://golang.org/pkg/net/http/#ResponseWriter does already say:

        // Depending on the HTTP protocol version and the client, calling
        // Write or WriteHeader may prevent future reads on the
        // Request.Body. For HTTP/1.x requests, handlers should read any
        // needed request body data before writing the response. Once the
        // headers have been flushed (due to either an explicit Flusher.Flush
        // call or writing enough data to trigger a flush), the request body
        // may be unavailable. For HTTP/2 requests, the Go HTTP server permits
        // handlers to continue to read the request body while concurrently
        // writing the response. However, such behavior may not be supported
        // by all HTTP/2 clients. Handlers should read before writing if
        // possible to maximize compatibility.
        Write([]byte) (int, error)

Unless there's some code in the standard library that writes before reading, I'm not sure there's anything for us to do here. Closing, but feel free to reopen if you disagree and I'm not understanding what this bug is about.