net/http: nginx seems to require that entire request be read before writing response #15789

cloudaice · May 22, 2016

go version go1.6.2 linux/amd64

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/data01/home/xiangchao01/code"
GORACE=""
GOROOT="/data01/home/xiangchao01/bin/go"
GOTOOLDIR="/data01/home/xiangchao01/bin/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Sorry i can't provide runnable program, but I think this is a common problem, I find this error in many projects。

Nginx  -> (My Go HTTP server)

Nginx use upstream, total QPS is about 3000，and about 1 second, I see an error log in my nginx error log, the log like this:

 2016/05/22 18:54:27 [error] 192493#0: *23511789339 readv() failed (104: Connection reset by peer) while reading upstream,.....

if I use go1.3.3 to build this project , there is no error log.

I catch the TCP pocket

ianlancetaylor · May 22, 2016

I doubt there is anything we can do if we don't have a way to recreate the problem. You will need to do some more debugging yourself to understand why and when the connection is being closed on the client side.

If this is a common problem, as you suggest, then I hope that somebody can provide a program that demonstrates it.

cloudaice · May 23, 2016

OK，I will try to provide a program.

cloudaice · May 24, 2016

I can provide program now.

The client code

package main

import (
        "bytes"
        "encoding/json"
        "fmt"
        "net/http"
)

func main() {
        for {
                req()
        }
}

func ret() string {
        s := "k"
        for i := 0; i < 12; i++ {
                s = s + s
        }
        return s
}

func req() {
        fmt.Println("start req")
        val := struct {
                Data string `json:"data"`
        }{Data: ret()}
        data, _ := json.Marshal(val)
        r, err := http.Post("http://127.0.0.1:9989", "application/json", bytes.NewBuffer(data))
        if err != nil {
                fmt.Println(err)
        } else {
                fmt.Println("Status Code: ", r.Status)
        }
        r.Body.Close()
}

The server code

package main

import (
        "compress/gzip"
        "encoding/json"
        "io"
        "io/ioutil"
        "net/http"
)

func main() {
        mux := http.NewServeMux()
        mux.HandleFunc("/", HelloHandler)
        server := &http.Server{Addr: ":8998", Handler: mux}
        server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
        reader, err := gzip.NewReader(r.Body) // I make this mistake Purposely.
        if err == nil {
                io.Copy(ioutil.Discard, reader)
        }
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusOK)
        val := struct {
                Message string `json:"message"`
        }{
                Message: "Hello World!",
        }
        body, _ := json.Marshal(val)
        w.Write(body)
}

The nginx config

worker_processes  1;

events {
    worker_connections  111024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;

    upstream myapp {
        server 127.0.0.1:8998;
    }

    server {
        listen       9989;
        server_name  localhost;

        location / {
            proxy_pass http://myapp;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

The nginx error log

2016/05/24 15:50:53 [error] 53509#0: *394941 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *394955 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395009 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395029 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395049 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395069 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395077 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395129 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395169 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395319 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"

when I build server code with go1.3.3 version, there is no "reset by peer" error in nginx error log,
but I build server code with go1.6.2 version, the "reset by peer" error appear in nginx error log.

In the server code, I use gzip reader to read body, I know this will return error, I just want to make this case.

nginx version is 1.9.15

@ianlancetaylor

davecheney · May 24, 2016

Does the error occur if you remove nginx from the scenario ?

On Tue, May 24, 2016 at 6:06 PM, 项超 notifications@github.com wrote:

I can provide program now.

The client code

package main

import (
"bytes"
"encoding/json"
"fmt"
"net/http"
)

func main() {
for {
req()
}
}

func ret() string {
s := "k"
for i := 0; i < 12; i++ {
s = s + s
}
return s
}

func req() {
fmt.Println("start req")
val := struct {
Data string json:"data"
}{Data: ret()}
data, _ := json.Marshal(val)
r, err := http.Post("http://127.0.0.1:9989", "application/json", bytes.NewBuffer(data))
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Status Code: ", r.Status)
}
r.Body.Close()
}

The server code

package main

import (
"compress/gzip"
"encoding/json"
"io"
"io/ioutil"
"net/http"
)

func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", HelloHandler)
server := &http.Server{Addr: ":8998", Handler: mux}
server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
reader, err := gzip.NewReader(r.Body) // I make this mistake Purposely.
if err == nil {
io.Copy(ioutil.Discard, reader)
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
val := struct {
Message string json:"message"
}{
Message: "Hello World!",
}
body, _ := json.Marshal(val)
w.Write(body)
}

The nginx config

worker_processes 1;

events {
worker_connections 111024;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;
upstream myapp {
    server 127.0.0.1:8998;
}

server {
    listen       9989;
    server_name  localhost;

    location / {
        proxy_pass http://myapp;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
}
}

The nginx error log

2016/05/24 15:50:53 [error] 53509#0: *394941 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *394955 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395009 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395029 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395049 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395069 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395077 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395129 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395169 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395319 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"

when I build server code with go1.3.3 version, there is no "reset by peer"
error in nginx error log,
but I build server code with go1.6.2 version, the "reset by peer" error
appear in nginx error log.

In the server code, I use gzip reader to read body, I know this will
return error, I just want to make this case.

@ianlancetaylor https://github.com/ianlancetaylor

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#15789 (comment)

cloudaice · May 24, 2016

If I remove nginx, there is no error .

cloudaice · May 24, 2016

I update the service code like this

package main

import (
        "compress/gzip"
        "encoding/json"
        "io"
        "io/ioutil"
        "net/http"
)

func main() {
        mux := http.NewServeMux()
        mux.HandleFunc("/", HelloHandler)
        server := &http.Server{Addr: ":8998", Handler: mux}
        server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
        reader, err := gzip.NewReader(r.Body)
        if err == nil {
                io.Copy(ioutil.Discard, reader)
        } else {
                io.Copy(ioutil.Discard, r.Body) // this is diffrent
        }
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusOK)
        val := struct {
                Message string `json:"message"`
        }{
                Message: "Hello World!",
        }
        body, _ := json.Marshal(val)
        w.Write(body)
}

There is no errors when I use the follow code

        if err == nil {
                io.Copy(ioutil.Discard, reader)
        } else {
                io.Copy(ioutil.Discard, r.Body) // this is diffrent
        }

It seems that I should read all of request data in body before write response.

ianlancetaylor · May 24, 2016

So it sounds like the problem is that when using nginx, the entire request must be read before the response is written. I'll leave this for analysis during the 1.8 release.

bradfitz · Nov 1, 2016

Yeah, this is a common situation with HTTP/1 implementations. I wouldn't even say "problem" because it's extremely ill-defined on whether you can read & write at the same time.

https://golang.org/pkg/net/http/#ResponseWriter does already say:

        // Depending on the HTTP protocol version and the client, calling
        // Write or WriteHeader may prevent future reads on the
        // Request.Body. For HTTP/1.x requests, handlers should read any
        // needed request body data before writing the response. Once the
        // headers have been flushed (due to either an explicit Flusher.Flush
        // call or writing enough data to trigger a flush), the request body
        // may be unavailable. For HTTP/2 requests, the Go HTTP server permits
        // handlers to continue to read the request body while concurrently
        // writing the response. However, such behavior may not be supported
        // by all HTTP/2 clients. Handlers should read before writing if
        // possible to maximize compatibility.
        Write([]byte) (int, error)

Unless there's some code in the standard library that writes before reading, I'm not sure there's anything for us to do here. Closing, but feel free to reopen if you disagree and I'm not understanding what this bug is about.

Tevic · May 19, 2017

It seems golang should ensure that a FIN packet is sent before any RST packet.
https://trac.nginx.org/nginx/ticket/1037
And gunicorn seems fixed this issue.
benoitc/gunicorn#874

ianlancetaylor changed the title ~~failed (104: Connection reset by peer)~~ net: failed (104: Connection reset by peer) May 22, 2016

ianlancetaylor added this to the Unplanned milestone May 22, 2016

ianlancetaylor modified the milestones: Go1.8, Unplanned May 24, 2016

ianlancetaylor changed the title ~~net: failed (104: Connection reset by peer)~~ net/http: nginx seems to require that entire request be read before writing response May 24, 2016

mostlygeek referenced this issue Sep 9, 2016
Merged
Read all request body and close before sending error #119

quentinmit added the NeedsInvestigation label Oct 7, 2016

bradfitz closed this Nov 1, 2016

g-k referenced this issue Mar 14, 2017
Closed
fix reading nginx body before writing response #52

golang locked and limited conversation to collaborators May 19, 2018

gopherbot added the FrozenDueToAge label May 19, 2018

golang/go

net/http: nginx seems to require that entire request be read before writing response #15789

net/http: nginx seems to require that entire request be read before writing response #15789

cloudaice commented May 22, 2016 •

edited

ianlancetaylor changed the title failed (104: Connection reset by peer) net: failed (104: Connection reset by peer) May 22, 2016

This comment has been minimized.

ianlancetaylor commented May 22, 2016

ianlancetaylor added this to the Unplanned milestone May 22, 2016

This comment has been minimized.

cloudaice commented May 23, 2016

This comment has been minimized.

cloudaice commented May 24, 2016 •

edited

This comment has been minimized.

davecheney commented May 24, 2016

This comment has been minimized.

cloudaice commented May 24, 2016

This comment has been minimized.

cloudaice commented May 24, 2016

This comment has been minimized.

ianlancetaylor commented May 24, 2016

ianlancetaylor modified the milestones: Go1.8, Unplanned May 24, 2016

ianlancetaylor changed the title net: failed (104: Connection reset by peer) net/http: nginx seems to require that entire request be read before writing response May 24, 2016

mostlygeek referenced this issue Sep 9, 2016

Read all request body and close before sending error #119

quentinmit added the NeedsInvestigation label Oct 7, 2016

This comment has been minimized.

bradfitz commented Nov 1, 2016

bradfitz closed this Nov 1, 2016

g-k referenced this issue Mar 14, 2017

fix reading nginx body before writing response #52

This comment has been minimized.

Tevic commented May 19, 2017

golang locked and limited conversation to collaborators May 19, 2018

gopherbot added the FrozenDueToAge label May 19, 2018

golang/go

Join GitHub today

net/http: nginx seems to require that entire request be read before writing response #15789

Comments

cloudaice commented May 22, 2016 • edited

ianlancetaylor changed the title failed (104: Connection reset by peer) net: failed (104: Connection reset by peer) May 22, 2016

This comment has been minimized.

ianlancetaylor commented May 22, 2016

ianlancetaylor added this to the Unplanned milestone May 22, 2016

This comment has been minimized.

cloudaice commented May 23, 2016

This comment has been minimized.

cloudaice commented May 24, 2016 • edited

This comment has been minimized.

davecheney commented May 24, 2016

This comment has been minimized.

cloudaice commented May 24, 2016

This comment has been minimized.

cloudaice commented May 24, 2016

This comment has been minimized.

ianlancetaylor commented May 24, 2016

ianlancetaylor modified the milestones: Go1.8, Unplanned May 24, 2016

ianlancetaylor changed the title net: failed (104: Connection reset by peer) net/http: nginx seems to require that entire request be read before writing response May 24, 2016

mostlygeek referenced this issue Sep 9, 2016

Read all request body and close before sending error #119

quentinmit added the NeedsInvestigation label Oct 7, 2016

This comment has been minimized.

bradfitz commented Nov 1, 2016

bradfitz closed this Nov 1, 2016

g-k referenced this issue Mar 14, 2017

fix reading nginx body before writing response #52

This comment has been minimized.

Tevic commented May 19, 2017

golang locked and limited conversation to collaborators May 19, 2018

gopherbot added the FrozenDueToAge label May 19, 2018

cloudaice commented May 22, 2016 •

edited

cloudaice commented May 24, 2016 •

edited