Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: nginx seems to require that entire request be read before writing response #15789

Closed
cloudaice opened this issue May 22, 2016 · 9 comments

Comments

Projects
None yet
7 participants
@cloudaice
Copy link

commented May 22, 2016

go version go1.6.2 linux/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/data01/home/xiangchao01/code"
GORACE=""
GOROOT="/data01/home/xiangchao01/bin/go"
GOTOOLDIR="/data01/home/xiangchao01/bin/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Sorry i can't provide runnable program, but I think this is a common problem, I find this error in many projects。

Nginx  -> (My Go HTTP server)

Nginx use upstream, total QPS is about 3000,and about 1 second, I see an error log in my nginx error log, the log like this:

 2016/05/22 18:54:27 [error] 192493#0: *23511789339 readv() failed (104: Connection reset by peer) while reading upstream,.....

if I use go1.3.3 to build this project , there is no error log.

I catch the TCP pocket

dingtalk20160522190258

@ianlancetaylor ianlancetaylor changed the title failed (104: Connection reset by peer) net: failed (104: Connection reset by peer) May 22, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 22, 2016

I doubt there is anything we can do if we don't have a way to recreate the problem. You will need to do some more debugging yourself to understand why and when the connection is being closed on the client side.

If this is a common problem, as you suggest, then I hope that somebody can provide a program that demonstrates it.

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone May 22, 2016

@cloudaice

This comment has been minimized.

Copy link
Author

commented May 23, 2016

OK,I will try to provide a program.

@cloudaice

This comment has been minimized.

Copy link
Author

commented May 24, 2016

I can provide program now.

The client code

package main

import (
        "bytes"
        "encoding/json"
        "fmt"
        "net/http"
)

func main() {
        for {
                req()
        }
}

func ret() string {
        s := "k"
        for i := 0; i < 12; i++ {
                s = s + s
        }
        return s
}

func req() {
        fmt.Println("start req")
        val := struct {
                Data string `json:"data"`
        }{Data: ret()}
        data, _ := json.Marshal(val)
        r, err := http.Post("http://127.0.0.1:9989", "application/json", bytes.NewBuffer(data))
        if err != nil {
                fmt.Println(err)
        } else {
                fmt.Println("Status Code: ", r.Status)
        }
        r.Body.Close()
}

The server code

package main

import (
        "compress/gzip"
        "encoding/json"
        "io"
        "io/ioutil"
        "net/http"
)

func main() {
        mux := http.NewServeMux()
        mux.HandleFunc("/", HelloHandler)
        server := &http.Server{Addr: ":8998", Handler: mux}
        server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
        reader, err := gzip.NewReader(r.Body) // I make this mistake Purposely.
        if err == nil {
                io.Copy(ioutil.Discard, reader)
        }
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusOK)
        val := struct {
                Message string `json:"message"`
        }{
                Message: "Hello World!",
        }
        body, _ := json.Marshal(val)
        w.Write(body)
}

The nginx config

worker_processes  1;

events {
    worker_connections  111024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;

    upstream myapp {
        server 127.0.0.1:8998;
    }

    server {
        listen       9989;
        server_name  localhost;

        location / {
            proxy_pass http://myapp;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

The nginx error log

2016/05/24 15:50:53 [error] 53509#0: *394941 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *394955 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395009 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395029 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395049 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395069 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395077 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395129 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395169 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395319 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"

when I build server code with go1.3.3 version, there is no "reset by peer" error in nginx error log,
but I build server code with go1.6.2 version, the "reset by peer" error appear in nginx error log.

In the server code, I use gzip reader to read body, I know this will return error, I just want to make this case.

nginx version is 1.9.15

@ianlancetaylor

@davecheney

This comment has been minimized.

Copy link
Contributor

commented May 24, 2016

Does the error occur if you remove nginx from the scenario ?

On Tue, May 24, 2016 at 6:06 PM, 项超 notifications@github.com wrote:

I can provide program now.

The client code

package main

import (
"bytes"
"encoding/json"
"fmt"
"net/http"
)

func main() {
for {
req()
}
}

func ret() string {
s := "k"
for i := 0; i < 12; i++ {
s = s + s
}
return s
}

func req() {
fmt.Println("start req")
val := struct {
Data string json:"data"
}{Data: ret()}
data, _ := json.Marshal(val)
r, err := http.Post("http://127.0.0.1:9989", "application/json", bytes.NewBuffer(data))
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Status Code: ", r.Status)
}
r.Body.Close()
}

The server code

package main

import (
"compress/gzip"
"encoding/json"
"io"
"io/ioutil"
"net/http"
)

func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", HelloHandler)
server := &http.Server{Addr: ":8998", Handler: mux}
server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
reader, err := gzip.NewReader(r.Body) // I make this mistake Purposely.
if err == nil {
io.Copy(ioutil.Discard, reader)
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
val := struct {
Message string json:"message"
}{
Message: "Hello World!",
}
body, _ := json.Marshal(val)
w.Write(body)
}

The nginx config

worker_processes 1;

events {
worker_connections 111024;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;

upstream myapp {
    server 127.0.0.1:8998;
}

server {
    listen       9989;
    server_name  localhost;

    location / {
        proxy_pass http://myapp;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
}

}

The nginx error log

2016/05/24 15:50:53 [error] 53509#0: *394941 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *394955 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395009 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395029 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395049 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395069 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395077 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395129 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395169 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"
2016/05/24 15:50:53 [error] 53509#0: *395319 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8998/", host: "127.0.0.1:9989"

when I build server code with go1.3.3 version, there is no "reset by peer"
error in nginx error log,
but I build server code with go1.6.2 version, the "reset by peer" error
appear in nginx error log.

In the server code, I use gzip reader to read body, I know this will
return error, I just want to make this case.

@ianlancetaylor https://github.com/ianlancetaylor


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#15789 (comment)

@cloudaice

This comment has been minimized.

Copy link
Author

commented May 24, 2016

If I remove nginx, there is no error .

@cloudaice

This comment has been minimized.

Copy link
Author

commented May 24, 2016

I update the service code like this

package main

import (
        "compress/gzip"
        "encoding/json"
        "io"
        "io/ioutil"
        "net/http"
)

func main() {
        mux := http.NewServeMux()
        mux.HandleFunc("/", HelloHandler)
        server := &http.Server{Addr: ":8998", Handler: mux}
        server.ListenAndServe()
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
        reader, err := gzip.NewReader(r.Body)
        if err == nil {
                io.Copy(ioutil.Discard, reader)
        } else {
                io.Copy(ioutil.Discard, r.Body) // this is diffrent
        }
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusOK)
        val := struct {
                Message string `json:"message"`
        }{
                Message: "Hello World!",
        }
        body, _ := json.Marshal(val)
        w.Write(body)
}

There is no errors when I use the follow code

        if err == nil {
                io.Copy(ioutil.Discard, reader)
        } else {
                io.Copy(ioutil.Discard, r.Body) // this is diffrent
        }

It seems that I should read all of request data in body before write response.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 24, 2016

So it sounds like the problem is that when using nginx, the entire request must be read before the response is written. I'll leave this for analysis during the 1.8 release.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.8, Unplanned May 24, 2016

@ianlancetaylor ianlancetaylor changed the title net: failed (104: Connection reset by peer) net/http: nginx seems to require that entire request be read before writing response May 24, 2016

@bradfitz

This comment has been minimized.

Copy link
Member

commented Nov 1, 2016

Yeah, this is a common situation with HTTP/1 implementations. I wouldn't even say "problem" because it's extremely ill-defined on whether you can read & write at the same time.

https://golang.org/pkg/net/http/#ResponseWriter does already say:

        // Depending on the HTTP protocol version and the client, calling
        // Write or WriteHeader may prevent future reads on the
        // Request.Body. For HTTP/1.x requests, handlers should read any
        // needed request body data before writing the response. Once the
        // headers have been flushed (due to either an explicit Flusher.Flush
        // call or writing enough data to trigger a flush), the request body
        // may be unavailable. For HTTP/2 requests, the Go HTTP server permits
        // handlers to continue to read the request body while concurrently
        // writing the response. However, such behavior may not be supported
        // by all HTTP/2 clients. Handlers should read before writing if
        // possible to maximize compatibility.
        Write([]byte) (int, error)

Unless there's some code in the standard library that writes before reading, I'm not sure there's anything for us to do here. Closing, but feel free to reopen if you disagree and I'm not understanding what this bug is about.

@Tevic

This comment has been minimized.

Copy link
Contributor

commented May 19, 2017

It seems golang should ensure that a FIN packet is sent before any RST packet.
https://trac.nginx.org/nginx/ticket/1037
And gunicorn seems fixed this issue.
benoitc/gunicorn#874

@golang golang locked and limited conversation to collaborators May 19, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.