Description
What version of Go are you using (go version
)?
$ go version go version go1.15.7 linux/amd64
Does this issue reproduce with the latest release?
YES
What operating system and processor architecture are you using (go env
)?
go env
Output
$ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/root/.cache/go-build" GOENV="/root/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/root/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/root/go" GOPRIVATE="" GOPROXY="https://goproxy.cn,direct" GOROOT="/usr/local/go" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GCCGO="gccgo" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build783936539=/tmp/go-build -gno-record-gcc-switches"
What did you do?
We are running simple proxy for retrieving blob files. Just in case, we set WriteTimeout
for proxy server. The logic is described by the following code.
package main
import (
"log"
"net/http"
"time"
)
// The content of filePath is in JSON format.
//
//
// {"architecture":"amd64","config":{"Hostname":"","Domainname":"","User":"0","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"],"Cmd":null,"Image":"sha256:cbdac00ef1904d59ccd11add20108257111b8f65a854170f2d31caa78c4bb1b2","Volumes":null,"WorkingDir":"/","Entrypoint":["/go-runner"],"OnBuild":null,"Labels":{"description":"go based runner for distroless scenarios","maintainers":"Kubernetes Authors"}},"container_config":{"Hostname":"","Domainname":"","User":"0","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"],"Cmd":["/bin/sh","-c","#(nop) COPY file:7fed31705f1c51a2c3f437d928dc68009c048406d85a556147a4a7aea5b15f7c in /usr/local/bin/kube-apiserver "],"Image":"sha256:cbdac00ef1904d59ccd11add20108257111b8f65a854170f2d31caa78c4bb1b2","Volumes":null,"WorkingDir":"/","Entrypoint":["/go-runner"],"OnBuild":null,"Labels":{"description":"go based runner for distroless scenarios","maintainers":"Kubernetes Authors"}},"created":"2020-12-18T12:21:06.637698261Z","docker_version":"19.03.8","history":[{"created":"1970-01-01T00:00:00Z","author":"Bazel","created_by":"bazel build ..."},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"LABEL maintainers=Kubernetes Authors","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"LABEL description=go based runner for distroless scenarios","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"WORKDIR /","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"COPY /workspace/go-runner . # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2020-11-24T01:39:04.46965334Z","created_by":"ENTRYPOINT [\"/go-runner\"]","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2020-12-18T12:21:06.637698261Z","created_by":"/bin/sh -c #(nop) COPY file:7fed31705f1c51a2c3f437d928dc68009c048406d85a556147a4a7aea5b15f7c in /usr/local/bin/kube-apiserver "}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:e7ee84ae4d1363ccf59b14bf34a79c245705dfd55429918b63c754d84c85d904","sha256:597f1090d8e9bd4f1847ea4b72a3c3ea1f0997011120816c9dae2fe858077b32","sha256:007430fdb20c04acd56f66f3032b747f8e5ac47edb6855a37ae9f37d7be15b81"]}}#
var filePath = "/tmp/75c7f711208082c548b935ab31e681ea30acccdce6b7abeecabae5bbfd326627"
type handler struct{}
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
time.Sleep(2 * time.Second) // handle some logic
http.ServeFile(w, r, filePath)
}
func main() {
s := &http.Server{
Addr: "127.0.0.1:9999",
Handler: &handler{},
WriteTimeout: 1 * time.Second,
}
log.Fatal(s.ListenAndServe())
}
If the handler takes time and the WriteTimeout
happens, the http.ServeFile
still can sends the data without HTTP Status line and header. We use the following code as client to reproduce the issue.
package main
import (
"io"
"io/ioutil"
"net/http"
)
func main() {
resp, err := http.Get("http://127.0.0.1:9999")
if err != nil {
panic(err)
}
defer resp.Body.Close()
io.Copy(ioutil.Discard, resp.Body)
}
The result is like:
➜ /tmp go run client.go
panic: Get "http://127.0.0.1:9999": net/http: HTTP/1.x transport connection broken: malformed HTTP status code "based"
goroutine 1 [running]:
main.main()
/tmp/client.go:12 +0x125
exit status 2
Go HTTP Server will call SetWriteDeadline()
in readRequest. When we call http.ServeContent/ServeFile
to write response, the go runtime will use (*response).ReadFrom to make it fast.
But we found that the chunkWriter.flush
doesn't return error so that the ReadFrom
doesn't know if the flush works.
// https://github.com/golang/go/blob/go1.15.7/src/net/http/server.go#L581
// ReadFrom is here to optimize copying from an *os.File regular file
// to a *net.TCPConn with sendfile.
func (w *response) ReadFrom(src io.Reader) (n int64, err error) {
...
...
w.w.Flush() // get rid of any previous writes
w.cw.flush() // make sure Header is written; flush data to rwc
...
}
In this case, both the status and the header are lost because netpoll checker return I/O timeout
. But the server still send data to client and the data is not valid http response, called malformed HTTP status
. The WriteTimeout
doesn't work well in ServeContent/ServeFile
mode and return confusing response. I think it should be aligned with normal mode, described by the following code.
// server.go
package main
import (
"log"
"net/http"
"time"
)
type handler struct{}
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
time.Sleep(2 * time.Second) // handle some logic
w.Write([]byte("oops"))
}
func main() {
s := &http.Server{
Addr: "127.0.0.1:9999",
Handler: &handler{},
WriteTimeout: 1 * time.Second,
}
log.Fatal(s.ListenAndServe())
}
// bash shell
➜ /tmp go run client.go
panic: Get "http://127.0.0.1:9999": EOF
goroutine 1 [running]:
main.main()
/tmp/client.go:12 +0x125
exit status 2
What did you expect to see?
I expected to see EOF
error
What did you see instead?
I saw that malformed HTTP status
.