net/http: ServeFile 2X slower in Go 1.21 #61530
Description
What version of Go are you using (go version)?
$ go version go version go1.21rc3 linux/amd64
Does this issue reproduce with the latest release?
Yes. Slower in go1.21.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GOARCH='amd64' GOBIN='/home/user/gocode/bin' GOCACHE='/home/user/.cache/go-build' GOENV='/home/user/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/home/user/gocode/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/home/user/gocode' GOPRIVATE='' GOPROXY='https://proxy.golang.org,direct' GOROOT='/home/user/gocode/src/github.com/golang/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/home/user/gocode/src/github.com/golang/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.21rc3' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3856449379=/tmp/go-build -gno-record-gcc-switches'
What did you do?
While testing Uber’s go monorepo against go1.21rc3, I noticed a large performance regression in logic that depends on http.ServeFile. git bisect points to this commit (#56480) as the cause.
While we should expect to performance benefits from the use of io.Copy due to the ability to upgrade to io.WriterTo or io.ReaderFrom, in the common use case of http.ServeFile, the ReadFrom method that is now used instead of io.CopyN is noticeably slower for the common case of serving relatively small files. Our profiling reveals larger GC costs as a result of the new copying implementation.
Because the user doesn’t really have means of tinkering with the underlying writer or reader used by http.ServeFile they can’t reasonably work around this performance issue.
A reproducible benchmark test is attached below, we see roughly a 2x performance regression on http.ServeFile.
package main
import (
"bufio"
"io"
"log"
"net/http"
"net/http/httptest"
"testing"
)
func BenchmarkFileServe(b *testing.B) {
b.StopTimer()
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
b.StartTimer()
http.ServeFile(w, r, "/tmp/example") // the contents of example are attached in the issue (336 lines json)
b.StopTimer()
}))
defer ts.Close()
for i := 0; i < b.N; i++ {
res, err := http.Get(ts.URL)
if err != nil {
log.Fatal(err)
}
// note: we see a noticeable performance regression with both allRead
// and scanRead.
err = allRead(res.Body)
// err = scanRead(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
}
}
func scanRead(r io.Reader) error {
scanner := bufio.NewScanner(r)
for scanner.Scan() { /* scan */ }
return scanner.Err()
}
func allRead(r io.Reader) error {
_, err := io.ReadAll(r)
return err
}
Contents of /tmp/example we used in the benchmark 336 line example JSON
[
{
"latlng": {
"lat": 0,
"lng": 0
},
"region": {
"name": "gopherland",
"id": 1
},
"golang": {
"namespace": "gopherland",
"ids": [
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers"
]
}
},
{
"latlng": {
"lat": 0,
"lng": 0
},
"region": {
"name": "gopherland",
"id": 1
},
"golang": {
"namespace": "gopherland",
"ids": [
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers",
"i-like-gophers"
]
}
}
]
in go1.20.5
$ go test -bench=.
goos: linux
goarch: amd64
pkg: exp/fileserver
cpu: AMD EPYC 7B13
BenchmarkFileServe-96 17074 69007 ns/op
PASS
ok exp/fileserver 10.056s
in go1.21rc2 (go1.21rc3 test times out for me after 660s so I can't get a clean result).
$ go test -bench=.
oos: linux
goarch: amd64
pkg: exp/fileserver
cpu: AMD EPYC 7B13
BenchmarkFileServe-96 10000 126070 ns/op
PASS
ok exp/fileserver 416.290s
benchstat
goos: linux
goarch: amd64
pkg: exp/fileserver
cpu: AMD EPYC 7B13
│ /tmp/rs0 │ /tmp/rs1 │
│ sec/op │ sec/op vs base │
FileServe-96 69.01µ ± ∞ ¹ 126.07µ ± ∞ ¹ ~ (p=1.000 n=1) ²