net/http/httputil: ReverseProxy no longer uses io.ReaderFrom/WriterTo

[Zariel]

Before go 1.8 httputil.ReverseProxy would use io.CopyBuffer which would use io.ReaderFrom or io.WriterTo if possible to avoid allocation. In fixing #16659 it removed using io.CopyBuffer and implements its own without the optimisations of io.CopyBuffer to add an extra log.

Please answer these questions before submitting your issue. Thanks!

What did you do?

https://play.golang.org/p/4GjT-hwI3t

What did you expect to see?

WriteTo

What did you see instead?

Read

System details

go version go1.9 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/chris/dev/go"
GORACE=""
GOROOT="/usr/local/opt/go/libexec"
GOTOOLDIR="/usr/local/opt/go/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOROOT/bin/go version: go version go1.9 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.9
uname -v: Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.11.3
BuildVersion:	15D21
lldb --version: lldb-340.4.110.1
gdb --version: GNU gdb (GDB) 8.0

[Zariel]

I suggest either reverting to calling io.CopyBuffer and logging the error returned from that or adding the interface checks to copyResponse. The former makes more sense to me as would also log write errors which adds more symmetry. I have a patch available if its an acceptable change.

[tomwans]

The former makes more sense to me as would also log write errors which adds more symmetry.

Looking at the CL that introduced this, it seems that the intent was to log only read errors: https://go-review.googlesource.com/c/go/+/30692/5/src/net/http/httputil/reverseproxy.go#264. That is why io.CopyBuffer was copied in. Whatever solution we use here should probably retain that property.

[gopherbot]

Change https://golang.org/cl/65690 mentions this issue: net/http/httputil: respect WriterTo and ReadFrom by using io.CopyBuffer

[tombergan]

I'm not sure how to resolve this without breaking #16659 or making errors spammy, which @bradfitz objected to. The issue: when WriteTo/ReadFrom returns an error, it's impossible to know if the error came from reading from src or writing to dst. That detail is hidden inside the WriteTo/ReadFrom implementation. Options are:

1. Leave the code as-is (don't support WriteTo/ReadFrom).
2. Add WriteTo/ReadFrom and log all errors from those methods, possibly making logs spammy.
3. Add WriteTo/ReadFrom but don't log any errors from those methods, possibly breaking net/http/httputil: ReverseProxy does not log errors during CopyBuffer #16659.

My inclination is to go with option 1. I don't believe that WriteTo or ReadFrom can use sendfile because resp.Body is never a simple file descriptor (not even when the incoming response is HTTP/1 without chunked encoding). At best WriteTo/ReadFrom might provide slightly more optimal buffering. I would want to see evidence of improved performance from WriteTo/ReadFrom over Read/Write before actually moving forward.

/cc @fraenkel, @Zariel

[fraenkel]

@tombergan Its more or less how we ended up with the current implementation. Since the http package doesn't provide any optimized ReadFrom/WriteTo, I would be ok with option 3.

[tombergan]

Actually, net/http's ResponseWriter implements ReaderFrom, so I think option 3 would break #16659.

To clarify: the WaitinfForInfo state is waiting for some justification that WriteTo/ReadFrom provide an actual (measurable) performance gain.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[Zariel]

Now that #10948 has landed Go's ReverseProxy will not be able to use splice due to #16659

[jameshartig]

Now that we have splice being used it's unfortunate that it's not used in ReverseProxy. I vote for using io.CopyBuffer here so there isn't a need to consistantly keep copying functionality over and any future improvements in the io package will benefit the ReverseProxy.

We currently log over 16,000 "http: proxy error: context canceled" errors per hour but I haven't seen any of the "read error" logs. Now that we have ErrorHandler if a user wants to filter out specific spammy errors (like the one above) they can do so in their own ErrorHandler.

@fraenkel, what are your thoughts?

[fraenkel]

@fastest963 I understand that you have never seen the issue but the reason why the code was changed was because we did. It was far too common to have downstream clients disappear and have no knowledge in the proxy as to who was at fault.
We can't just switch to io.Copy or io.CopyBuffer because we now lose this detail.

Do you have any data that shows using splice provides value? Which is what @tombergan was asking for above.