Severe performance regression when using ServeHTTP

[tamird]

Extracting from discussion in #549. See https://github.com/cockroachdb/rpc-bench; relevant parts as of this writing:

name                 time/op
GRPCServe_1K-4         72.6µs ±10%
GRPCServe_64K-4         785µs ± 8%
GRPCServeHTTP_1K-4      178µs ± 2%
GRPCServeHTTP_64K-4    1.32ms ± 2%

name                 speed
GRPCServe_1K-4       28.3MB/s ± 9%
GRPCServe_64K-4       167MB/s ± 7%
GRPCServeHTTP_1K-4   11.5MB/s ± 2%
GRPCServeHTTP_64K-4  99.4MB/s ± 2%

name                 alloc/op
GRPCServe_1K-4         15.6kB ± 0%
GRPCServe_64K-4         711kB ± 0%
GRPCServeHTTP_1K-4     88.5kB ± 0%
GRPCServeHTTP_64K-4     801kB ± 0%

name                 allocs/op
GRPCServe_1K-4           96.0 ± 0%
GRPCServe_64K-4           162 ± 0%
GRPCServeHTTP_1K-4        166 ± 0%
GRPCServeHTTP_64K-4       671 ± 0%

[tamird]

@bradfitz any progress on this?

[bradfitz]

No, I've been busy with Go 1.7 stuff for the May 1st freeze. Hopefully I can spend a bit of time on this soon, but then I'm out for a month starting in a couple weeks.

[mwitkow]

@bradfitz is there some way in which we could help? :)

[bradfitz]

is there some way in which we could help? :)

Sure. Look at the http2 code and find ways to improve it, while not hurting readability or maintainability.

[mwitkow]

Any pointers as to how to start? Is there any particular test you pprof to find bottlenecks? :)

[bradfitz]

Honestly I forget. That code was mostly written for correctness and not performance. There's probably a lot of low-hanging fruit in there but there might not even be enough benchmarks yet to start using pprof & friends. You'd probably have to write at least a few new ones.

[tamird]

@iamqizhao this is a serious issue - can we include it for GA?

[iamqizhao]

@tamird Sorry. We are approaching GA very closely and so far this is treated as an experimental effort and we do not expect the real usage for this feature at the current stage. This is in Brad's plate and I am going to talk to him after GA to see how to make progress on this.

[tamird]

This is still a major inconvenience for us. I've updated the performance numbers at https://github.com/cockroachdb/rpc-bench; the relevant ones are:

name                         time/op
GRPCServe_1K-24                40.5µs ± 1%
GRPCServe_64K-24                365µs ± 1%
GRPCServe_Stream_1K-24         22.7µs ± 1%
GRPCServe_Stream_64k-24         298µs ± 1%
GRPCServeHTTP_1K-24             137µs ± 1%
GRPCServeHTTP_64K-24            585µs ± 1%
GRPCServeHTTP_Stream_1K-24     38.7µs ± 1%
GRPCServeHTTP_Stream_64k-24     455µs ± 1%

name                         speed
GRPCServe_1K-24              50.5MB/s ± 1%
GRPCServe_64K-24              359MB/s ± 1%
GRPCServe_Stream_1K-24       90.3MB/s ± 1%
GRPCServe_Stream_64k-24       439MB/s ± 1%
GRPCServeHTTP_1K-24          14.9MB/s ± 1%
GRPCServeHTTP_64K-24          224MB/s ± 1%
GRPCServeHTTP_Stream_1K-24   52.9MB/s ± 1%
GRPCServeHTTP_Stream_64k-24   288MB/s ± 1%

name                         alloc/op
GRPCServe_1K-24                16.3kB ± 0%
GRPCServe_64K-24                711kB ± 0%
GRPCServe_Stream_1K-24         11.6kB ± 0%
GRPCServe_Stream_64k-24         707kB ± 0%
GRPCServeHTTP_1K-24            34.2kB ± 0%
GRPCServeHTTP_64K-24            763kB ± 0%
GRPCServeHTTP_Stream_1K-24     12.4kB ± 0%
GRPCServeHTTP_Stream_64k-24     739kB ± 0%

name                         allocs/op
GRPCServe_1K-24                  97.0 ± 0%
GRPCServe_64K-24                  147 ± 0%
GRPCServe_Stream_1K-24           22.0 ± 0%
GRPCServe_Stream_64k-24          72.0 ± 0%
GRPCServeHTTP_1K-24               169 ± 0%
GRPCServeHTTP_64K-24              275 ± 0%
GRPCServeHTTP_Stream_1K-24       29.0 ± 0%
GRPCServeHTTP_Stream_64k-24       138 ± 0%

So ServeHTTP's throughput is 50%-70% lower than Serve's, and its latency is 60%-350% higher.

[bradfitz]

@tamird, the grpc-go team is undergoing a changing of the guard at the moment so this isn't much of a priority.

And I don't officially work on grpc-go. It's not surprising that ServeHTTP is slower considering it's shoehorned into the existing ServerTransport interface somewhat awkwardly.

I started working on a new design for all of grpc-go in https://github.com/bradfitz/grpc-go but it's not ready yet for wider discussion or testing. It was mostly a proof of concept. But it's something I intend to get back to. @philips et al at CoreOS are also interested.

[Iulian7]

Is there any update on this?