Experiments with RPC Servers
The code here is for experimenting with very basic gRPC/Thrift stuff, except that I wrote a minimal service from scratch instead of simply running the helloworld examples bundled in the distributions.
Included here are:
- a gRPC service proto,
proto/agent.proto, - a gRPC Go client test and benchmark
client.go,client_test.go - a gRPC Go server implementation
server/server.go - a gRPC C++ server implementation
server/server.cc - a gRPC Java server implementation
server/AgentIOServer.java(contributed by @vidubey) - a Thrift IDL for the same service
thrift/agent.thrift - a Thrift Go implementation of the same service
thrift/gen-go - a Thrift C++ implementation of the same service
thrift/gen-cpp
The following instructions assume that you have all the required gRPC components installed in the right place to be able to build new services. If you aren't able to build github.com/grpc/grpc-go/examples/helloworld and github.com/grpc/grpc/examples/cpp/helloworld from the grpc-go and grpc repos respectively, you'll most likely not be able to build the following either.
To build the Go stubs and skeletons, execute the following:
[agent]$ protoc -I proto --go_out=plugins=grpc:. proto/agent.proto
Then, build the Go server:
[agent]$ cd server
[server]$ go build server.go
Run the server:
[server]$ ./server -addr :9001 -nettype tcp
Then run the client test and benchmark:
[server]$ cd ..
[agent]$ go test -bench .
To build the C++ stubs, skeletons and server, execute the following:
[agent]$ cd server
[server]$ make
Then run the C++ server:
[server]$ ./servercpp
Server listening on 0.0.0.0:9001
And re-run the benchmark as above.
The Thrift example has all the auto-generated code and binaries (amd64/darwin). It can be run by itself as follows:
[agent]$ cd thrift/gen-go/agent/server
[server]$ go build
[server]$ ./server &
Starting the simple server... on localhost:9090
Then run the benchmark client as follows:
[server]$ cd ../client
[client]$ go test -bench .
The Thrift C++ server binary is at thrift/gen-cpp/servercpp
I've got the following benchmarking results on a Darwin 15.2.0, 2.5 GHz Intel Core i7 (4x2 cores w/ HT), 1600 MHz DDR3 RAM:
$ go test -bench . -benchtime 10s
PASS
BenchmarkGRPCClient-8 300000 58358 ns/op
ok agent 18.114s
$ go test -bench . -benchtime 10s
PASS
BenchmarkGRPCClient-8 500000 27313 ns/op
BenchmarkGRPCClientLarge-8 500000 31714 ns/op
ok agent 31.974s
$ go test -bench . -benchtime 10s
PASS
BenchmarkGRPCClient-8 500000 29197 ns/op
ok agent 15.005s
PASS
BenchmarkThriftClient-8 500000 23798 ns/op
BenchmarkThriftClientLarge-8 500000 26091 ns/op
ok agent/thrift/gen-go/agent/client 25.585s
PASS
BenchmarkThriftClient-8 1000000 14158 ns/op
BenchmarkThriftClientLarge-8 1000000 15388 ns/op
ok agent/thrift/gen-go/agent/client 29.942s
On the same machine, I got the following numbers with a single non-concurrent client:
$ go test -bench . -benchtime 10s -cpu 1
PASS
BenchmarkThriftClient 50000 317681 ns/op
ok agent/thrift/gen-go/agent/client 19.106s
$ go test -bench . -benchtime 10s -cpu 1
PASS
BenchmarkThriftClient 300000 41697 ns/op
ok agent/thrift/gen-go/agent/client 12.955s
$ go test -bench . -benchtime 10s -cpu 1
PASS
BenchmarkThriftClient 300000 46840 ns/op
ok agent/thrift/gen-go/agent/client 14.539s
$ go test -bench . -benchtime 10s -cpu 1
PASS
BenchmarkThriftClient 500000 38221 ns/op
ok agent/thrift/gen-go/agent/client 19.518s
$ go test -bench . -benchtime 10s -cpu 1
PASS
BenchmarkThriftClient 300000 42343 ns/op
ok agent/thrift/gen-go/agent/client 13.153s
$ go test -bench . -benchtime 10s -cpu 1
PASS
BenchmarkGRPCClient 200000 113525 ns/op
ok agent 23.940s
$ go test -bench . -benchtime 10s -cpu 1
PASS
BenchmarkGRPCClient 200000 105424 ns/op
ok agent 22.226s
$ go test -bench . -benchtime 10s -cpu 1
PASS
BenchmarkGRPCClient 100000 108150 ns/op
ok agent 12.098s
TL;DR: Thrift is way faster but gRPC is way easier
Between implementations of gRPC, grpc-go is faster than grpc-java
and 2x faster than grpc-c++ in the concurrent scenario. In the
sequential scenario, grpc-go again wins, albeit with an
insignificant margin.
Working with Thrift is difficult in many ways. It generates a ton of
code – 999 sloc of Go, compared to 117 of Go with grpc-go. It is
very difficult to figure out the boilerplate needed to get it going
(see thrift/gen-go/agent/client and
thrift/gen-go/agent/server). The client that it generates is not
concurrent, which means manually controlling concurrent access to
Thrift clients. That's an avoidable developer burden, IMHO. [Thrift
connections are not multiplexed]
(https://mail-archives.apache.org/mod_mbox/thrift-user/201208.mbox/%3CA0F963DCF29346458CDF2969683DF6CC70F90B3A@SC-MBX01-2.TheFacebook.com%3E)
either. This is unlike GRPC where I was able to instantiate multiple
clients sharing the same underlying connection, though it was ~ 10%
slower so you'd probably only use this to multiplex lots of idle
clients.
The C++ backend that Thrift built out-of-the-box is single threaded. I
did find references to a non-blocking server implementation using
framed transport, but couldn't get it set up easily (missing
libthriftnb).
In terms of performance, though, the reward for putting up with
terrible documentation and somewhat more difficult code is the
all-crushing performance – 2x faster than the next fastest, grpc-go
when used in the "Buffered Transport" mode. "Framed Transport" mode is
not as terribly fast but is still ~ 15% faster than grpc-go. The
*Large set of benchmarks use > 1kB payload, both for request and
response to see if the speed disparity reduces, but the results don't
indicate any significant difference.
Unfortunately, I could not dig up any documentation outlining the
limitations of TBufferedTransport to decide whether it makes sense
to compare Thrift with TBufferedTransport against gRPC's default
transport.
To be fair, gRPC is also using framed transport while also doing a lot more book-keeping for context, traces and latency profiling out of the box.
$ go version
go version go1.5.2 darwin/amd64
$ java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
$ thrift -version
Thrift version 0.9.3
$ protoc --version
libprotoc 3.0.0