go-micro client rpc-call do not provide the connection pool

[henrypfhu]

Just as the subject, go-micro rpc client should provide the connection pool for the target server port.

I change the sample code go-micro/example/client/codegen/codegen.go to test the performance of go-micro rpc.

The changed snippet of code as below:

func call(i int, wg *sync.WaitGroup){
    defer wg.Done()

    for j := 0; j < i; j++ {
        rsp, err := cl.Call(context.Background(), &example.Request{Name: "John"})
        if err != nil {
            fmt.Println("call err: ", err, rsp)
            return
        }
    }
    //fmt.Println("Call:", i, "rsp:", rsp.Msg)
}


func main() {
    cmd.Init()
    wg := new(sync.WaitGroup)

    var start, stop int
    start = time.Now().Nanosecond()
    fmt.Println("\n--- Call example ---\n")
    wg.Add(5)
    go call(10000, wg)
    go call(10000, wg)
    go call(10000, wg)
    go call(10000, wg)
    go call(10000, wg)
    wg.Wait()
    stop = time.Now().Nanosecond()
    println("Call example Time used: ", stop-start)
}

During the client running, the console will throw exception as below:

call err:  {"id":"go.micro.client","code":500,"detail":"Error sending request: dial tcp 10.8.0.10:50661: bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.","status":"Internal Server Error"} <nil>
call err:  {"id":"go.micro.client","code":500,"detail":"Error sending request: dial tcp 10.8.0.10:50661: bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.","status":"Internal Server Error"} <nil>
call err:  {"id":"go.micro.client","code":500,"detail":"Error sending request: dial tcp 10.8.0.10:50661: bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.","status":"Internal Server Error"} <nil>
call err:  {"id":"go.micro.client","code":500,"detail":"Error sending request: dial tcp 10.8.0.10:50661: bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.","status":"Internal Server Error"} <nil>
call err:  {"id":"go.micro.client","code":500,"detail":"Error sending request: dial tcp 10.8.0.10:50661: bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.","status":"Internal Server Error"} <nil>

I checked the go-micro rpc_client.go code, found that every remote method call will create the tcp/ip connection, and after revocation this connection will be destroied. That will cause bad performance issue in the high load system.

rpc_client.go code snippet:

func (r *rpcClient) call(ctx context.Context, address string, req Request, resp interface{}) error {
       ......
    c, err := r.opts.Transport.Dial(address, transport.WithTimeout(r.opts.DialTimeout))
    if err != nil {
        return errors.InternalServerError("go.micro.client", fmt.Sprintf("Error sending request: %v", err))
    }
    var once sync.Once
      ......

Hope go-micro client can provide connection pool, only that go-micro can be used in the real production system.

[asim]

This is a bit of a crude example of overloading a server to a point that the host itself cannot provide the resources required to serve the request. If I take the same example and apply it against a basic net/http server it will yield the same result.

Here's a gist https://gist.github.com/asim/11aa2e31ddece57b41bf. And the result:

call err:  Get http://localhost:9099: dial tcp [::1]:9099: socket: too many open files
2016/03/19 10:55:36 http: Accept error: accept tcp [::]:9099: accept: too many open files; retrying in 5ms
call err:  Get http://localhost:9099: EOF
2016/03/19 10:55:36 http: Accept error: accept tcp [::]:9099: accept: too many open files; retrying in 5ms
call err:  Get http://localhost:9099: EOF
2016/03/19 10:55:36 http: Accept error: accept tcp [::]:9099: accept: too many open files; retrying in 10ms
call err:  Get http://localhost:9099: EOF
2016/03/19 10:55:36 http: Accept error: accept tcp [::]:9099: accept: too many open files; retrying in 20ms
call err:  Get http://localhost:9099: EOF

I think its probably first important to determine an appropriate benchmarking test to gauge current performance, then target the bottlenecks. The above is not a test of a "real production system".

[henrypfhu]

@asim, thanks for your quick response. Just as you mentioned, even net/http client request without client connection pool will cause the issue.

My point is that go-micro rpc framework will be more robust and performance boosting to reuse the established long-live connection.

Http2 protocol can be used to keep long-live connection between go-micro rpc client and server. Maybe go-micro can rely on golang.com/x/net/http2/client_conn_pool.go to provide keep-alive long connection feature to boost the performance.

[asim]

I will look into the connection pooling however the thing to remember is that go-micro is a pluggable library made up of a number of components. A client Call will use the client, selector, registry and transport. The server will also make use of the transport. We provide sane defaults but if you want to build performance optimised plugins for each interface you can do this. You're not limited to using the defaults within go-micro. Example plugins are here https://github.com/micro/go-plugins.

Connection pooling is potentially something solved at the transport layer. Either by implementing a new http2 plugin or by adding a connection pooling Option to the existing transport.

[asim]

First steps. Benchmarking the transport https://github.com/micro/go-micro/blob/master/transport/http/http_test.go

go test -bench=. -benchtime=10s
testing: warning: no tests to run
PASS
BenchmarkTransport1-4      50000        277117 ns/op
BenchmarkTransport8-4     100000        190061 ns/op
BenchmarkTransport16-4    100000        186685 ns/op
BenchmarkTransport64-4    100000        194432 ns/op
BenchmarkTransport128-4   100000        251565 ns/op
ok      github.com/micro/go-micro/transport/http    107.467s

This is on my horrific 1.7 GHz Intel Core i5 2011 macbook air.