clientv3: configure grpc.MaxCallSendMsgSize, grpc.MaxCallRecvMsgSize

[gyuho]

For server-side, etcd --max-request-bytes must be configured--defaults to 1.5MB.

v3.2

https://github.com/coreos/etcd/blob/1fa227da71556c77f083ebfdccd1e298e4d1bf65/embed/config.go#L44

v3.3+

https://github.com/coreos/etcd/blob/b0a7623be82c332942994ac2a09c70d930ebbb77/embed/config.go#L50

External projects can configure this with embed.NewConfig().MaxRequestBytes = 5 * 1024 * 1024.

For client-side, gRPC remote calls must be configured with grpc.MaxCallSendMsgSize and grpc.MaxCallRecvMsgSize that correspond to the server setting.

kubernetes/kubernetes#51099 happen when these two parameters do not match.

Another problem in etcd-side, clientv3 was not setting request call limit, thus defaulting to 4MB (clients can only receive <4MB response).

So even with etcd --max-request-bytes 5.5MB(default value), client requests with 5MB data will fail with rpc error: code = ResourceExhausted desc = grpc: received message larger than max (56710706 vs. 4194304).

TODO

• let users configure client-side request limit, or just set grpc.MaxCallSendMsgSize(math.MaxInt32), grpc.MaxCallRecvMsgSize(math.MaxInt32) to all API calls, since server would return errors anyway.
• backport this patch with v3.2.12

Reference: kubernetes/kubernetes#57160 (comment)

/cc @jpbetz @xiang90