Error generated when messages size is too big

[nmengin]

Kind

Enhancement

Description

When a huge number of Swarmkit objects (services, configs and secrets) are created in the same time, this error appears:

level=error msg="agent: session failed" backoff=100ms error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7263689 vs. 4194304)"

This error seems to happen on all the cluster nodes in two differents use cases:

• When a big swarmkit object is created/updated or when a lot of swamkit objects are created/updated in the same, UC1 below.
• When the cluster leader comes down, UC2 below.

UC1

The error seems to happen because the gRPC client connection used for the Assignments method can not received messages which have a size greater than 4194304.
This method is called thanks to the ConnectionBroker Client Connection which is defined in two functions:

• Here to define the local connection to use on the leader
• Here to define the remote connection to use by the others nodes

UC2

The error seems to happen because the grpc client connection used in the Raft Transport (the structure which manages remote raft peers and sends messages to them) can not received messages which have a size greater than 4194304.

The Transport ClientConnection is initialized here.

Proposition

I propose to add an option on the Node Config to allow users to add a slice of grpc.DialOption which can be provided to the internal ClientConnections.

[kjelle]

Hello. We experience this problem, using docker 18.06.1-ce in swarm mode with api 1.38. The docker manager tries to call /v1.38/tasks which generates an error:
error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6908543 vs. 4194304)" rpc=/docker.swarmkit.v1.Control/ListTasks

[Marc3001]

Same here with our swarm cluster. We have ~350 services running and we are not able to list tasks as grpc message is reaching limit.
Also sometimes our worker is not able to respond any of our manager request and logs on worker are filled with these errors.

level=error msg="agent: session failed" backoff=100ms error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4823143 vs. 4194304)" module=node/agent node.id=sfygnj6xhjm420khg0pe6nn9x
level=info msg="manager selected by agent for new session: { }" module=node/agent node.id=sfygnj6xhjm420khg0pe6nn9x
level=info msg="waiting 90.674605ms before registering session" module=node/agent node.id=sfygnj6xhjm420khg0pe6nn9x

Then we are not able to update service or start/stop swarm tasks

[Marc3001]

Something I don't get is since 17.06 #2378 grpc max size should be 128Mb not 4Mb right ?
Why do we still have 4Mb max size on some messages ?

[kjelle]

I just moby/moby#37941 (comment) - we have messages on ~319 MB...

@Marc3001 setting a high max size really doesn't fix the problem. We experience this problem with messages way bigger than 128MB...