Having large numbers of networks can cause node allocation to fail

[dperny]

Since 9fa9ce1, nodes are now allocated with a network attachment for every network.

Unfortunately, this can cause the size of a Node object to exceed the maximum raft message size, which prevents it from being committed to the object store.

From moby/moby#36792

Forgot to mention the same error was triggered by a different action than the one initially reported on this issue, in my case it appeared when creating ~1700 networks and deploying one service on each network.

Jun 01 00:39:52 ip-172-16-0-128 dockerd[1284]: time="2018-06-01T00:39:52.220386603Z" level=error msg="Failed to commit allocation of network resources for node rul9pnxcc2hpj3o7eya1redpk" error="raft: raft message is too large and can't be sent" module=node node.id=281crvu

There are many possible fixes, including disallowing too many attachments, but one way or another we can't wind up with raft messages too large like this.

/cc @ctelfer

[ddtmachado]

Hey @dperny this 9fa9ce1 also degraded the performance of docker managers and service creation in general on the situation described by @eduardolundgren

I can show more precise values, steps and the kind of stress test on the next run. but right now I see like 300% increase in CPU usage on the swarm leader (average use was 15% CPU while running 17.09 and 50% running versions after that commit like 17.12, 18.03 and 18.05)

Basic scenario was a swarm cluster of 3 managers and 12 workers, then running a script that created 2000 networks and 2000 services using those networks.

[eduardolundgren]

@dperny any update on this issue?