Set grpc max message size to 128MB.

[anshulpundir]

Signed-off-by: Anshul Pundir anshul.pundir@docker.com

[codecov]

Codecov Report

Merging #2375 into master will increase coverage by 0.09%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2375      +/-   ##
==========================================
+ Coverage   60.35%   60.45%   +0.09%     
==========================================
  Files         128      128              
  Lines       26267    26268       +1     
==========================================
+ Hits        15854    15880      +26     
+ Misses       9011     8993      -18     
+ Partials     1402     1395       -7

[nishanttotla]

I'd like to know the full side effects of doing this. My hunch is that message encoding gets affected for larger message size limits. Also related: https://stackoverflow.com/questions/34128872/google-protobuf-maximum-size

Ping @stevvooe.

[wsong]

I'd also recommend that we make this configurable. The default value for this parameter can stay at 128MB or whatever, but it'd be nice if we read this value out of /etc/docker/daemon.json and passed it into the swarmkit code in the moby/moby repo.

[anshulpundir]

I'd also recommend that we make this configurable. The default value for this parameter can stay at 128MB or whatever, but it'd be nice if we read this value out of /etc/docker/daemon.json and passed it into the swarmkit code in the moby/moby repo.

Agreed @wsong

I'd like to know the full side effects of doing this.

I'd like to also :)

My hunch is that message encoding gets affected for larger message size limits.

I think encoding and decoding depends on actual message size and the structure of the data. For example, a deeply nested struct would be more costly to encode/decode. I don't think changing the limit affects encoding/decoding performance. @nishanttotla

[aluzzardi]

The default value for this parameter can stay at 128MB or whatever, but it'd be nice if we read this value out of /etc/docker/daemon.json and passed it into the swarmkit code in the moby/moby repo.

I'm not sure about that - when moving to the "proper" solution (streaming), the config option won't make sense anymore.

I don't know if it makes sense to introduce a new option in a patch release just so we can deprecate it immediately after.

[aluzzardi]

nit: This could be a constant

[stevvooe]

Use shift syntax: 128 << 20.

[aluzzardi]

Overall looks good

What is going to happen when we reach 128M? Any way we could log a proper error message?

[nishanttotla]

I think some log messages should be included to bubble up errors from hitting the gRPC limit (either in this PR or as an immediate follow up).

If we increase the limit to 128MB, then is it worth the engineering effort to implement streaming in addition? If we do implement streaming later, then we'd want to bring the limit down again. This may cause backward compatibility issues.

I agree with @aluzzardi that if we introduce this option, it doesn't make sense to immediately deprecate it.

[anshulpundir]

We just met and agreed to not add the knob to adjust the max grpc message size.
I will manually test the positive test case today and investigate adding an integration test for positive and negative scenarios.

[stevvooe]

-1 on making this configurable.

If we increase the limit to 128MB, then is it worth the engineering effort to implement streaming in addition?

Yes, this is a horrible thing to have as a scaling limit and it needs to be addressed. Besides patching this as a workaround, the next focus should be actually fixing this to handle arbitrary snapshot sizes. Increasing the message size limit and calling it good is not an acceptable engineering solution. Increasing this limit exposes us to message floods on all services, not just the raft service.

Is there anything we can add to log messages that are over some threshold, so that we can have idea of the level of impact this is having? Prom counters bucketed by size might be appropriate. Such large messages will have an effect on stability at around 10MB and up with protobuf. The allocations required for processing a 128MB message may cause stability issues. I'd really like to know if there are messages that are larger than 1MB transiting swarm GRPC endpoints, as those are likely causing "buffer bloat" and allocation spikes. We really need to have an easy way for users to understand when they are impacted by this problem.

[wsong]

Yeah, after meeting offline, I agree that this should not be configurable.

Adding a metric to track large (i.e. > 1MB) GRPC messages is definitely a good idea, but it should probably go in a separate PR. Logging the length of every single message before sending it might have some unforeseen performance implications.

[stevvooe]

Logging the length of every single message before sending it might have some unforeseen performance implications.

No really what I was suggesting:

if len(msg) > 1 << 20 {
 // log it.
}

Better would be to have bucketed counters (unrolled below):

if len(msg) >= 1 << 20 {
  gt1MB++
}

if len(msg) >= 4 << 20 {
  gt4MB++
}

if len(msg) >= 10 << 20 {
  gt10MB++
}

if len(msg) >= 16 << 20 {
  gt16MB++
}
...

Notice that this is a cumulative counter setup, rather than bucketed, which allows easy aggregates.

probably go in a separate PR.

Sure, but let's make sure it gets backported: right now, this condition is hard to diagnose and debug.

[anshulpundir]

Manually tested for now. Positive test-case for around 120Mib. Negative for around 240Mib. Will explore the metrics and error reporting on a separate PR.

[andrewhsu]

Hmm...looks like the linter changed a few hours ago: golang/lint#319

I'm seeing these errors from the circleci job:

🐳 lint
ca/keyreadwriter.go:190:2: redundant if ...; err != nil check, just return error instead.
manager/allocator/network.go:175:2: redundant if ...; err != nil check, just return error instead.
manager/controlapi/network.go:99:2: redundant if ...; err != nil check, just return error instead.
manager/controlapi/service.go:59:2: redundant if ...; err != nil check, just return error instead.
manager/controlapi/service.go:164:2: redundant if ...; err != nil check, just return error instead.
manager/controlapi/service.go:484:2: redundant if ...; err != nil check, just return error instead.
manager/dispatcher/dispatcher.go:857:3: redundant if ...; err != nil check, just return error instead.
manager/logbroker/broker_test.go:405:3: redundant if ...; err != nil check, just return error instead.
manager/logbroker/broker_test.go:527:3: redundant if ...; err != nil check, just return error instead.
manager/logbroker/broker_test.go:658:3: redundant if ...; err != nil check, just return error instead.
manager/orchestrator/update/updater.go:387:4: redundant if ...; err != nil check, just return error instead.
manager/scheduler/scheduler.go:95:2: redundant if ...; err != nil check, just return error instead.
manager/state/raft/raft.go:1286:2: redundant if ...; err != nil check, just return error instead.
manager/state/raft/raft.go:1851:2: redundant if ...; err != nil check, just return error instead.
manager/state/raft/storage/storage.go:229:2: redundant if ...; err != nil check, just return error instead.
manager/state/raft/transport/transport.go:238:2: redundant if ...; err != nil check, just return error instead.
make: *** [lint] Error 1

[marcusmartins]

@anshulpundir @nishanttotla the CI is failing on lint. Do you know why?

[thaJeztah]

Can we pin the linter to a specific version to prevent that?

[wsong]

Alternately, it may be faster to just open up a separate PR to fix those places in the code; it looks like there might not be too many.

[anshulpundir]

Alternately, it may be faster to just open up a separate PR to fix those places in the code; it looks like there might not be too many.

Doing this right now.