Make 'docker push' faster at the small expense of extra registry disk space

[rboyer]

Docker push currently uses default gzip settings to ship layers to the registry server. This is somewhat cpu intensive and annoying when doing frequent pushes (as you would when inserting Docker into your CI pipeline).

Instead of using the default gzip settings use gzip.BestSpeed which is faster but compresses less completely than the defaults. This pushes gzip closer to Google's Snappy compression library in spirit ("high speeds and reasonable compression").

Timing comparision using the google/golang image over localhost into a sandbox registry:

# stock docker 1.3.1
$ time sudo docker push localhost:5050/google/golang
...
real    1m22.158s
user    0m0.217s
sys 0m0.138s

# docker 1.3.1 with the BestSpeed change
...
real    0m46.427s
user    0m0.224s
sys 0m0.133s

This is somewhat related to #7291 but that issue is locked from non-contributor comments.

[dmp42]

cc @stevvooe @unclejack

[unclejack]

I'm sorry, but the gains in compression speed aren't worth it for everyone. There are a lot of users out there who will get their TCP connections dropped due to bad connectivity before they get a chance to complete the upload of a layer. Some are uploading large images for Android build environments. As you can imagine, these environments are pretty big and a single layer can exceed 5-8 GB, even when it is compressed.

There are also users who have "data allowances" set by their ISPs.

We should focus on improving the performance of the gzip package, instead of lowering the compression level. There's a lot of room for improvement there. I'm not saying that the gzip package from Go's stdlib hasn't had work done to it, I'm just saying it could use some more.

There are some tests I need to do to propose an alternative solution. Meanwhile, please don't merge this.

edit: This would also make image downloads slower and less cacheable by the CDN.

NOT LGTM

[crosbymichael]

We think this needs to be configurable but needs a proposal first. This may fix your issue with CPU load because you have faster bandwidth. However, someone running docker with larger CPUs and lower bandwidth will be hurt by this change.

Could you please open a proposal if you would like to have this setting configurable so that we can discuss before making code changes?

Thanks!

[bobrik]

@crosbymichael I can open a proposal.

I'm thinking about adding cli flags for docker push. Base images could be pushed with best compression since they don't change that often, but changing layers on top of them could be compressed with faster compression level/algorithm. What do you think?

docker push --compression-algo=gzip --compression-level=3 my-image

[crosbymichael]

@bobrik Have you or are you able to create some benchmarks on the different compression algos and levels for docker images. I know the main reason why we don't use compression is because of close CPU on a few of the providers causing pulls to be extremely slow when pulling and pushing images.

Maybe this is just an edge case but I'm not sure we have any real benchmarks. This may give us a better idea if we can change the default algo used with docker or if we are able to make a smart decision on what to use based on the machine. It may help by us not having to add a few of these flags to push and pull.

What do you think?

[unclejack]

I think we need to pursue the option of using cgzip or shelling out to gzip first before we attempt to add other flags. This would be far more useful than adding a ton of flags.

Please keep in mind that the compress/gzip package has lower compression ratio than the regular gzip binary found on most distributions. The lower compression ratio and lower performance should be good reasons to pursue shelling out to gzip first.

[bobrik]

Ok let's discuss cgzip or shelling out to gzip in #10181 first. We can get back to flags later.

[shouze]

👍 for compression