Skip to content
This repository has been archived by the owner. It is now read-only.

NG: HTTP Transport compression for Layer downloads #694

Closed
stevvooe opened this issue Nov 7, 2014 · 5 comments

Comments

@stevvooe
Copy link
Contributor

commented Nov 7, 2014

Because layer files are not compressed, we may waste bandwidth transferring them between registry services and docker agents, or via external CDNs, such as cloudfront. Let's take some time to understand the tradeoffs here and what it would to support this.

@stevvooe stevvooe added this to the GO-RC milestone Nov 7, 2014
@wking

This comment has been minimized.

Copy link
Contributor

commented Nov 7, 2014

On Thu, Nov 06, 2014 at 06:25:32PM -0800, Stephen Day wrote:

Because layer files are not compressed, we may waste bandwidth
transferring them between registry services and docker agents, or
via external CDNs, such as cloudfront. Let's take some time to
understand the tradeoffs here and what it would to support this.

Uncompressed images also waste space in the storage. For example,
unpacking Gentoo's package tree (~60 MB as a .tar.xz) adds a layer
that consumes ~300 MB. I'd just have clients chose from a list of
supported compression schemes and set Content-Encoding appropriately
when uploading. That means the registry would have to decode on the
fly to calculate/confirm the content-addressable storage id, but I
think the upload-time space savings are worth a few extra CPU cycles.
Of course, this means that you'd need to lock out simultaneous writes
to the same id, but we should be able to figure that out. Folks who
don't want to spend CPU cycles on decompression could just configure
an empty list of allowed encodings, and we'd return the list of
allowed encodings for a given registry via the _ping endpoint.

@wking wking referenced this issue Nov 7, 2014
3 of 9 tasks complete
@dmp42

This comment has been minimized.

Copy link
Contributor

commented Nov 7, 2014

but I think the upload-time space savings are worth a few extra CPU cycles.

Actually, this is likely not accurate, and the reason why layer compression was removed in the past. (cc @vieux)

We need to support both compressed and uncompressed uploads - and content-negotiation for download (likely being able to serve both).

@wking

This comment has been minimized.

Copy link
Contributor

commented Nov 7, 2014

On Fri, Nov 07, 2014 at 12:06:55AM -0800, Olivier Gambier wrote:

but I think the upload-time space savings are worth a few extra
CPU cycles.

Actually, this is likely not accurate, and the reason why layer
compression was removed in the past. (cc @vieux)

Ah, maybe just a symptom of my local registry having few users and
small disks ;). With a configurable list of allowed upload encodings,
it would be easy to pick the appropriate setting for a given registry,
but collecting the existing wisdom on this issue will help pick a good
default and give good guidance in the docs for that config setting.

@dmp42

This comment has been minimized.

Copy link
Contributor

commented Nov 7, 2014

+1 ;)

@docker docker locked and limited conversation to collaborators Jan 14, 2015
@stevvooe

This comment has been minimized.

Copy link
Contributor Author

commented Jan 14, 2015

Superseded by docker/distribution#65.

@stevvooe stevvooe closed this Jan 14, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
3 participants
You can’t perform that action at this time.