Compress layers on push to a v2 registry

[dmcgowan]

When buffering to file add support for compressing the tar contents. Since digest should be computed while writing buffer, include digest creation during buffer.

Pull auto detects compression requiring no additional changes.

Ping @jlhawn

[jlhawn]

LGTM

@jfrazelle Thanks for adding this to the 1.6.0 milestone! It is going to be very important for the release as it significantly reduces payload sizes which in turn reduces registry storage usage and bandwidth requirements.

Smaller downloads == Faster downloads == Happier users 👍

[jessfraz]

ping @crosbymichael this is enhancements to make things faster if its
alright with you we should consider for 1.6

On Fri, Mar 27, 2015 at 2:03 PM, Josh Hawn notifications@github.com wrote:

LGTM

@jfrazelle https://github.com/jfrazelle Thanks for adding this to the
1.6.0 milestone! It is going to be very important for the release as it
significantly reduces payload sizes which in turn reduces registry storage
usage and bandwidth requirements.

Smaller downloads == Faster downloads == Happier users [image: 👍]

—
Reply to this email directly or view it on GitHub
#11869 (comment).

[crosbymichael]

I'm not sure this works, you need to close the underlying file and reset the hash manually. you cannot cast the multiwriter here.

[crosbymichael]

nvm, i see that is going on now, if it's a gzip.Writer then the hash will not be populated, is this correct?

[dmcgowan]

Yes, sorry it is kind of confusing. Yes it is only for closing the gzip writer, the multiwriter does not need to be closed.

[thaJeztah]

If it's confusing; perhaps some docs describing the function to prevent future coders tripping over it?

[jlhawn]

maybe we can be more explicit with the code:

func bufferToFile(f *os.File, src io.Reader, compress bool) (int64, digest.Digest, error) {
    var (
        d = digest.NewCanonicalDigester() // Uses sha256.
        w = io.MultiWriter(f, &d)
        gz *gzip.Writer
    )

    if compress {
        gz = gzip.NewWriter(w)
        w = gz
    }

    _, err := io.Copy(w, src)
    if gz != nil {
        gz.Close()
    }
    if err != nil {
        return 0, "", err
    }

    n, err := f.Seek(0, os.SEEK_CUR)
    if err != nil {
        return 0, "", err
    }

    if _, err := f.Seek(0, os.SEEK_SET); err != nil {
        return 0, "", err
    }

    return n, d.Digest(), nil
}

[dmcgowan]

Added a comment as @thaJeztah suggested, should be all that is required

[thaJeztah]

Thanks!

[dmcgowan]

Updated to make code easier to read based on @jlhawn's recommendation

[crosbymichael]

Thanks for the readability changes. I'm getting rusty and cannot read code anymore.

[jlhawn]

@dmcgowan I was talking to @crosbymichael and @NathanMcCauley IRL about this PR and we agreed that if bufferToFile() isn't called from anywhere else in the project that we should just not have a compressed argument and just always compress.

[crosbymichael]

@dmcgowan can you remove that bool?

[dmcgowan]

Updated

[jlhawn]

L^GTM

[crosbymichael]

@jlhawn that LGTM is not parse-able

[LK4D4]

@dmcgowan Could you rebase against master and fix compilation errors?

[crosbymichael]

LGTM