Start to flesh out crane optimize. #879

mattmoor · 2020-12-19T00:32:29Z

This is a hidden command, which roundtrips a remote image to a target image through tarball.LayerFromOpener(layer.Uncompressed).

Right now this does nothing to force estargz (still need GGCR_EXPERIMENT_ESTARGZ=1) or prioritize files (need estargz.WithPrioritizedFiles(foo)), but want to start the convo.

Fixes: #878

This is a hidden command, which roundtrips a remote image to a target image through `tarball.LayerFromOpener(layer.Uncompressed)`. Right now this does nothing to force estargz (still need `GGCR_EXPERIMENT_ESTARGZ=1`) or prioritize files (need `estargz.WithPrioritizedFiles(foo)`), but want to start the convo. Fixes: google#878

codecov-io · 2020-12-19T01:28:09Z

Codecov Report

Merging #879 (48acc2c) into master (8b4c3b5) will decrease coverage by 1.57%.
The diff coverage is 14.41%.

@@            Coverage Diff             @@
##           master     #879      +/-   ##
==========================================
- Coverage   74.33%   72.76%   -1.58%     
==========================================
  Files         107      108       +1     
  Lines        4523     4655     +132     
==========================================
+ Hits         3362     3387      +25     
- Misses        661      758      +97     
- Partials      500      510      +10

Impacted Files	Coverage Δ
pkg/crane/optimize.go	`0.00% <0.00%> (ø)`
pkg/v1/tarball/layer.go	`71.17% <61.53%> (-1.06%)`	⬇️
pkg/v1/layout/write.go	`47.53% <0.00%> (-0.89%)`	⬇️
pkg/v1/mutate/mutate.go	`72.50% <0.00%> (+0.52%)`	⬆️
pkg/v1/mutate/index.go	`79.41% <0.00%> (+1.63%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b4c3b5...48acc2c. Read the comment docs.

mattmoor · 2020-12-19T01:28:22Z

Alright, I also have:

crane optimize gcr.io/distroless/static:nonroot ghcr.io/mattmoor/distroless/static:nonroot --prioritize=etc/passwd

... working to prioritize /etc/passwd, though I'm not sure there's a good way to tell whether it found the file in question, and I get different digests when I pass /etc/passwd vs. etc/passwd vs. nothing 🤔

cc @ktock @AkihiroSuda FYI

cmd/crane/cmd/optimize.go

jonjohnsonjr · 2020-12-21T18:30:09Z

cmd/crane/cmd/optimize.go

+
+	cmd := &cobra.Command{
+		Use:     "optimize SRC DST",
+		Hidden:  true,


Because this is hidden, I don't really care that much, but I would rather this be called something like [e]stargz[ify] (and documented as such), because "optimize" doesn't mean much.

I'll let you and @imjasonh fight about this. That was my original proposal and I care almost not at all.

pkg/crane/optimize.go

jonjohnsonjr · 2020-12-21T18:31:59Z

cmd/crane/cmd/optimize.go

+		Hidden:  true,
+		Aliases: []string{"opt"},
+		Short:   "Optimize a remote container image from src to dst",
+		Args:    cobra.ExactArgs(2),


It would be nice if this allowed one arg and would just retag the optimized version, with an optional -t flag for the dst if you want to leave src alone.

IDK how I feel about in-place optimization, but most of the places I want this for are things that aren't yet producing these images.

jonjohnsonjr · 2020-12-21T18:37:45Z

pkg/crane/optimize.go

+			Descriptor: v1.Descriptor{
+				URLs:        desc.URLs,
+				MediaType:   desc.MediaType,
+				Annotations: desc.Annotations,
+				Platform:    desc.Platform,
+			},


We could probably do something in mutate to make this a little nicer.

jonjohnsonjr · 2020-12-21T18:42:13Z

pkg/crane/optimize.go

+	return remote.Write(dstRef, oimg, o.remote...)
+}
+
+func optimizeImage(img v1.Image, prioritize []string) (v1.Image, error) {


What you're doing follows the patterns from crane.Copy, which makes sense, but it would be nice to expose this separately as well, maybe OptimizeImage, so it can be used with non-remote images.

It might also make sense for some of this boilerplate to live in mutate.

jonjohnsonjr · 2020-12-21T18:43:21Z

pkg/crane/optimize.go

+)
+
+// Optimize optimizes a remote image or index from src to dst.
+func Optimize(src, dst string, prioritize []string, opt ...Option) error {


My "as long as it's hidden" concession for the command also kind of applies here. I don't think we should expose this in pkg/crane until we're happy with exposing it in the CLI as well.

Given how things are structured, that isn't particularly feasible (without deviating significantly from the broader style). What did you have in mind?

You could keep this under cmd/crane/cmd, which is what I've done for gcrane stuff I don't want to expose.

jonjohnsonjr · 2020-12-21T18:44:48Z

pkg/crane/optimize.go

+	return remote.WriteIndex(dstRef, oidx, o.remote...)
+}
+
+func optimizeIndex(idx v1.ImageIndex, prioritize []string) (v1.ImageIndex, error) {


We really need a:

func Map(f func(v1.Image) (v1.Image, error), i v1.ImageOrIndex) (v1.ImageOrIndex, error)

I believe I have one downstream somewhere 😇

https://github.com/mattmoor/mink/blob/4afdeb3398c4d56401d4638ebd5cb2cb85ad12ad/pkg/bundles/map.go#L118

Oh yeah you cced me on this earlier. My only issue is the pushing bit, we are missing an abstraction here for an ImageOrIndex thing (strawman here: #835)

pkg/crane/optimize.go

cmd/crane/cmd/optimize.go

mattmoor · 2020-12-21T19:08:01Z

cmd/crane/cmd/optimize.go

+
+	cmd := &cobra.Command{
+		Use:     "optimize SRC DST",
+		Hidden:  true,


I'll let you and @imjasonh fight about this. That was my original proposal and I care almost not at all.

mattmoor · 2020-12-21T19:09:26Z

cmd/crane/cmd/optimize.go

+		Hidden:  true,
+		Aliases: []string{"opt"},
+		Short:   "Optimize a remote container image from src to dst",
+		Args:    cobra.ExactArgs(2),


IDK how I feel about in-place optimization, but most of the places I want this for are things that aren't yet producing these images.

mattmoor · 2020-12-21T19:11:44Z

pkg/crane/optimize.go

+	return remote.WriteIndex(dstRef, oidx, o.remote...)
+}
+
+func optimizeIndex(idx v1.ImageIndex, prioritize []string) (v1.ImageIndex, error) {


https://github.com/mattmoor/mink/blob/4afdeb3398c4d56401d4638ebd5cb2cb85ad12ad/pkg/bundles/map.go#L118

mattmoor · 2020-12-21T19:13:34Z

pkg/crane/optimize.go

+)
+
+// Optimize optimizes a remote image or index from src to dst.
+func Optimize(src, dst string, prioritize []string, opt ...Option) error {


Given how things are structured, that isn't particularly feasible (without deviating significantly from the broader style). What did you have in mind?

mattmoor · 2020-12-21T19:15:10Z

pkg/crane/optimize.go

+
+		olayers = append(olayers, mutate.Addendum{
+			Layer:     olayer,
+			History:   cfg.History[i],


I may drop this as I'm not sure it is actually correct.

pkg/crane/optimize.go

jonjohnsonjr · 2020-12-21T19:34:37Z

Moving this down from inline comment so it doesn't get lost...

IDK how I feel about in-place optimization, but most of the places I want this for are things that aren't yet producing these images.

Given that it's ostensibly backward-compatible, I don't see a huge issue, but I understand the reservations.

If we do in-place, one thing that would be nice for one-liners is to print the resulting image to stdout:

$ crane optimize foo@sha256:abc
foo@sha256:def

So you could do stuff like:

$ crane export $(crane optimize $(ko publish ./cmd/ko)) | tar -tvf -

That might not make sense for in-place by tag, but:

$ crane optimize foo:latest
foo:latest@sha256:def

Would be kinda nice? I like the pattern in general for mutation stuff, especially if you don't want to write me a compiler so this stuff only works in-registry. I'd be okay with needing to set a flag to overwrite tags, as well.

mattmoor · 2020-12-21T19:43:17Z

BTW, I think being able to collect missed files (containerd/stargz-snapshotter#216) will be very useful for surfacing user errors with the prioritized file stuff (thanks @ktock!).

mattmoor force-pushed the optimize-cmd branch from bae7180 to b6faed6 Compare December 19, 2020 01:04

Add --prioritize flag to prioritize files

f4b6650

This was referenced Dec 19, 2020

Propagate not-found error during sort containerd/stargz-snapshotter#212

Merged

Support "/"-prefixed prioritized files containerd/stargz-snapshotter#213

Merged

Allow some of prioritized files are missed during sort containerd/stargz-snapshotter#216

Merged

jonjohnsonjr reviewed Dec 21, 2020

View reviewed changes

Fix headers, drop history

c1b9c5c

mattmoor commented Dec 21, 2020

View reviewed changes

Drop unused variable

fbf72a6

mattmoor added 2 commits December 21, 2020 12:24

Add explicit option for estargz

4ca4734

Add a warning comment to crane.Optimize

48acc2c

jonjohnsonjr approved these changes Dec 21, 2020

View reviewed changes

mattmoor merged commit c95d7bd into google:master Dec 22, 2020

mattmoor deleted the optimize-cmd branch December 22, 2020 00:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start to flesh out crane optimize. #879

Start to flesh out crane optimize. #879

mattmoor commented Dec 19, 2020

codecov-io commented Dec 19, 2020 •

edited

mattmoor commented Dec 19, 2020

jonjohnsonjr Dec 21, 2020

mattmoor Dec 21, 2020

jonjohnsonjr Dec 21, 2020

mattmoor Dec 21, 2020

jonjohnsonjr Dec 21, 2020

jonjohnsonjr Dec 21, 2020

jonjohnsonjr Dec 21, 2020

mattmoor Dec 21, 2020

jonjohnsonjr Dec 21, 2020

jonjohnsonjr Dec 21, 2020

mattmoor Dec 21, 2020

mattmoor Dec 21, 2020

jonjohnsonjr Dec 21, 2020

mattmoor Dec 21, 2020

mattmoor Dec 21, 2020

mattmoor Dec 21, 2020

mattmoor Dec 21, 2020

mattmoor Dec 21, 2020

jonjohnsonjr commented Dec 21, 2020

mattmoor commented Dec 21, 2020

Start to flesh out crane optimize. #879

Start to flesh out crane optimize. #879

Conversation

mattmoor commented Dec 19, 2020

codecov-io commented Dec 19, 2020 • edited

Codecov Report

mattmoor commented Dec 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonjohnsonjr commented Dec 21, 2020

mattmoor commented Dec 21, 2020

codecov-io commented Dec 19, 2020 •

edited