Squash some layers

[dtrudg]

In SingularityCE, we have added the ability to create a writable overlay in an OCI-SIF file. A writable overlay is an ext3 filesystem image, as an additional final layer on the OCI image encapsulated in the OCI-SIF.

We can push an OCI-SIF, containing an overlay, into an OCI registy. We then have a remote image with N original layers, plus a final ext3 layer - the writable overlay. Usually an OCI-SIF will have a single non-overlay layer, since we squash OCI images by default on a singularity pull --oci. However, we can optionally create a multi-layer OCI-SIF, and if an overlay is added and the resulting image pushed, we then have this in the registry:

• Layer 1 - squashfs
• Layer 2 - squashfs
• Layer N - squashfs
• Layer N+1 - ext3 overlay

On singularity pull of this image from the registry, the default in SingularityCE is to squash all layers. We use ocitools mutate.Squash for this. However, the squash will fail here, because we don't have the ability to squash an ext3 layer.

Users would probably expect that the non-overlay layers are squashed, but the overlay layer is preserved as a writable overlay on pull (because writable overlays travel with the image, as-is, in native mode)

This requires the ability to squash a subset of layers, e.g. implement:

func SquashSubset(base v1.Image, start, end int) (v1.Image, error)

Any thoughts on this @tri-adam @wobito ?

The corresponding SingularityCE issue is sylabs/singularity#3135

We could, alternatively do things like....

• Change nothing in oci-tools, and require that images with an overlay are always pulled with --keep-layers in SingularityCE, and fail otherwise.
• Change nothing in oci-tools, and infer --keep-layers in SingularityCE when pulling an image that has an overlay. This means an image with an overlay is never squashed on pull.
• Change mutate.Squash so that it can squash ext3 also... which would mean calling mutate.Squash turns an image with a writable overlay into a single-layer r/o image, with the overlay content present.

[wobito]

Users would probably expect that the non-overlay layers are squashed, but the overlay layer is preserved as a writable overlay on pull (because writable overlays travel with the image, as-is, in native mode)

I would agree as the default behaviour is to squash (non-overlays) and leaving the overlay, purposed here:

func SquashSubset(base v1.Image, start, end int) (v1.Image, error)

makes sense over the alternates (more work yes), but then using overlay seal (mentioned in sylabs/singularity#3135) to do that final conversion lines up nicely in my head.

[tri-adam]

Think it might make sense to generalize things a bit... was going to write something up but figure it might be simpler to show an example. Would this do what you need? #71

[dtrudg]

Think it might make sense to generalize things a bit... was going to write something up but figure it might be simpler to show an example. Would this do what you need? #71

I think that would work, yeah... but I'm not entirely convinced about the generalization for the following reasons...

1. This makes it possible to select discontinuous ranges. I.E. in a 5-layer image, select layers 1,2,4,5, but not 3 ... which doesn't really make sense for a squash.

2. Writing a selector function on v1.Layer() means quite a lot of additional .Digest() or similar calls in order to select what we want, given that v1.Layer() has no concept of its position in the image. I can't really think of a situation where a squash operation is not more easily expressed with layer indices... given we want to squash things because they are in a certain position in the stack of layers... rather than any other criterion? Maybe I'm missing something?

Having said that... it's not a big deal. If you prefer the selector approach then it'll work for Singularity 👍

[tri-adam]

I think that would work, yeah... but I'm not entirely convinced about the generalization for the following reasons...

1. This makes it possible to select discontinuous ranges. I.E. in a 5-layer image, select layers 1,2,4,5, but not 3 ... which doesn't really make sense for a squash.
2. Writing a selector function on v1.Layer() means quite a lot of additional .Digest() or similar calls in order to select what we want, given that v1.Layer() has no concept of its position in the image. I can't really think of a situation where a squash operation is not more easily expressed with layer indices... given we want to squash things because they are in a certain position in the stack of layers... rather than any other criterion? Maybe I'm missing something?

Having said that... it's not a big deal. If you prefer the selector approach then it'll work for Singularity 👍

Those are good points, I agree. I'd like to keep the building blocks fairly generic to support future needs, but there's no reason that can't be done with the squash API you suggest. I'll give that a re-work shortly, thanks!

[tri-adam]

Updated #71... @dtrudg PTAL, thank you!