support running `chisel cut` on an existing chiseled root fs

[cjdcordeiro]

chisel cut should already be able to operate with a populated --root directory, however, the handling of tampered --root paths and the updating of existing Chisel manifests still needs to be worked in.

[polarathene]

In the meantime for reference, 2 consecutive instances of the same instruction will:

• RUN (chisel cut ...) => Adds the same layer weight (if cache is persisted into the layer as well it causes this to be duplicated in weight too).
• COPY (from chisel --root) => The 2nd COPY does not add layer weight for identical content (presumably this is less reliable at being minimal when extending an image).
• COPY --link => The 2nd COPY --link does add the same weight (presumably because each --link treats the source as a FROM scratch + COPY and that adds a difference somewhere 🤷‍♂)

[cjdcordeiro]

@polarathene @lbussell

Just a quick update on this: as we're planning for our next roadmap, we're considering introducing the ability for Chisel to cut on top of an existing filesystem. This addressed some of the frequently asked "extensibility" questions, and possibly improves on the process described in https://github.com/dotnet/dotnet-docker/blob/main/documentation/ubuntu-chiseled.md#how-do-i-install-additional-packages-on-chiseled-images. Before committing to this though, I'd like to share the expected feature behaviour, and preemptively get your concerns, if any. The plan is:

1. doing a chisel cut ... --root myfs foo_bar, on an existing myfs path where a previous chisel cut ... --root myfs baz_qux was executed, will effectively be the same as chisel cut ... --root myfs baz_qux foo_bar
   • in other words, whatever slices already existed, Chisel will attempt to reinstall them all over again. Why is that? First, we want Chisel to be deterministic, and building on top of --root would result on an unpredictable filesystem, especially due to the execution of mutation scripts. Second, we don't support version pinning in Chisel, meaning that Chisel always installs the latest versions of the software.
   • in practice, doing a second chisel cut on an existing chiseled filesystem is analogous to an "upgrade" + "reinstall"
2. if the resulting content of the subsequent chisel cut is the same as what's already in the --root path, the Chisel won't replace it! This is important, especially during container builds, so that there is no duplicated content across layers

Please let me know if this approach helps with your use cases.

[lbussell]

@cjdcordeiro thanks for the update. I had a few questions/concerns about installing on top of root filesystems that have already been "tampered" with.

For example, let's look at a really simplified pseudocode of one of the .NET chiseled images (actual dockerfile here):

FROM ubuntu AS chisel

# 1. install chisel
RUN curl github.com/canonical/chisel ...

# 2. create non-root user and group
# creates /rootfs/etc/passwd, /rootfs/etc/group, /rootfs/home/<user>
RUN groupadd && useradd ...

# 3. install slices
RUN chisel cut --root /rootfs 

# runtime-deps image
FROM scratch AS runtime-deps
COPY --from=chisel /rootfs/ /
USER $UID

In this example, is the chisel cut in step #3 going to fail, since we've already put some things in /rootfs? If so, we'd need to rearrange our layers, but that's not a big deal. My concern is about when users want to add slices to the image.

A user will want to add a slice to our image, I'd imagine the workflow would look something like this:

FROM ubuntu AS chisel
# assume chisel is installed
COPY --from=runtime-deps / /rootfs
RUN chisel cut --root /rootfs foo_bar

FROM runtime-deps
COPY --from=chisel /rootfs /

But since our runtime-deps image has some extras in it anyways, will this fail too? Or will chisel only refuse to install when there are conflicts? The same thing applies to our other .NET images as well, since we don't acquire .NET using slices.

But wait, this pattern will duplicate the files in /rootfs anyways! The final COPY instruction will result in all the files under /rootfs that already exist in runtime-deps being duplicated.

For the de-duplication to work, would we need to run chisel on the target rootfs? e.g.

FROM base_image
RUN --mount=type=bind,from=chisel,source=/chisel,target=/usr/bin/chisel \
    ["chisel", "cut", "--release", "ubuntu-24.04", "--root", "/", "foo_bar" ]

In this scenario, will chisel detect itself as evidence of tampering?
Additionally, chisel needs certs to reach out to Ubuntu servers, so something like this is necessary if ca_certs is not installed on the target rootfs:

FROM scratch
RUN --mount=type=bind,from=chisel,source=/chisel,target=/usr/bin/chisel \
    --mount=type=bind,from=chisel,source=/etc/ssl/certs/ca-certificates.crt,target=/etc/ssl/certs/ca-certificates.crt \
    ["chisel", "cut", "--release", "ubuntu-24.04", "--root", "/", "foo_bar" ]

If ca_certs has been installed, and it will be re-installed with the new cut operation, will that result in a conflict as well?

I'd like to hear what you think about these scenarios. Maybe I'm overthinking it 😃

[cjdcordeiro]

In this example, is the chisel cut in step #3 going to fail, since we've already put some things in /rootfs? If so, we'd need to rearrange our layers, but that's not a big deal. My concern is about when users want to add slices to the image.

This is a behaviour we still need to define. But I agree, that in your example, it would just require some rearranging.

But since our runtime-deps image has some extras in it anyways, will this fail too? Or will chisel only refuse to install when there are conflicts?

Thanks for sharing that example. It might be a good argument to be a bit more permissive in these cases. The point is that: we definitely have the means to check for discrepancies, i.e. crosschecking the chisel manifest with the rootfs contents. So we have options to consider: bail out on conflict VS just ignore and overwrite VS overwrite if a CLI acknowledgement is provided. At this stage, I can't really commit to a specific UX just yet, and wanted primarily to bring your attention to the chisel a + chisel b <=> chisel a+b factor.

For the de-duplication to work, would we need to run chisel on the target rootfs? e.g.

And yes, using the RUN --mount is your best option in this case. I don't think chisel would conflict with itself, cause it is not trying to overwrite itself anyway.

If ca_certs has been installed, and it will be re-installed with the new cut operation, will that result in a conflict as well?

Indeed you need the ca certs. If not installed yet, your RUN --mount should do the work, but indeed we have to consider the same scenario as above "if chisel is also cutting ca-certificates, will they conflict with the mounted ones and persist?". TBD

[polarathene]

Please let me know if this approach helps with your use cases.

If some of the content has changed, will only the difference be applied? Or will it be a naive copy of all contents which duplicates the weight of unchanged files?

Just a quick update on this: as we're planning for our next roadmap, we're considering introducing the ability for Chisel to cut on top of an existing filesystem

I vaguely recall having a WIP reply to another issue in the repo where I did this with a scratch image for / as the rootfs instead of relying on COPY, but it was a bit awkward as the RUN had to use the JSON array syntax RUN ["x", "y", "z"] to avoid using a shell I think (created /dev/ entries IIRC).

---

Regarding @lbussell comment about ca-certificates, I recall that being one annoyance due to a fixed location, but I'm pretty sure as long as the file required existed at that correct path, it was a non-issue installing to / as the rootfs, I also had chisel at the root IIRC.

---

especially due to the execution of mutation scripts.

I am familiar with this type of concern with a project I maintain, so that's perfectly fine.

we don't support version pinning in Chisel, meaning that Chisel always installs the latest versions of the software.

I rarely pin packages, but it does sometimes introduce issues without it. I know that can sometimes break building a Dockerfile for an older image release for example.

At this stage, I can't really commit to a specific UX just yet, and wanted primarily to bring your attention to the chisel a + chisel b <=> chisel a+b factor.

👍 I'm good with that. My main interest is avoiding image inefficiency with redundant copies. Ideally any new layer is only adding what is needed / changed.

[lbussell]

chisel a + chisel b <=> chisel a+b

This idea is fine to me - it is pretty much what I would expect to happen.

[cjdcordeiro]

If some of the content has changed, will only the difference be applied? Or will it be a naive copy of all contents which duplicates the weight of unchanged files?

It will be a full re-install of all slices. if the resulting contents end up being the same as what's already in the rootfs you're writing to, then chisel won't write again (thus avoid layer inefficiency)

[polarathene]

It will be a full re-install of all slices.

Ah ok, so each time a new chisel cut layer is added that adds a new slice, the full weight of an earlier chisel cut + new slice is added again. A separate tool would be needed to only copy the actual diff, that's usually the case 😓

[cjdcordeiro]

It will be a full re-install of all slices.

Ah ok, so each time a new chisel cut layer is added that adds a new slice, the full weight of an earlier chisel cut + new slice is added again. A separate tool would be needed to only copy the actual diff, that's usually the case 😓

No, that's not the case.

if the resulting contents end up being the same as what's already in the rootfs you're writing to, then chisel won't write again (thus avoid layer inefficiency)

So, for example, let's say chisel cut foo_bar generates the following layer:

/
/myfile
/var/lib/chisel/manifest.wall

In your Docker build, this would be Layer 1.

Then, again in your Docker build, in layer 2, you do RUN chisel cut baz_qux. Chisel will see that there is already a slice installed from a previous operation, and will try to update+install it again (as if it you were running chisel cut foo_bar baz_qux), The result: unless the foo package's /myfile was updated upstream, or a mutation script ends up modifying it, Chisel may conclude that there's no need to override Layer 1's /myfile because the outcome of this 2nd chisel cut will provide the same /myfile (same digest)