[Discussion] Drop mtree completely

[Itxaka]

I dont think we should support mtree anymore, here are my reasons:

• It came as a feature request with not a lot of requirements nor any reasons behind it other than Chain Of Trust
• It clutters the metadata files with the mtree objects
• It slows down the creation of artifacts and the install/upgrade

And the big one, we have cosign in place which has the following advantages:

• Its much more integrated in the pipeline
• Its the future of signing containers
• It can be automatically created and pushed
• Artifacts can be validated by anyone with the cosign binary as the registry is public
• No keys involved so the human vulnerability factor is heavily removed (no one to tamper with the values)
• signatures live together with the artifacts, no need to hack our way around it to support them, like currently

So I propose dropping them altogether, as I don't see any advantages over cosign. Sure they sha512 each file inside the container but what advantage is that over the signature of the whole container as that includes all the files?

Investing a bit of time into getting a proper cosign lib/interface/glue to use cosign directly as part of elemental would be required, but it doesn't seem too difficult to mimic the cosign cmd part.

Thoughts?

[davidcassany]

I am also in favor of dropping mtree support, I see little gain on it and something we, and probably everyone else, is likely to end up skipping or bypassing.

I am a noob regarding cosign, so I can't really tell where and how it compares in our use cases. Mtree has the capability to actually verify something that is already installed in the system, something I doubt we can easily achieve with cosign. But, in any case, with the current implementation we have I doubt this is easily achievable, too many moving bits and potential install/upgrade hooks. To me the mtree support we have now seams unlikely to be used as is or even incomplete to take advantage of its potential.

If we consider really supporting mtree I would take it from another perspective and use it as an informative/debug tool to diff images or layers. Basically to diff active vs passive vs recovery vs any other accessible container. Or even something to verify/create delta upgrades if we ever consider this possibility. However even in such cases we could have a look at higher level tools which are likely to relay on mtree too, like container-diff and similar pieces.

[mudler]

While I do agree that cosign should be more than enough, this is actually something that bubbled up as a discussion recently, and a feature we have been asked for (again). Mtree could be used to check if a system image matches the container checksum, and that's something we will have to provide (sooner, or later)

[mudler]

I am also in favor of dropping mtree support, I see little gain on it and something we, and probably everyone else, is likely to end up skipping or bypassing.

I am a noob regarding cosign, so I can't really tell where and how it compares in our use cases. Mtree has the capability to actually verify something that is already installed in the system, something I doubt we can easily achieve with cosign. But, in any case, with the current implementation we have I doubt this is easily achievable, too many moving bits and potential install/upgrade hooks. To me the mtree support we have now seams unlikely to be used as is or even incomplete to take advantage of its potential.

I couldn't agree more. IMHO, we should streamline it's approach offering mtree capabilities directly from the elemental-cli, for example (fictional example ahead):

elemental-cli generate-sha <image>
docker push <image>:signature

And then should be picked up by our upgrade process automatically.
Similarly, at that point we can offer a "verify" command to check an image file against a checksum

If we consider really supporting mtree I would take it from another perspective and use it as an informative/debug tool to diff images or layers. Basically to diff active vs passive vs recovery vs any other accessible container. Or even something to verify/create delta upgrades if we ever consider this possibility. However even in such cases we could have a look at higher level tools which are likely to relay on mtree too, like container-diff and similar pieces.

I had quite few experiences with container-diff. while it goes well for basic usage, it goes off very soon when comparing either big images and alikes. I'm not very keen into it. On luet at beginning that was used to compute the containers diff, but had to move away from it as didn't scaled on big images.

[Itxaka]

Mtree could be used to check if a system image matches the container checksum, and that's something we will have to provide (sooner, or later)

I dont know, for single files mtree is basically a xattr + sha512 + ls -la. If you want yoy can have a checksum of the artifact AND sign that checksum with cosign, so its tamper proof. Plus, doesnt luet already provide the checksum of the files in the metadata? And the metadata is already signed with cosign? So we basically got that covered already.

Where mtree shines is on a whole system or a list of files, where you can get all that stuff in a tracked file and check them afterwards. BUT currently due to limitations on the docker part, we are not even getting the xattrs, so its basically a dumb sha512 + filelist of every file in the system, which I dont think its very useful for anything really. If you got a checksum of the artifact then you are already verifying that the contents havent changed since you built it.

I dont know, what were the reasons given for it in the recent discussions? were any points arisen on how to use, what to verify, etc..? Or it was just "we want mtree" and to hell with details?

[mudler]

Mtree could be used to check if a system image matches the container checksum, and that's something we will have to provide (sooner, or later)

I dont know, for single files mtree is basically a xattr + sha512 + ls -la. If you want yoy can have a checksum of the artifact AND sign that checksum with cosign, so its tamper proof. Plus, doesnt luet already provide the checksum of the files in the metadata? And the metadata is already signed with cosign? So we basically got that covered already.

would cosign would give you the ability to check against an uncompressed container image?

Where mtree shines is on a whole system or a list of files, where you can get all that stuff in a tracked file and check them afterwards. BUT currently due to limitations on the docker part, we are not even getting the xattrs, so its basically a dumb sha512 + filelist of every file in the system, which I dont think its very useful for anything really. If you got a checksum of the artifact then you are already verifying that the contents havent changed since you built it.

Indeed, that's where we need it and Xattrs is not so important here. we want to verify that the files content are the same - extra permissions like xattrs could invalidate functioning, but we want to prevent tampering of the content here. Let's say that someone modifies manually the active image by adding extra files, or changing scripts - we ideally want to be able to verify that (after the image was pulled, and already verified by cosign).

About Xattrs, probably we should revisit where this limitation is coming from - afaict luet atm does not have that limitation and packages built with Xattrs bits are correctly pushed over, and test suites are in place in regard to that. Note the luet api lets you also create images from directories, docker, etc, preserving those extra bits.

[Itxaka]

would cosign would give you the ability to check against an uncompressed container image?

Well, in theory yes, but it would be extremely slowly.

But we dont generate uncompressed container images rigth? We generate and distribute OCI artifacts, if you want to extract it afterwards then that is cool for us, but the point of cos-toolkit is to build, ship and maintain cloud-init driven Linux derivatives based on container images so that is our output, isnt it? Everything we build, or push is an OCI artifact, so it makes sense to sign those OCI artifacts as the only source of truth.

About Xattrs, probably we should revisit where this limitation is coming from - afaict luet atm does not have that limitation and packages built with Xattrs bits are correctly pushed over, and test suites are in place in regard to that. Note the luet api lets you also create images from directories, docker, etc, preserving those extra bits.

Wasnt this part of buildX? the COPY stanza didnt preserve the xattrs or something like that? Its probably written down on one of the mtree cards from some time ago.

Let's say that someone modifies manually the active image by adding extra files, or changing scripts - we ideally want to be able to verify that (after the image was pulled, and already verified by cosign).

Well, after you download, verify and extract the artifact...why do we care? mtree wont help in there either as the verification would fail if stuff is added afterwards, but I dont see the advantage? Our verification can only cover from where we build the artifact to when we hand it over to an user, that its our security domain, to make sure it was not tampered between those steps and a simple sha512 can verify that. Whatever you do with the image afterwards fall waaay outside of our domain, hence if you need verification in that part, you have to generate yours, for example in https://github.com/c3os-io/c3os we are providing the artifacts for the base and they are being verified, but afterwards the artifacts that c3os generates (if any) should be signed by c3os, not us.

[davidcassany]

we should streamline it's approach offering mtree capabilities directly from the elemental-cli, for example (fictional example ahead):

This is nice. However I see little benefit on that, I struggle to see mtree validation as gate or boolean check (we have quite few things altering images once unpacked), I'd use more to get a notion of a distance between the deployed system and some reference image. So rather than use it to check integrity within our processes I'd use it as a way to manually check integrity on demand and result in some sort of report (diffed files or something similar).

Wasnt this part of buildX? the COPY stanza didnt preserve the xattrs or something like that? Its probably written down on one of the mtree cards from some time ago.

Xattrs are not included in containers by design, and it makes sense this way. If you think of regular container use cases xattrs such as SELinux labels do not belong to the container, they belong to the host security policies. The fact that our containers are the host is pretty much a corner case. IIRC docker stack has no support for xattr and there are no explicit plans to ever support something like that. In podman stack there are chances to port few xattr, but there is an explicit motivation not to include SELinux labels in there, its been discussed on github issues more than once and only a subset of xattrs is supported. Moreover there is another limitation to remember here, adding SELinux labels requires privileges and here we are blocked by docker stack once again, privileged docker builds are not possible (I think with buildx they are but I do not fully remember now). Podman stack allows to run privileged builds under some very specific configuration.

All that to say we should forget about trying to get xattrs into container images (specially for SELinux), as this will lead to a painful path including missing upstream features that were already denied by the community at some point. So it is pretty unlikely we ever consider including xattrs bits to check a current deployment against some container image.

[mudler]

would cosign would give you the ability to check against an uncompressed container image?

Well, in theory yes, but it would be extremely slowly.

But we dont generate uncompressed container images rigth? We generate and distribute OCI artifacts, if you want to extract it afterwards then that is cool for us, but the point of cos-toolkit is to build, ship and maintain cloud-init driven Linux derivatives based on container images so that is our output, isnt it? Everything we build, or push is an OCI artifact, so it makes sense to sign those OCI artifacts as the only source of truth.

Yes, but we do apply container images as part of the system, turning them into images. This process is not verifiable afterwards, without something on the lines of mtree imho.

About Xattrs, probably we should revisit where this limitation is coming from - afaict luet atm does not have that limitation and packages built with Xattrs bits are correctly pushed over, and test suites are in place in regard to that. Note the luet api lets you also create images from directories, docker, etc, preserving those extra bits.

Wasnt this part of buildX? the COPY stanza didnt preserve the xattrs or something like that? Its probably written down on one of the mtree cards from some time ago.

Yep, that disappeared from luet API by using directly containerd, I don't remember now specifically if it was a luet issue or not, but then the limitation would be only when using COPY keywords in a Dockerfile

Let's say that someone modifies manually the active image by adding extra files, or changing scripts - we ideally want to be able to verify that (after the image was pulled, and already verified by cosign).

Well, after you download, verify and extract the artifact...why do we care? mtree wont help in there either as the verification would fail if stuff is added afterwards, but I dont see the advantage? Our verification can only cover from where we build the artifact to when we hand it over to an user, that its our security domain, to make sure it was not tampered between those steps and a simple sha512 can verify that. Whatever you do with the image afterwards fall waaay outside of our domain, hence if you need verification in that part, you have to generate yours, for example in https://github.com/c3os-io/c3os we are providing the artifacts for the base and they are being verified, but afterwards the artifacts that c3os generates (if any) should be signed by c3os, not us.

sure, indeed I'm not speaking of us signing artifacts, but indeed a derivative. Let's take c3os an example, it should be good to go to generate both mtree and cosign signatures, in a way that then we can verify them easily with the CLI.

At this point I'd ping @agracey here as this came up already in previous conversations, please correct me if I am wrong. The capability that we want to achieve here is to be able to assess the booting image against a published one after upgrade/deployment happens. During upgrades this isn't a problem, as everything is handled by kubernetes, and we can have other means of verification like #1183 which actually gate updates on the host.

What we miss here is something to verify what we have deployed is matching a published OCI artifact, and as far as I can tell mtree is our best candidate for the solution here as gives enough means of flexibility in order to achieve that.

[Itxaka]

What we miss here is something to verify what we have deployed is matching a published OCI artifact, and as far as I can tell mtree is our best candidate for the solution here as gives enough means of flexibility in order to achieve that.

That sounds like a mess to be honest. we are comparing static files, to a running system with the entrypoint of cloud-config files. I can already see so many issues in there. Plus currently that its impossible with our current luet-mtree implementation.

[mudler]

What we miss here is something to verify what we have deployed is matching a published OCI artifact, and as far as I can tell mtree is our best candidate for the solution here as gives enough means of flexibility in order to achieve that.

That sounds like a mess to be honest. we are comparing static files, to a running system with the entrypoint of cloud-config files. I can already see so many issues in there. Plus currently that its impossible with our current luet-mtree implementation.

why? images are mounted RO and are supposed to not be modified, they should be verifiable as well in theory as those are not really moving pieces in a booting system. I know that there are some limitations with mtree - that we get away with few assumptions (like, some layouts folder needs to exists, and so on so forth) and that we can not guarantee the whole integrity of it ( extra permissions, etc. ) but I don't see this as something impossible to achieve - we do that practically already with upgrade channels, by verifying during extraction.

[Itxaka]

why? images are mounted RO and are supposed to not be modified, they should be verifiable as well in theory as those are not really moving pieces in a booting system. I know that there are some limitations with mtree - that we get away with few assumptions (like, some layouts folder needs to exists, and so on so forth) and that we can not guarantee the whole integrity of it ( extra permissions, etc. ) but I don't see this as something impossible to achieve - we do that practically already with upgrade channels, by verifying during extraction.

Well, for example, testing this with excludes and so on, mtree stops dead as soon as it encounters /proc, due to it making lstat to the files:

[2022-05-18 09:36:43] lstat /proc/1823/fd/7: no such file or directory

This is done on a deep, non-exported, non overridable, critical part of the mtree lib, which means we either push for this to be overridable upstream, or just re implement the whole mtree ourselves, change to another library and so on :/

Unfortunately, excludes are done AFTER mtree has walked the whole root path, so no easy way out of this. I dont think mtree was meant for this usage

At this point, looking at mtree code and seeing what the end goal is here, its easier to ship a .git folder with the artifacts LOL

[mudler]

why? images are mounted RO and are supposed to not be modified, they should be verifiable as well in theory as those are not really moving pieces in a booting system. I know that there are some limitations with mtree - that we get away with few assumptions (like, some layouts folder needs to exists, and so on so forth) and that we can not guarantee the whole integrity of it ( extra permissions, etc. ) but I don't see this as something impossible to achieve - we do that practically already with upgrade channels, by verifying during extraction.

Well, for example, testing this with excludes and so on, mtree stops dead as soon as it encounters /proc, due to it making lstat to the files:

[2022-05-18 09:36:43] lstat /proc/1823/fd/7: no such file or directory

Where did you tested this? on a mounted system? for a quick test should mount active.img (or the one you target for verification) on a path and generate a checksum there, not from the current /

[Itxaka]

Where did you tested this? on a mounted system? for a quick test should mount active.img (or the one you target for verification) on a path and generate a checksum there, not from the current /

Mounted system of course. Checking the mtree values of the img makes no sense to me? We verified the contents already on extracting the artifact (SIGNED with cosign) to that .img file and those confirmed that the container (and its files inside) were not tampered with. What purpose serves rechecking that again?

Maybe im not understanding the point of this, but rechecking a RO img file which we already verified on extraction? why not just sha the img and write it down on a paper/push it to somewhere? If the point is checking that it has ot be tampered with, why check the files inside? Its much easier to check the whole file, isnt it?

[mudler]

Where did you tested this? on a mounted system? for a quick test should mount active.img (or the one you target for verification) on a path and generate a checksum there, not from the current /

Mounted system of course. Checking the mtree values of the img makes no sense to me? We verified the contents already on extracting the artifact (SIGNED with cosign) to that .img file and those confirmed that the container (and its files inside) were not tampered with. What purpose serves rechecking that again?

there is a value in all of this, as you can then check what's the image you have deployed, and what is the remote one. Suppose you boot in debug mode, which actually disables RO on the images. at that point you could likely edit something and that image becomes invalidated. This mechanisms would allow you to tell that the image deployed is the same one that is being published.

This could help along the way in so many other process, among:

• debugging
• troubleshooting
• verifying that corruption didn't happened at fs level

We don't ship image files, but we create them on the fly during the upgrade process, so it is not possible for a derivative to check against a pre-built image file unless they build the img file and push it, consuming it in place of the upgrade process - but that would break completely our mission to be a single image container - at that point you have a split.

Maybe im not understanding the point of this, but rechecking a RO img file which we already verified on extraction? why not just sha the img and write it down on a paper/push it to somewhere? If the point is checking that it has ot be tampered with, why check the files inside? Its much easier to check the whole file, isnt it?

yes, that is exactly what mtree is supposed to do indeed - takes a checksum of something in a custom way. The fact that we don't publish .img files that aren't directly consumed makes this convoluted. On the other hand, if we start shipping .img file, you will loose ALL the benefits of having container images bootable, such as:

• tight integration with container ecosystem technologies already there (security scanners, admission controls, proxies, w/e hooks to a container image)
• one single image for all - currently we source from a container image to avoid double headed deployments. No snowflakes because the image is one, and that's the only thing you (derivative developer) care about
• Convoluted setups - Having two images then will make some things more simple (like checksumming, etc) but other things far more complex. Extra tooling, and helpers, and documentation would be required in order to understand what takes to convert a container image to an image file. At that point, we would loose completely our mission focus

[Itxaka]

I dont know man, I just dont see it. It also helps that nobody else does it, which points me to thinking that it may not be a good idea, maintainable, etc...

But sure, if its required, lets go with it. Should probably integrate it on elemental asap.

[mudler]

Let's wait for @agracey's feedback. If we don't have such requirements, I'm totally fine dropping it, less to maintain.

[agracey]

We have three asks from partners that I know of:

• Secure Boot
• Measured Boot
• FIDO Device Onboarding*

I don't know much about mtree itself or the deeper technical requirements of these three asks. What I can do is get a call together with you and the partners that are asking about this. They seem to have a really good understanding of what's needed.

From the exploration that I did, one of the places that we can give some real value is that the .img we are booting into can have a known value and be signed by the partner's private key. This gives the ability to "measure" some components that a more traditional OS struggles with.

*I don't think FDO is relevant to this... but I haven't read the spec yet to know so I'm listing it anyways.

[Itxaka]

This above has nothing to do with mtree I think? If that is what is required then I dont see how mtree works it into that.

A different thing is if we want to provide an extra value in either providing the mtree values and comparing them to the img contents, in which case I would totally understand, but would make me drop the current mtree implementation and reimplement it on elemental side. How about creating oci artifacts for the mtree data so we no longer have to attach them to the metadata? How about...using cosign to sign those mtree artifacts/containers ?

there is a whole world out there and we are just scratching the surface :P

[mudler]

We have three asks from partners that I know of:

* Secure Boot

* Measured Boot

* FIDO Device Onboarding*

Ok, this looks quite distant from our conversations. What I was asked for is that if we have a mechanism in place to check back the image we are booting from after install/upgrade/deploy. in this case mtree can play a role here.

From the exploration that I did, one of the places that we can give some real value is that the .img we are booting into can have a known value and be signed by the partner's private key. This gives the ability to "measure" some components that a more traditional OS struggles with.

The .img files are generated during the upgrade process, so no signature are attached to any of them. This is exactly the core of the conversation.

mtree is a facility that would allow to check against the img content checksum after upgrade/deployments, as our only mean to verify that is by verifying against a checksum of the container image content. That a checksum can be signed, that's something that can be added on top to verify the provenience of the sha.

[agracey]

Ok, then I think I'm not understanding enough about the question to answer well. It might be worth chatting in a huddle tomorrow.

[agracey]

I will say that if it's not needed currently, then I wouldn't worry about it and the simplicity in maintenance has more value.

[mudler]

after discussing with @agracey and @kkaempf we have agreed to try to find out if there are such requirements to keep mtree hashing functionality on our codebase.

For the time being - as we don't know yet if this is formally part of requirements - my suggestion is to keep it as it is and just keep it disabled by default.

This is just to avoid us going back at this if we get asked about it ever again. I want to be really sure we don't need any of this verification mechanism before dropping it entirely.

It would be far more expensive to drop it now and re-add it back later on, as we could even take design choices which could actually make it more painful to re-integrate it later, or maybe skipping tests which would break it entirely.

[frelon]

Technically mtree is now dropped, closing.