why each image must only have one layer lines, but not a multiple-root layer tree?

[XenoAmess]

Is the reason just be "docker did it this way so we remain it"?
Or there be some benificials in not allowing multiple layer roots in same image...

[sudo-bmitch]

I'm not understanding the question. Could you phrase it differently, perhaps with an example?

[cyphar]

I think they're asking why we have a linear stack of layers, rather than letting you define subtrees.

@XenoAmess

The short version is that in general overlay filesystems work really well (and are designed for) when you have a set of directories that get stacked on top of each other. Docker based their format on AUFS (which we then inherited in OCI) but even modern overlay filesystems (like overlayfs) still work best with this model -- in fact, some implementations of stacking filesystems using snapshotting filesystems would struggle to work with anything else.

I wrote about this issue a long time ago, and I have had prototypes on how to fix it, but it's a fairly tricky problem to solve. tar archives suck for a lot of reasons, but it's quite difficult to change something this fundamental with the OCI image-spec -- distributions are very reticent to accept new features that make their lives harder, even if it would be a general benefit for everyone.

Maybe I'll take another crack at it one day.

[XenoAmess]

I think they're asking why we have a linear stack of layers, rather than letting you define subtrees.

@cyphar Yes indeed.

It would be a general benefit for everyone.

It would save quite lot of people for 95%+ bandwidth in app image upgration, if allowing subtrees.

I'm not understanding the question. Could you phrase it differently, perhaps with an example?

@sudo-bmitch I thought this be a common question as actually everybody suffers from it, thus need no much explains, maybe I was wrong.
I would prepare some documents about the problem, and how it troubles me(and others), and why allowing subtrees can help. well I will finish it this week and put it here...

[cyphar]

There already is a document about it, we wrote it back in 2019. https://github.com/project-machine/puzzlefs is one project trying to solve this problem.

[XenoAmess]

There already is a document about it, we wrote it back in 2019. https://github.com/project-machine/puzzlefs is one project trying to solve this problem.

seems it be still quite prototype...
would look forward for it in kernel&k8s

[sudo-bmitch]

I'm not understanding the question. Could you phrase it differently, perhaps with an example?

I thought this be a common question as actually everybody suffers from it, thus need no much explains, maybe I was wrong.

It was more that the question didn't quite parse for me. One layer lines could refer to the json encoding of layer descriptors. OCI works as a trailing spec, so I would want to see existing implementations of a tree structure. I'm also not sure how existing tooling would migrate to that design.

Could the same file be modified in two separate tree branches and there's some kind of merging logic of all the branches, allowing someone to assemble an image from multiple other images. I've seen that suggested many times, and the implementation logic quickly fails when trying to define the merging logic, particularly when the selected images conflict with each other (how do you merge the debian package manager state from multiple debian images each with different packages, and then throw in an alpine image that replaces lots of utilities in /usr/bin). While it's easy to define a happy path with an example (Nix being a common reference), OCI would need to define all of the edge cases and unhappy path resolutions.

If each sub tree is related to the filesystem structure, so one file can only be modified in one branch, that would imply that the current layered logic of building images would change to creating a lot of branches per step in the build process. The implementation I can quickly come up with becomes fractal, with a potentially exponential growth of subtrees for each step in the build process. That would encounter scaling concerns from generic build tooling.

For alternatives to tar, there's been work on that. Some folks are pushing for erofs, squashfs, and dm-verity. But as bad as tar is, there's a lot of tooling in the ecosystem built up around it, making it very portable and interoperable. So replacements need to not only show a single working implementation, but a library ecosystem that enables other tooling to adopt it. Even something like adding support for blake3 as a digest algorithm hits resistance because it's not yet part of the Go standard library.

[cyphar]

@sudo-bmitch FWIW, my main issue with dm-verity, squashfs, and erofs is that there is no out-of-kernel parsing library for them, and they are not designed to be parsed (or generated!) by userspace tools. (squashfs somewhat gets points in this department, but it then immediately loses them because you need to shell out to the squashfs binaries to do anything with it.)

I still think a minimal CAS that can be parsed using standard tools with a well-defined form and eventually be kernel mounted (like puzzlefs) is preferable, but I there is undoubtedly a lot of extra work to do. I don't necessarily agree that tar is somewhat acceptable "because it's standard" -- our usage of tar is quite non-standard, and tar is very poorly standardised (yes, there is ustar and PAX but tar is generally a complete minefield of compatibility mishaps). The current situation mainly only works okay because the Go standard library happens to provide a per-entry parsing library for tar -- if container runtimes hadn't used Go or Go didn't include that, we probably would not still be using tar. And even today there are all sorts of compatibility issues with Go-generated tar archives and ctime/atime and some other PAX extensions...

[XenoAmess]

OCI works as a trailing spec, so I would want to see existing implementations of a tree structure.

seems I'd better write some more docs for my design first. will do it in this weekend.

though I have only experience for handling the image packaging/repackaging/compressing/transforming, but no knowledge in runtime fs systems, so maybe the design I bring is not quite suitable. whatever, I accept any feedback/advice then.

if container runtimes hadn't used Go or Go didn't include that, we probably would not still be using tar. And even today there are all sorts of compatibility issues with Go-generated tar archives and ctime/atime and some other PAX extensions...

well I don't really think Go matters, or at least, matters that much. Go is just the lucky impl language that some people pick for some tools(means mainly docker here), and maybe it actually be a wrong choice(IMO)... designing a protoco basing on what be in Go's stupid poor standard lib doesn't sounds quite good/correct...

[hsiangkao]

@sudo-bmitch FWIW, my main issue with dm-verity, squashfs, and erofs is that there is no out-of-kernel parsing library for them, and they are not designed to be parsed (or generated!) by userspace tools. (squashfs somewhat gets points in this department, but it then immediately loses them because you need to shell out to the squashfs binaries to do anything with it.)

i'm not sure why erofs and squashfs don't have out-of-kernel parsing library, erofs-utils has liberofs and squashfs also has squashfs-tools-ng which aims to provide a userspace library for squashfs.

even ext4 has tar2ext4 and dm-verity also has go library implementation here

I still think a minimal CAS that can be parsed using standard tools with a well-defined form and eventually be kernel mounted (like puzzlefs) is preferable, but I there is undoubtedly a lot of extra work to do.

I hope to have an honest conversation, but I've reiterated in previous threads that EROFS already supports CAS since Linux 6.1 (but all CAS chunkes has to be landed in limited blobs), it just doesn't use typical CDC or its varient fastCDC, but if you really want me to implement FastCDC in erofs-utils, that is doable and no need to land any new kernel change.

If you'd like to keep each chunk as a seperate file, I've already explained the reason in this reply why it's impossible in the kernel (if you don't trust me you could open a thread in the -fsdevel mailing list to discuss breaking files into variable chunks and store each chunk into a file and then how to handle mmap page fault in this way):
project-machine/puzzlefs#114 (comment)
In short, puzzlefs kernel implementation has never implemented mmap support at all (so it's impossible to launch real containers), and it's very hard to open unfixed files (even a single file due to the context limitation) in the kernel page fault context (please see what Dave Chinner said in the comment I mentioned).
That is quite different from what composefs does because composefs only open files out of page fault contexts which is what overlayfs behaves.

In other words, if there had been a feasible way to implement this as a kernel filesystem, I would have done it a long time ago. However, it has been quite controversial on my side, and I believe I’m not the only one who thinks like this.

[XenoAmess]

I still think things may not be so complex, but I don't know if I think it too easiely.

1. as we do accept .wh. files, I think it be not bad to add another grammar as .import. files, for example .import.abcde.txt
2. in the .import.abcde.txt file we describe which file/folder is imported from which block
   for example,

/root/data/abcde.txt from sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a538888

3. describe the blocks into image manifest

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7",
    "size": 7023
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0",
      "size": 32654
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b",
      "size": 16724
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736",
      "size": 73109
    }
  ],
//here
"blocks": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a538888",
      "size": 12345
    }
],
//here
  "subject": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270",
    "size": 7682
  },
  "annotations": {
    "com.example.key1": "value1",
    "com.example.key2": "value2"
  }
}

4. store block using a samilar way to the layers (as it can also be seen as some file patches)

In this way, we give opertunity to image makers to share common libs into blocks, and share them between multiple images's multiple layers.

[tianon]

While an interesting idea, a change like that would require updates to not only every single registry server (to ensure blocks are treated properly as children of the manifest), but also every single runtime client to handle these new files, and the failure modes if they aren't updated before trying to use the new functionality are pretty nasty.

From the registry, the failure mode might be that garbage collection simply deletes the blocks because it doesn't recognize them as part of the manifest (making it impossible for updated runtimes to pull/use those blobs).

The failure mode in clients is that they'll simply not do anything with these new files, which means the resulting image will be "missing" the files specified by these blocks entirely.

[XenoAmess]

While an interesting idea, a change like that would require updates to not only every single registry server (to ensure blocks are treated properly as children of the manifest), but also every single runtime client to handle these new files, and the failure modes if they aren't updated before trying to use the new functionality are pretty nasty.

From the registry, the failure mode might be that garbage collection simply deletes the blocks because it doesn't recognize them as part of the manifest (making it impossible for updated runtimes to pull/use those blobs).

The failure mode in clients is that they'll simply not do anything with these new files, which means the resulting image will be "missing" the files specified by these blocks entirely.

I thought they shall implemented json schema to prevent image manifest json containing unknown block field, thus make the image noticely errored during the push/pull, then make people notice they shall upgrade their client/server for these new image additions.
maybe I be wrong.
if things be in that case, "schemaVersion": 2 might have to inc.

[sudo-bmitch]

I still don't understand what the issue is you are attempting to solve. And questions asking for clarification above have gone ignored. How would the tgz "blocks" help image distribution over the existing tgz layers?

[cyphar]

@tianon It really is quite unfortunate that (as a spec) we added annotations, but we didn't provide a way for extensions to create their own new reference mechanisms with some kind of references or weak-pointer-like reference system in the base spec so that legacy systems would at least not garbage-collect the wrong data when we want to add extensions.

This is basically the reason we ended up with the hacks necessary for OCI artefacts and the similar hacks that stuff like puzzlefs uses.

[hsiangkao]

Another thing I have to mention that is while per-chunk CDC seems efficient for space saving in principle, but that is another illusion if you consider treat each variable-sized chunk as individual files on some real on-disk fs.

Because the typical CDC algorithm only cuts data into chunks which have best-effort block-aligned (e.g. from 16k to 64k), but in practice all chunks will not be aligned with the EXT4/XFS/... filesystem block (and if considering compression it will be even worse), so the unaligned chunks will waste extra space in practice (and the smaller cut point size you choose, and more space will be wasted), and since those chunks are also non-block-aligned, reflink or page cache sharing is also impossible.

EROFS supports a varient CDC but typically divide data into chunks which will compressed data into block-aligned chunks (e.g. currently from 4k to 1m compressed chunks) and already do compressed data deduplication, so it will never waste the underlayfs EXT4/XFS space because the chunk size is always predictable.

Since EROFS deduplicated compressed data are typically all block-aligned, you could always use XFS/btrfs reflink to form container images from the block-aligned blobs in the local global content-addressable-storage chunk pools and with this approach there is no need to open too many chunk files.