OCI based IPA image distribution

[diconico07]

This discussion aims to clarify what we want to effectively do to store IPA images as OCI artifacts before doing a new design doc.
This is the continuation of the discussions in the metal3-io/metal3-docs#712 PR

Here is a quick summary of the initial design proposal and the discussions around it, I skipped the details specific to the proposal on purpose to drive this discussion around how we should store/distribute :

• Store the IPA image as a regular docker/OCI image, IPA downloader could then extract the kernel from the image and use the rootfs as initrd.
• @dtantsur pointed that Ironic upstream tooling to build IPA image outputs kernel and initramfs already and that it would be better to directly distribute those as OCI artifact rather than doing this kind of repackaging, especially since Ironic has some OCI fetching capabilities that could remove the need for IPA downloader at some point.

I then agreed on this subject but I believe we want to have the IPA image distributed under a single OCI tag, to do so we would need to define a manifest structure to hold both the kernel and initramfs within it (and I could even argue we'd like the EFI bootloader here as well).
However I also think we need alignment with Ironic on this subject, so that Ironic could at some point also consume that single manifest. During community meeting, @dtantsur said he could help conveying the idea to Ironic PTG.

My renewed proposal would be to have the tag pointing to a regular OCI index, with each entry in the index pointing to an arch specific manifest (standard multi-arch support), then the architecture specific manifest would contain annotated pointers to both the kernel and initramfs, allowing a compatible fetcher to distinguish the kernel from initramfs (or even ISO). The manifest would be compliant with the ORAS specifications.
The artifactType could be something akin to application/vnd.ironic.image
The annotation could be the standard org.opencontainers.image.title with a well known filename, or a custom annotation like org.opendev.openstack.ironic.image-role (with values like kernel, initrd or iso).

Here is a manifest example (checksums are placeholders) with both the image.title and custom annotation:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "artifactType": "application/vnd.ironic.image",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "size": 2
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:d2a84f4b8b650937ec8f73cd8be2c74add5a911ba64df27458ed8229da804a26",
      "size": 12,
      "annotations": {
        "org.opencontainers.image.title": "ironic-python-agent.kernel",
        "org.opendev.openstack.ironic.ki-role": "kernel"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:d2a84f4b8b650937ec8f73cd8be2c74add5a911ba64df27458ed8229da804a26",
      "size": 12,
      "annotations": {
        "org.opencontainers.image.title": "ironic-python-agent.initramfs",
        "org.opendev.openstack.ironic.ki-role": "initrd"
      }
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2023-08-03T00:21:51Z"
  }
}

[juliakreger]

Hi, Ironic maintainer here (and the person who drove Ironic's OCI url support).

I personally like the basic idea to use basic idea presented here, but I see tensions between use models which we need to figure out. On the developer/process engineer the mental idea would be to just point to the higher level tag for auto-identification and download of the needful artifacts, however if I put on the security hat many customers and business tend to wear coupled with the context of some recent CVEs I've worked, the requirement of just trusting the remote server and images to use to a tag (even an immutable tag) becomes a security risk which needs to be managed. Specifically on the consuming side I need to know and assure the checksums match what I'm expecting. Meaning the high security operational side consumption model is very different than the development side convenience of use model. The key takeaway in such a use model is that the information security side of folks will expect digest references with individual mappings That doesn't really inherently conflict with what is proposed, just actual operational usage will vary and quite possibly operators may just be willing or need to point at the specific digest representing the json structure for the end artifacts. As a note, we're not presently modeled that way, we assume any digest manifest with multiple layers is an actual container, because dimension wise, the OCI model can head in basically either direction. That is not insurmountable, but declaring a specific artifact or media type in the schema would be the needful to label the digest.

On the side of getting other artifacts, I do really like the idea of extending the retrieval logic to support URLs instead of just expecting files as referenced by path. Operational security wise, I think the same concern appears though and I think the use pattern required would need to boil down to explicit structural artifact digests being pointed at in a 1:1 mapping to maintain the service side configuration and control of the end destination, because otherwise somehow the expected locations need to be modeled and stored in the OCI server which seems like the door is being opened to a mix of attacks and also sort of ties the operators hands if they need to massage the structure because all trust gets placed in the OCI server yet their infosec team is going to demand a digest reference as any starting point and then they would have to independently be able to audit the oci server's contents. Not saying impossible, just the structural barrier is human context based as well.

I'm not sure how comfortable other ironic cores will be with invoking the file download service to retrieve items on service start, but its worth a discussion. Hopefully my minimally caffeinated mind this morning is getting this across in a way which makes sense! Please let me know if you have any questions or want to discuss.

In the mean time, it looks like @dtantsur has started to raise the discussion on Ironic's PTG etherpad. I'm going to go put some thoughts in there as well.

[diconico07]

Thanks for the thoughts.
On the security side, I agree that most security minded users will either want to use direct digest, or at least signature verification, but that's another layer of complexity and I prefer incremental changes, even if it means security minded persons here would have to take extra steps to either use the current way of giving an OCI URI, or if there is a way to give an OCI URI with digest pointing to a manifest (the Content-Type could be a way to tell which is which).

For other artifacts, I guess you are talking about the EFI bootloader I mentioned. For this one specifically, I would tend to consider it part of the overall image because of Secure Boot, as it contains the shim, it becomes deeply linked to the image you are trying to boot (you won't be able to boot a CentOS or openSUSE image with a Debian shim). This would be a whole other can of worm, and I only mentioned it here to say it could be expanded.

[dtantsur]

Pardon if I'm using wrong words, but we can have a manifest pointing at other manifests, right? So in theory we can have both: ability to point at digests separately AND ability to point at one digest and say "take from here", no?

[juliakreger]

So @dtantsur is sort of mentally pointing to what I was thinking as the start of heavy administrative burden, but its not a wrong model. While not exactly oras compatible, its how things like podman's multiarch machine os images are distributed. In the model, you can nest manifest so the outer most manifest can be referred to utilizing the digest value, and for anything under that changing, requires the outer most digest to be changed because each layer down is referred to using the sha256 hash digest value, so you do get incremental layers of security beyond that to guard the artifacts, but it sort of goes to where do you want to manage the burden or not.

The actual return content type is likely, at least, I suspect ought not to be trusted, the inner values are the key because they are also guarded by the digest value. The outer most manifest gets referred to via it's digest value which matches it's digest checksum value of the contents and that maps to the next level of structures. Its a model where you can't change any single value, you have to change everything that rolls up in the manifest structure.

@diconico07 Going back to your comment about EFI loaders, doing efi loaders by deploy ramdisk is not something we support today. It would require a distinct feature. Most operators have a fixed "deployment ramdisk" and while they can roll the needful in to the ISO, most operators we deal with standardize on one OS. That OS doesn't have to match the OS deployed either, since we don't download and place them. They are purely used for the ramdisk side of things to facilitate the deployment. Requested images to be deployed should already have all of the needful artifacts for them to boot in a disk image. In other words, we should tease the needful and requirement apart. Hope that makes sense!

And P.S. Dmitry, you were not using the wrong words. At least from my point of view.