As a user I want to differentiate between different nature of images

[ipanova]

Is your feature request related to a problem? Please describe.
As a user when I inspect a manifest through pulp API I can tell whether the image is :

• bootable
• application based
• flatpak
• etc

Describe the solution you'd like
Expose Image Annotations/Labels based on which a user would be able to say/search what kid of image it is.
For example flatpak image contains org.flatpak.ref label. This label identifies the image as Flatpak.

Additional context
We might need to inspect both Label and Annotations depending on the Docker/Oci image format.
In OCI spec labels are superseded by annotations but docker format still uses labels.

Note: Even if the image is derived, the labels/annotations are passed down to the child image.

[ipanova]

Docker images can use annotations format but they will be present in the config.Labels directive, e.g.
They will not have `annotations on the manifest because it is not part of their spec.

$ skopeo inspect docker://docker.io/bitnami/redis:latest --tls-verify=false --raw  --config |jq .config.Labels
{
  "com.vmware.cp.artifact.flavor": "sha256:1e1b4657a77f0d47e9220f0c37b9bf7802581b93214fff7d1bd2364c8bf22e8e",
  "org.opencontainers.image.base.name": "docker.io/bitnami/minideb:bullseye",
  "org.opencontainers.image.created": "2024-01-22T10:33:28Z",
  "org.opencontainers.image.description": "Application packaged by VMware, Inc",
  "org.opencontainers.image.licenses": "Apache-2.0",
  "org.opencontainers.image.ref.name": "7.2.4-debian-11-r3",
  "org.opencontainers.image.title": "redis",
  "org.opencontainers.image.vendor": "VMware, Inc.",
  "org.opencontainers.image.version": "7.2.4"
}

OCI images can use annotations but not config.Labels

$ skopeo inspect docker://cgr.dev/chainguard/wolfi-base --tls-verify=false --raw  |jq .annotations
{
  "org.opencontainers.image.authors": "Chainguard Team https://www.chainguard.dev/",
  "org.opencontainers.image.source": "https://github.com/chainguard-images/images/tree/main/images/wolfi-base",
  "org.opencontainers.image.url": "https://edu.chainguard.dev/chainguard/chainguard-images/reference/wolfi-base/"
}

But some(most?) OCI images still can use config.Labels but not annotations skopeo inspect docker://registry.fedoraproject.org/0ad:latest --tls-verify=false --raw --config |jq

And probably some OCI images can have both populated :)

Annotations are a fairly new directive , not many images leverage them and we should be prepared to look in both places.
There is clear benefit to leverage annotations because one can just download/parse the metadata and not the image itself.

[ipanova]

Note: Both Labels/annotations are indeed passed down to the child image(if unspecified). If specified, they will be overridden.

[lubosmj]

Furthermore, we are required to identify whether a manifest list is bootable or not. This can be done by parsing the first available listed manifest or a manifest with the amd64 architecture.

The ultimate goal is to enable users to filter manifests by their nature from the REST API. With @ipanova, we have considered two design options:

1. Adding two boolean fields to the model/serializer: is_bootable and is_flatpak. Pros: easy to implement and maintain, good performance while filtering content. Cons: we may end up having a bunch of fields per every manifest's nature (e.g., is_future_type).
2. Adding a labels json field to the manifest/serializer model. On the viewset, we define a custom search filter that will enable users to filter content by specific values stored inside the object's json field. Pros: robust and resilient to newly emerged natures, everything accessible from one place, visible in the API. Cons: performance implications of exposing a complex search filter (https://stackoverflow.com/questions/58807182/how-to-apply-filters-on-a-posgres-jsonfield-in-django-rest-framework, https://www.django-rest-framework.org/api-guide/filtering/#searchfilter, https://github.com/pulp/pulp_ansible/blob/6335fe32e699ed7686483cffc58e988c8b9c040f/pulp_ansible/app/galaxy/v3/filters.py#L39).

Unfortunately, we will need to extract data from config blobs' labels in both cases and place it to the manifest model. We cannot ensure that manifests' annotations are the only source of truth.

It might be worth implementing both of the suggested options (they are complementary, synergistic), so that the users will be able to filter manifests by is_bootable while at the same time they can preview what other labels were used.

http :5001/pulp/api/v3/content/container/manifests?is_bootable=True

{
  "count": 123,
  "next": "http://api.example.org/accounts/?offset=400&limit=100",
  "previous": "http://api.example.org/accounts/?offset=200&limit=100",
  "results": [
    {
      "pulp_href": "http://example.com",
      "pulp_created": "2019-08-24T14:15:22Z",
      "artifact": "http://example.com",
      "digest": "string",
      "schema_version": 0,
      "media_type": "string",
      "listed_manifests": [
        "http://example.com"
      ],
      "is_bootable": True,     # NEW
      "config_blob": "http://example.com",
      "labels": {     # NEW
        "key": "value"
      },
      "blobs": [
        "http://example.com"
      ]
    }
  ]
}

[ipanova]

I think, I like the idea exposing both the boolean field and the labels and here is why:

• labels is a directive of image manifest, extracting them from it and exposing onto a manifest list feels a bit more than hacky, especially when there can be different set of labels on each child manifest.

Besides, labels, we should also consider exposing annotations.

[ianballou]

From a Katello use case this sounds perfect. We can immediately use this to display pertinent information about the manifests in the UI.

I think from our standpoint we're most likely to use is_flatpak and is_bootable , so +1 to having the booleans.

[ianballou]

I'm going to push for having Katello integrate with this as soon as possible. I'll need to give time for users to pre-migrate in Katello 4.13 and 4.14, but after that we can unleash this setting (plus we want to rework our UI at the same time).

I have a request: can we get information about how long the pre-migration will take on a well-loaded Pulp environment? Perhaps a good start would be an environment that has all of the OpenStack-related container repositories synced.
We'll also need to provide information about how the pre-migration might affect Pulp (and Katello) performance while the migration is running. Will sync tasks be blocked for long, for example?

[ipanova]

@ianballou
sync and any other api calls won't be blocked because this django-admin command works directly with the DB.
I think you already have an instance of calling such command in the post-install phase, don't you? It's this one https://github.com/pulp/pulp_container/blob/main/pulp_container/app/management/commands/container-repair-media-type.py

[ianballou]

I think you already have an instance of calling such command in the post-install phase, don't you?

We do, but we block the upgrade on it in that case, so it's a little different.

[ianballou]

Thanks for mentioning that though, I forgot that foreman-maintain is a good place to include the pre-migration. That's where we run the media repair script.

[ipanova]

so you would be calling this one in foreman-maintain as well and it won't be blocking anything, correct?

[ipanova]

Would you still want to setup a testing env to see how long the command will take? It will not block the upgrade and api calls/tasks. The only place where you will see the overhead is DB and IO

[ipanova]

As agreed per call would be good to provide some numbers so they can be shared in the user docs