Add OCI support to manifest subcommand #3990

jedevc · 2023-01-27T12:46:49Z

- What I did

Introduced support for OCI images to the docker manifest command.

- How I did it

https://github.com/distribution/distribution/ already support oci schemas, so all we need to do is connect into it. A few minor changes to the on-disk storage format are also required to correct whitespace issues 😢

This is incredibly not beautiful set of patches, but given docker manifest doesn't see active development, I'm not sure it's worth refactoring too much of the structure of the code.

Ideally, as a follow-up I'd like to add support to merge manifest lists together - this is proving a little tricky though.

- How to verify it

Use docker manifest with an OCI image!

- Description for the changelog

Add support for OCI images to docker manifest

- A picture of a cute animal (not mandatory but encouraged)

cc @thaJeztah @crazy-max

cli/command/manifest/util.go

Signed-off-by: Justin Chadwell <me@jedevc.com>

This behavior should not break any more use cases than before. Previously, if the mismatch occured, we would actually push a manifest that we then never referred to in the manifest list! If this was done in a new repository, the command would fail with an obscure error from the registry - the content wouldn't exist with the descriptor we expect it to. Signed-off-by: Justin Chadwell <me@jedevc.com>

This prevents us needing to attempt to reconstruct the exact indentation registry side, which is not canonical - so may differ. Signed-off-by: Justin Chadwell <me@jedevc.com>

jedevc · 2023-01-27T16:03:23Z

I've started working on manifest merging: it's a bit trickier, we need some more logic in the store, but it works 🎉

I won't put it into this PR, since it's some more substantial changes, but here's the initial sketch: jedevc/cli@manifest-oci...jedevc:cli:manifest-merging.

thaJeztah · 2023-01-27T16:36:50Z

cli/command/manifest/testdata/inspect-annotate.golden

@@ -14,6 +14,7 @@
 			"variant": "v7"
 		}
 	},
+	"Raw": "ewogICAic2NoZW1hVmVyc2lvbiI6IDIsCiAgICJtZWRpYVR5cGUiOiAiYXBwbGljYXRpb24vdm5kLmRvY2tlci5kaXN0cmlidXRpb24ubWFuaWZlc3QudjIranNvbiIsCiAgICJjb25maWciOiB7CiAgICAgICJtZWRpYVR5cGUiOiAiYXBwbGljYXRpb24vdm5kLmRvY2tlci5jb250YWluZXIuaW1hZ2UudjEranNvbiIsCiAgICAgICJzaXplIjogMTUyMCwKICAgICAgImRpZ2VzdCI6ICJzaGEyNTY6NzMyOGY2ZjhiNDE4OTA1OTc1NzVjYmFhZGM4ODRlNzM4NmFlMGFjYzUzYjc0NzQwMWViY2U1Y2YwZDYyNDU2MCIKICAgfSwKICAgImxheWVycyI6IFsKICAgICAgewogICAgICAgICAibWVkaWFUeXBlIjogImFwcGxpY2F0aW9uL3ZuZC5kb2NrZXIuaW1hZ2Uucm9vdGZzLmRpZmYudGFyLmd6aXAiLAogICAgICAgICAic2l6ZSI6IDE5OTA0MDIsCiAgICAgICAgICJkaWdlc3QiOiAic2hhMjU2Ojg4Mjg2ZjQxNTMwZTkzZGZmZDRiOTY0ZTFkYjIyY2U0OTM5ZmZmYTRhNGM2NjVkYWI4NTkxZmJhYjAzZDQ5MjYiCiAgICAgIH0KICAgXQp9",


Silly question; would JSON-in-JSON (raw data, but as string) work? I realise this data is not primarily intended for users to read, but having something to hold-onto when looking at it could still be useful (instead of having to decode)

Yeah, so we still have the content from the registry in the file as a sub-object, since deprecating that would probably be way more work 😢

As long as we have that, and Raw just duplicates that content full with indentation, I'm happy to have it be unreadable base64 😄 Especially as the content fed to the hashing algorithm isn't a string, it's a bytes array, keeping in as close format to the original is easier IMO.

No super strong preferences here, though I admit I'm not a fan of JSON-in-JSON 👀

Still some discussion happening on base64 vs JSON-in-JSON 😅

After @neersighted describing a preference for base64 here, I'm convinced -- the data is still available, and the Go struct should mostly match anyways for users. Additionally, jq has a base64 decoder built-in, so it's still very very accessible (~ jq '.Raw | @base64d | fromjson').

base64 is great 👍

thaJeztah · 2023-01-30T12:21:21Z

cli/manifest/types/types.go


 	// SchemaV2Manifest is used for inspection
-	// TODO: Deprecate this and store manifest blobs


LOL, so there even was a TODO for this (added in 1fd2d66 / #1156)

Looks like there even was a PR that did so, but got closed? 🤔 #1144

thaJeztah

LGTM

neersighted

LGTM, I actually prefer the base64 encoding as the it makes it more obvious this is an opaque blob.

thaJeztah · 2023-01-30T19:25:35Z

cli/command/manifest/push.go

+			dt, err = json.MarshalIndent(imageManifest.OCIManifest, "", "  ")
+			if err != nil {
+				return mountRequest{}, err
+			}


It just occurred to me; as this is "net new"; should we error in this case (no raw data present) instead of re-constructing the manifest data?

I think I prefer the symmetry with the manifest v2 -- or at the very least, if we reconstruct for one and not the other, we should make it more obvious why that happens in the code.

Agreed, I think the symmetry makes sense.

I'm curious though - here I've used 2-spaces as "the default". In hindsight, 3 spaces might make more sense? BuildKit did used to produce 3-space OCI images, so maybe it makes more sense to use the same 3 spaces as for schema2?

I'm curious though - here I've used 2-spaces as "the default". In hindsight, 3 spaces might make more sense? BuildKit did used to produce 3-space OCI images, so maybe it makes more sense to use the same 3 spaces as for schema2?

I guess there's no "good" answer for that one. Quick thoughts;

In practice, we should never® hit this branch; existing ~/.docker/manifests/.. JSON files would not have OCI manifests, and new files will have the Raw field propagated

If we want to keep symmetry; we could also consider going the reverse path; for existing files that have the "struct", but don't have the Raw field propagated;

add the Raw data (based on the struct), using the "old" formatting (3 spaces)

or (worst case); try 3 spaces, and if that doesn't match the digest, try 2 spaces, and otherwise bail out?

deprecate this code-path (only use .Raw)

I guess it's fine to leave those for a follow-up.

thaJeztah

LGTM, let's bring this one in

see docker/buildx#1509 moby/moby#43126 docker/cli#3990

TalWeisler · 2023-06-21T14:42:13Z

Is the change supported in RHEL8?

thaJeztah · 2023-06-21T14:46:19Z

@TalWeisler it's part of the docker cli v23.0.0 and up, so if you have that package installed, then it should be supported. We don't provide packages for RHEL (other than for s390x architectures), but you should be able to install the CentOS 8 packages (see https://docs.docker.com/engine/install/centos/)

TalWeisler · 2023-06-22T08:02:55Z

Thanks @thaJeztah.
After looking further, the command docker manifest inspect seems to work fine from v23.0.0, but the API request GET /v2/<name>/manifests/<reference> returns 404 Not Found :

{
    "errors": [
        {
            "code": "MANIFEST_UNKNOWN",
            "message": "OCI index found, but accept header does not support OCI indexes"
        }
    ]
}

The request include the Accept headers:
application/vnd.oci.image.manifest.v2+json, application/vnd.oci.image.manifest.v1+json
Tested on ubuntu2004 and rhel8
Image: ubuntu:latest

thaJeztah · 2023-06-22T08:55:14Z

@TalWeisler ah, thanks for that; that looks like a possible bug. What registry are you testing this against? Is that Docker Hub, or another registry?

I can see two (potential) issues here;

docker CLI (for docker manifest inspect) not setting all the Accept headers; is possible that not all registries look at that (the OCI distribution spec does not really define Content-Negotiation (or not properly))
The error response is a bit odd; for content-negotiation, I'd expect a 415 status to be returned (not a 404); see RFC9110, section 15.5.1

TalWeisler · 2023-06-22T09:01:42Z

@thaJeztah
https://registry-1.docker.io/v2/library/ubuntu/manifests/latest

Headers:

Accept: application/vnd.oci.image.manifest.v1+json
Accept: application/vnd.docker.distribution.manifest.v1+json
Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
Accept: application/json
Accept: application/vnd.oci.image.manifest.v2+json
Authorization

tianon · 2023-06-22T17:00:46Z

application/vnd.oci.image.manifest.v2+json isn't a thing - that was probably supposed to be application/vnd.oci.image.index.v1+json? 😅

thaJeztah · 2023-06-24T16:42:37Z

I did a quick check to see what headers it passes; here's a quick "registry" to debug the requests;

package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Println(r.Method, r.URL, r.Header)
		w.Write([]byte("{}"))
	})
	_ = http.ListenAndServe(":5001", nil)
}

Now running docker manifest inspect;

docker manifest inspect --insecure localhost:5001/foo/bar

I get these logs:

GET /v2/ map[Accept-Encoding:[gzip] Connection:[close] User-Agent:[Docker-Client/24.0.2 (darwin)]]
GET /v2/foo/bar/manifests/latest map[Accept:[application/vnd.docker.distribution.manifest.list.v2+json application/vnd.oci.image.index.v1+json application/vnd.oci.image.manifest.v1+json application/vnd.docker.distribution.manifest.v2+json] Accept-Encoding:[gzip] Connection:[close] User-Agent:[Docker-Client/24.0.2 (darwin)]]
GET /v2/ map[Accept-Encoding:[gzip] Connection:[close] User-Agent:[Docker-Client/24.0.2 (darwin)]]
GET /v2/foo/bar/manifests/latest map[Accept:[application/vnd.docker.distribution.manifest.v2+json application/vnd.docker.distribution.manifest.list.v2+json application/vnd.oci.image.index.v1+json application/vnd.oci.image.manifest.v1+json] Accept-Encoding:[gzip] Connection:[close] User-Agent:[Docker-Client/24.0.2 (darwin)]]

So (besides the /v2/ endpoint, which is for detecting registry version and authenticatoin), the following "Accept" headers are sent:

application/vnd.docker.distribution.manifest.list.v2+json ("Docker Image Manifest Version 2, Schema 2" manifest list (pre-dates the OCI manifest lists)
application/vnd.oci.image.index.v1+json (OCI manifest list)
application/vnd.oci.image.manifest.v1+json (OCI image manifest)
application/vnd.docker.distribution.manifest.v2+json ("Docker Image Manifest Version 2, Schema 2" image manifest)

So, I think that looks good.

application/vnd.oci.image.manifest.v2+json isn't a thing - that was probably supposed to be application/vnd.oci.image.index.v1+json? 😅

Yes, that looks odd. I did a quick search, and found a reference to that (invalid) mediatype in 3 locations;

2 in github.com/anchore/harbor-scanner-adapter;

https://github.com/anchore/harbor-scanner-adapter/blob/aafc46f138503405e4419afe758d3b6173a5624e/pkg/adapter/adapter.go#L10
https://github.com/anchore/harbor-scanner-adapter/blob/aafc46f138503405e4419afe758d3b6173a5624e/pkg/http/api/v1/handler.go#L25

	DockerImageMimeType           = "application/vnd.oci.image.manifest.v2+json"

1 in some comment describing that Buildah must produce that mediatype for Docker
https://github.com/smart-edge-open/sriov-fec-operator/blob/45fa1e052bdcf93653cdbb8c5916cbe519f5f273/Makefile#L6-L7

# Container format for podman. Required to build containers with "ManifestType": "application/vnd.oci.image.manifest.v2+json",
export BUILDAH_FORMAT=docker

{
    "errors": [
        {
            "code": "MANIFEST_UNKNOWN",
            "message": "OCI index found, but accept header does not support OCI indexes"
        }
    ]
}

I did find code in the CNCF distribution repository about that error, but don't see it handling wrong media types.
https://github.com/distribution/distribution/blob/87b280718d3898c8c78ddf8f358c58f6e92e3c9e/registry/handlers/manifests.go#L169-L176

I do think it's odd to return a 404 (and not a 415), but that may have been for backward compatibility. It's worth looking though if that's defined somewhere in the spec.
https://github.com/distribution/distribution/blob/87b280718d3898c8c78ddf8f358c58f6e92e3c9e/registry/api/v2/errors.go#L75C1-L82

TalWeisler · 2023-06-25T09:42:16Z

@thaJeztah Thank you!

see docker/buildx#1509 moby/moby#43126 docker/cli#3990

crazy-max reviewed Jan 27, 2023

View reviewed changes

cli/command/manifest/util.go Show resolved Hide resolved

jedevc force-pushed the manifest-oci branch from ebaaefe to f4cddec Compare January 27, 2023 13:51

jedevc added 3 commits January 27, 2023 13:51

manifest: add support for oci image types

070825b

Signed-off-by: Justin Chadwell <me@jedevc.com>

manifest: save raw manifest content on download

67b9617

This prevents us needing to attempt to reconstruct the exact indentation registry side, which is not canonical - so may differ. Signed-off-by: Justin Chadwell <me@jedevc.com>

jedevc force-pushed the manifest-oci branch from f4cddec to 67b9617 Compare January 27, 2023 13:56

jedevc mentioned this pull request Jan 27, 2023

Revert "chore: use double spaces when marshalling manifests/indexes" moby/buildkit#3539

Closed

thaJeztah added impact/changelog status/2-code-review kind/bugfix PR's that fix bugs labels Jan 27, 2023

thaJeztah added this to the 23.0.0 milestone Jan 27, 2023

thaJeztah reviewed Jan 27, 2023

View reviewed changes

martadinata666 mentioned this pull request Jan 29, 2023

New OCI manifest issue containrrr/shepherd#92

Closed

jedevc mentioned this pull request Jan 30, 2023

Add manifest merging support #3994

Open

1 task

thaJeztah reviewed Jan 30, 2023

View reviewed changes

thaJeztah approved these changes Jan 30, 2023

View reviewed changes

neersighted approved these changes Jan 30, 2023

View reviewed changes

vvoland approved these changes Jan 30, 2023

View reviewed changes

thaJeztah reviewed Jan 30, 2023

View reviewed changes

crazy-max approved these changes Jan 31, 2023

View reviewed changes

thaJeztah approved these changes Jan 31, 2023

View reviewed changes

thaJeztah merged commit 3a11830 into docker:master Jan 31, 2023

thaJeztah mentioned this pull request Jan 31, 2023

engine: 23.0.0 release notes docker/docs#16421

Merged

yosifkit mentioned this pull request Feb 1, 2023

ubuntu:22.04 (at least) - no such manifest when getting the manifest with docker manifest inspect docker-library/official-images#13999

Closed

neersighted mentioned this pull request Feb 2, 2023

docker manifest inspect doesn't support OCI manifests moby/moby#44898

Closed

polarathene mentioned this pull request Feb 7, 2023

chore(deps): Bump docker/build-push-action from 3.3.0 to 4.0.0 docker-mailserver/docker-mailserver#3066

Merged

ddelange mentioned this pull request Feb 7, 2023

Add layer sizes to buildx imagetools inspect docker/buildx#1558

Open

aler9 added a commit to bluenviron/mediamtx that referenced this pull request Feb 13, 2023

fix generation of Docker images with buildx v0.10

efa75a1

see docker/buildx#1509 moby/moby#43126 docker/cli#3990

JoMallari98 pushed a commit to JoMallari98/rtsp-server that referenced this pull request Feb 25, 2023

fix generation of Docker images with buildx v0.10

d222de3

see docker/buildx#1509 moby/moby#43126 docker/cli#3990

This was referenced Feb 28, 2023

Docker Manifest Inspect Not working for Ubuntu Latest images moby/moby#45077

Closed

missing docker manifest for v3.11.4 hairyhenderson/gomplate#1652

Closed

patrick-merchant mentioned this pull request Apr 4, 2023

Documentation: Docker 23.0.0 required to deploy Atlantis with ACI runatlantis/atlantis#3291

Closed

pwschuurman mentioned this pull request May 10, 2023

Update buildkit version from 0.10.6 -> 0.11.3 kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#1218

Merged

eliteprodev pushed a commit to eliteprodev/rtc-simple-server that referenced this pull request May 17, 2023

fix generation of Docker images with buildx v0.10

5c30d92

see docker/buildx#1509 moby/moby#43126 docker/cli#3990

wagoodman mentioned this pull request Jun 26, 2023

Fix OCI and Docker media types anchore/harbor-scanner-adapter#49

Merged

eliteprodev added a commit to eliteprodev/rtc-simple-server that referenced this pull request Sep 8, 2023

fix generation of Docker images with buildx v0.10

c8684b6

see docker/buildx#1509 moby/moby#43126 docker/cli#3990

polarathene mentioned this pull request Oct 15, 2023

[TODO]: CI - Assess if provenance: false is still necessary docker-mailserver/docker-mailserver#3582

Open

mallardduck mentioned this pull request May 9, 2024

[BUG] rancher/shell:v0.1.24 reporting manifest invalid rancher/rancher#45424

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add OCI support to manifest subcommand #3990

Add OCI support to manifest subcommand #3990

jedevc commented Jan 27, 2023

jedevc commented Jan 27, 2023

thaJeztah Jan 27, 2023

jedevc Jan 30, 2023

thaJeztah Jan 30, 2023

tianon Jan 30, 2023

rumpl Jan 30, 2023

thaJeztah Jan 30, 2023

thaJeztah left a comment

neersighted left a comment

thaJeztah Jan 30, 2023 •

edited

neersighted Jan 30, 2023

jedevc Jan 30, 2023

thaJeztah Jan 31, 2023

thaJeztah left a comment

TalWeisler commented Jun 21, 2023

thaJeztah commented Jun 21, 2023

TalWeisler commented Jun 22, 2023 •

edited

thaJeztah commented Jun 22, 2023

TalWeisler commented Jun 22, 2023

tianon commented Jun 22, 2023

thaJeztah commented Jun 24, 2023

TalWeisler commented Jun 25, 2023


		// SchemaV2Manifest is used for inspection
		// TODO: Deprecate this and store manifest blobs

Add OCI support to manifest subcommand #3990

Add OCI support to manifest subcommand #3990

Conversation

jedevc commented Jan 27, 2023

jedevc commented Jan 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thaJeztah left a comment

Choose a reason for hiding this comment

neersighted left a comment

Choose a reason for hiding this comment

thaJeztah Jan 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thaJeztah left a comment

Choose a reason for hiding this comment

TalWeisler commented Jun 21, 2023

thaJeztah commented Jun 21, 2023

TalWeisler commented Jun 22, 2023 • edited

thaJeztah commented Jun 22, 2023

TalWeisler commented Jun 22, 2023

tianon commented Jun 22, 2023

thaJeztah commented Jun 24, 2023

TalWeisler commented Jun 25, 2023

thaJeztah Jan 30, 2023 •

edited

TalWeisler commented Jun 22, 2023 •

edited