Cannot pull images pushed from oci reference on Fly.io

[andrewbaxter]

Sorry in case this is a vendor specific issue.

I have an image in oci directory format that I'm pushing to Fly.io's private repo. The copy works with no errors, however launching an instance fails with an error like Could not find image "registry.fly.io/myimage:mytag".

Copying from the oci directory to anything else (docker-daemon, docker, docker-archive) and then from that (docker-daemon, docker, docker-archive) to fly works fine. So if I understand it correctly, either Skopeo is doing something different when it copies from an oci directory vs docker-daemon/docker, or else the image is broken and docker-daemon/docker is fixing up the pushed image when it receives it. Since copying to docker archive first works, Skopeo must be doing something different when it copies from an oci dir vs when it copies from a docker archive.

I did look through the code and I couldn't see any big branches that would make it different in the main copy code. Do you have any idea what could be going on?

I cloned skopeo today to try it, but also had issues with 1.10.0 (and I was using those to try to reproduce this issue in go code calling Copy directly).

[mtrmac]

Thanks for reaching out.

The above is really not enough to go on. At the very least, please collect the precise and full command line, and a log of skopeo --debug copy, for the process publishing the image. And after publishing the image, see if skopeo inspect can find the image.

[andrewbaxter]

Sure thing!

$ skopeo --debug copy oci:/home/andrew/myimage docker://registry.fly.io/bread-prod:pgwrapper8
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf" 
DEBU[0000] No credentials matching registry.fly.io/bread-prod found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching registry.fly.io/bread-prod found in /home/andrew/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.fly.io/bread-prod in credential helper containers-auth.json in file /home/andrew/.docker/config.json 
DEBU[0000]  No signature storage configuration found for registry.fly.io/bread-prod:pgwrapper8, using built-in default file:///home/andrew/.local/share/containers/sigstore 
DEBU[0000] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fly.io 
DEBU[0000] Skipping scan of /etc/docker/certs.d/registry.fly.io due to permission error: open /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0000] Using blob info cache at /home/andrew/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image oci:/home/andrew/myimage
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking if we can reuse blob sha256:05e15430f890f3add1d17620f9db1495ab404d8aa6c02821d9f788ea06a248d4: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0000] Checking /v2/bread-prod/blobs/sha256:05e15430f890f3add1d17620f9db1495ab404d8aa6c02821d9f788ea06a248d4 
DEBU[0000] GET https://registry.fly.io/v2/              
DEBU[0000] Ping https://registry.fly.io/v2/ status 401  
DEBU[0000] HEAD https://registry.fly.io/v2/bread-prod/blobs/sha256:05e15430f890f3add1d17620f9db1495ab404d8aa6c02821d9f788ea06a248d4 
DEBU[0000] ... already exists                           
DEBU[0001] Skipping blob sha256:05e15430f890f3add1d17620f9db1495ab404d8aa6c02821d9f788ea06a248d4 (already present): 
Copying blob 05e15430f890 skipped: already exists  
DEBU[0001] Checking if we can reuse blob sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true 
DEBU[0001] Checking /v2/bread-prod/blobs/sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 
Copying blob 05e15430f890 skipped: already exists  
DEBU[0001] ... already exists                           
Copying blob 05e15430f890 skipped: already exists  
Copying blob c158987b0551 skipped: already exists  
DEBU[0001] No compression detected                      
DEBU[0001] Compression change for blob sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec ("application/vnd.oci.image.config.v1+json") not supported 
DEBU[0001] Using original blob without modification     
DEBU[0001] Checking /v2/bread-prod/blobs/sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec 
DEBU[0001] HEAD https://registry.fly.io/v2/bread-prod/blobs/sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec 
Copying config c397661e98 [>-------------------------------------] 8.0b / 325.0b
Copying config c397661e98 done  
Writing manifest to image destination
DEBU[0002] PUT https://registry.fly.io/v2/bread-prod/manifests/pgwrapper8 
Storing signatures

$ skopeo --debug inspect docker://registry.fly.io/bread-prod:pgwrapper8
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf" 
DEBU[0000] Trying to access "registry.fly.io/bread-prod:pgwrapper8" 
DEBU[0000] No credentials matching registry.fly.io/bread-prod found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching registry.fly.io/bread-prod found in /home/andrew/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.fly.io/bread-prod in credential helper containers-auth.json in file /home/andrew/.docker/config.json 
DEBU[0000]  No signature storage configuration found for registry.fly.io/bread-prod:pgwrapper8, using built-in default file:///home/andrew/.local/share/containers/sigstore 
DEBU[0000] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fly.io 
DEBU[0000] Skipping scan of /etc/docker/certs.d/registry.fly.io due to permission error: open /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0000] GET https://registry.fly.io/v2/              
DEBU[0000] Ping https://registry.fly.io/v2/ status 401  
DEBU[0000] GET https://registry.fly.io/v2/bread-prod/manifests/pgwrapper8 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[0001] Downloading /v2/bread-prod/blobs/sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec 
DEBU[0001] GET https://registry.fly.io/v2/bread-prod/blobs/sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec 
DEBU[0002] Using registries.d directory /etc/containers/registries.d 
DEBU[0002] No credentials matching registry.fly.io/bread-prod found in /run/user/1000/containers/auth.json 
DEBU[0002] No credentials matching registry.fly.io/bread-prod found in /home/andrew/.config/containers/auth.json 
DEBU[0002] Found credentials for registry.fly.io/bread-prod in credential helper containers-auth.json in file /home/andrew/.docker/config.json 
DEBU[0002]  No signature storage configuration found for registry.fly.io/bread-prod:pgwrapper8, using built-in default file:///home/andrew/.local/share/containers/sigstore 
DEBU[0002] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0002] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fly.io 
DEBU[0002] Skipping scan of /etc/docker/certs.d/registry.fly.io due to permission error: open /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0002] GET https://registry.fly.io/v2/              
DEBU[0002] Ping https://registry.fly.io/v2/ status 401  
DEBU[0002] GET https://registry.fly.io/v2/bread-prod/tags/list 
FATA[0002] Error determining repository tags: fetching tags list: invalid status code from registry 404 (Not Found)

[mtrmac]

Thanks!

That looks normal enough so far, but the inspect failure is not relevant. What does skopeo --debug -n inspect docker://registry.fly.io/bread-prod:pgwrapper8 show ?

[andrewbaxter]

$ skopeo --debug inspect -n docker://registry.fly.io/bread-prod:pgwrapper8
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf" 
DEBU[0000] Trying to access "registry.fly.io/bread-prod:pgwrapper8" 
DEBU[0000] No credentials matching registry.fly.io/bread-prod found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching registry.fly.io/bread-prod found in /home/andrew/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.fly.io/bread-prod in credential helper containers-auth.json in file /home/andrew/.docker/config.json 
DEBU[0000]  No signature storage configuration found for registry.fly.io/bread-prod:pgwrapper8, using built-in default file:///home/andrew/.local/share/containers/sigstore 
DEBU[0000] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fly.io 
DEBU[0000] Skipping scan of /etc/docker/certs.d/registry.fly.io due to permission error: open /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0000] GET https://registry.fly.io/v2/              
DEBU[0000] Ping https://registry.fly.io/v2/ status 401  
DEBU[0000] GET https://registry.fly.io/v2/bread-prod/manifests/pgwrapper8 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[0001] Downloading /v2/bread-prod/blobs/sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec 
DEBU[0001] GET https://registry.fly.io/v2/bread-prod/blobs/sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec 
{
    "Name": "registry.fly.io/bread-prod",
    "Digest": "sha256:cee0d0a94d43262895a30fd4dfc7079b37edb3921a6b16122575056a33910b6f",
    "RepoTags": [],
    "Created": null,
    "DockerVersion": "",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:05e15430f890f3add1d17620f9db1495ab404d8aa6c02821d9f788ea06a248d4",
        "sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715"
    ],
    "LayersData": [
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "Digest": "sha256:05e15430f890f3add1d17620f9db1495ab404d8aa6c02821d9f788ea06a248d4",
            "Size": 1087445,
            "Annotations": null
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "Digest": "sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715",
            "Size": 3370706,
            "Annotations": null
        }
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

[andrewbaxter]

I tried it on the one I created from the docker archive (working) and got

$ skopeo --debug inspect -n docker://registry.fly.io/bread-prod:pgwrapper7
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf" 
DEBU[0000] Trying to access "registry.fly.io/bread-prod:pgwrapper7" 
DEBU[0000] No credentials matching registry.fly.io/bread-prod found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching registry.fly.io/bread-prod found in /home/andrew/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.fly.io/bread-prod in credential helper containers-auth.json in file /home/andrew/.docker/config.json 
DEBU[0000]  No signature storage configuration found for registry.fly.io/bread-prod:pgwrapper7, using built-in default file:///home/andrew/.local/share/containers/sigstore 
DEBU[0000] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.fly.io 
DEBU[0000] Skipping scan of /etc/docker/certs.d/registry.fly.io due to permission error: open /etc/docker/certs.d/registry.fly.io: permission denied 
DEBU[0000] GET https://registry.fly.io/v2/              
DEBU[0000] Ping https://registry.fly.io/v2/ status 401  
DEBU[0000] GET https://registry.fly.io/v2/bread-prod/manifests/pgwrapper7 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0001] Downloading /v2/bread-prod/blobs/sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec 
DEBU[0001] GET https://registry.fly.io/v2/bread-prod/blobs/sha256:c397661e981b97c2cac8f69fea3df4c9f166c53466370bf5078177a6d430b9ec 
{
    "Name": "registry.fly.io/bread-prod",
    "Digest": "sha256:7a46fc6e1a996f60fa26de13bfda4cc5e09644e07fc59b184d1b2fe3c7305122",
    "RepoTags": [],
    "Created": "0001-01-01T00:00:00Z",
    "DockerVersion": "",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:05e15430f890f3add1d17620f9db1495ab404d8aa6c02821d9f788ea06a248d4",
        "sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715"
    ],
    "LayersData": [
        {
            "MIMEType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "Digest": "sha256:05e15430f890f3add1d17620f9db1495ab404d8aa6c02821d9f788ea06a248d4",
            "Size": 1087445,
            "Annotations": null
        },
        {
            "MIMEType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "Digest": "sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715",
            "Size": 3370706,
            "Annotations": null
        }
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

[mtrmac]

That looks just fine. If some the consumer doesn’t find the image, I think that needs to be debugged in / within that consumer.

One thing of note to possibly examine in more detail is that the copy command creates an OCI-formatted image (obviously, from an OCI input), and the inspect shows a Docker v2s2-formatted one (from a closer-to-Docker input, though the real reason is an implementation detail of Skopeo). I don’t know how that would translate into a consumer just not seeing any image (vs. complaining about an unsupported format), but it’s a thing to pay attention to. See also skopeo copy --format.

[andrewbaxter]

application/vnd.oci.image.manifest.v1+json vs application/vnd.docker.distribution.manifest.v2+json? Right, I didn't notice that. I thought docker:// would only accept docker manifests, but I guess that's not the case. So in this case the consumer might not recognize oci manifests basically?

The only difference I noticed was that Created gets filled in during the 2nd somewhere (maybe there's some default being applied during some deserialization path?)... (of course, it's an optional field).

I'll try oci -> oci-archive -> docker and see if that has the same issue, and then forward this information to Fly.

Edit: oci-archive didn't work as well, so that seems to confirm that it's an upstream oci issue. Adding Created didn't help (as expected).

And thanks for your help! I wouldn't have discovered the manifest difference.

[andrewbaxter]

Okay, I got a reply from them. They were able to confirm that their issue is with oci manifests, and they asked me to work around it by using docker manifests (v2s2, which I assume I can convert to with the format parameter).