Error fetching metadata from Github Container Registry

[christian-schlichtherle]

Describe the bug
We are using the ArgoCD Image Updater 0.12.1 for Continuous Deployment to our development environment. More specifically, we are using the update-strategy digest to observe image updates for the tag main on an image that is hosted on ghcr.io for our organization, hence the repos are private.

This was working fine for months until this morning. Sometime today, it stopped working and the Image Updater keeps logging error messages like the following:

time="2023-01-20T16:40:15Z" level=error msg="Error fetching metadata for ***/postgresql-maintenance:main - neither V1 or V2 or OCI manifest returned by registry: manifest unknown: OCI index found, but Accept header does not support OCI indexes" alias=postgresql-maintenance application=***-dev image_name=***/postgresql-maintenance image_tag="sha256:eb41aef1bf95aca9aa38baa12e3be5505293388135d7111eb9a585576eb80651" registry=ghcr.io

Looks like the behavior of ghcr.io has been changed, but maybe I should change the Accept header to whatever is required, but how?

As a side, I have checked the validity of my Personal Access Token (PAT) for ghcr.io - it's fine. I even created a new PAT just in case, but that changed nothing.

To Reproduce
Setup a private repo on GitHub, push some image, then configure the ArgoCD image updater using it's annotations and watch the logs.

Expected behavior
An update to the images.

Additional context
see above

Version
see above

Logs
see above

[lusid]

We are having the exact same issue. Just started happening today. We are also using digest with ghcr.io, I also checked that my PAT is working fine, and it is reporting the exact same error. I also get the error when I try to use the argocd-image-updater test approach.

export GITHUB_PULLSECRET="username:token"
argocd-image-updater test --credentials env:GITHUB_PULLSECRET ghcr.io/org/image:tag --update-strategy latest

ERRO[0001] Error fetching metadata for org/image:tag - neither V1 or V2 or OCI manifest returned by registry: manifest unknown: OCI index found, but Accept header does not support OCI indexes application=test image_alias= image_digest= image_name=ghcr.io/org/image image_tag=tag registry_url=ghcr.io

[tomjohnburton]

Same issue here, started yesterday
Perhaps Github rolled out an update

[jannfis]

If someone has a public image that reproduces this issue, I'd be grateful for that and would push out a fix quickly once I figured out what's wrong.

[dir01]

@jannfis

/ $ argocd-image-updater test ghcr.io/dir01/undercast-bot:latest --update-strategy=newest-build
DEBU[0000] Creating in-cluster Kubernetes client
INFO[0000] retrieving information about image            image_alias= image_digest= image_name=ghcr.io/dir01/undercast-bot image_tag=latest registry_url=ghcr.io
DEBU[0000] setting rate limit to 20 requests per second  prefix=ghcr.io registry="https://ghcr.io"
DEBU[0000] Inferred registry from prefix ghcr.io to use API https://ghcr.io
INFO[0000] Fetching available tags and metadata from registry  application=test image_alias= image_digest= image_name=ghcr.io/dir01/undercast-bot image_tag=latest registry_url=ghcr.io
ERRO[0001] Error fetching metadata for dir01/undercast-bot:latest - neither V1 or V2 or OCI manifest returned by registry: manifest unknown: OCI index found, but Accept header does not support OCI indexes  application=test image_alias= image_digest= image_name=ghcr.io/dir01/undercast-bot image_tag=latest registry_url=ghcr.io
INFO[0001] Found 0 tags in registry                      application=test image_alias= image_digest= image_name=ghcr.io/dir01/undercast-bot image_tag=latest registry_url=ghcr.io
INFO[0001] latest image according to constraint is ghcr.io/dir01/undercast-bot:latest  application=test image_alias= image_digest= image_name=ghcr.io/dir01/undercast-bot image_tag=latest registry_url=ghcr.io

[dir01]

Not sure if it's applicable, but in manual reproduction of this issue I found out that adding Accept:application/vnd.oci.image.index.v1+json to the list of headers fixes it

> http get https://ghcr.io/v2/dir01/undercast-bot/manifests/latest Authorization:"Bearer $token" Accept:application/vnd.docker.distribution.manifest.v2+json Accept:application/vnd.docker.distribution.manifest.list.v2+json
HTTP/1.1 404 Not Found
Content-Length: 117
Content-Type: application/json
Date: Sat, 21 Jan 2023 09:38:12 GMT
X-GitHub-Request-Id: D504:6F36:69D56E:6F02DE:63CBB284
docker-distribution-api-version: registry/2.0

{
    "errors": [
        {
            "code": "MANIFEST_UNKNOWN",
            "message": "OCI index found, but Accept header does not support OCI indexes"
        }
    ]
}

but with addition of Accept:application/vnd.oci.image.index.v1+json

> http get https://ghcr.io/v2/dir01/undercast-bot/manifests/latest Authorization:"Bearer $token" Accept:application/vnd.docker.distribution.manifest.v2+json Accept:application/vnd.docker.distribution.manifest.list.v2+json Accept:application/json Accept:application/vnd.oci.image.index.v1+json
HTTP/1.1 200 OK
Content-Length: 2385
Content-Type: application/vnd.oci.image.index.v1+json
Date: Sat, 21 Jan 2023 09:38:22 GMT
ETag: "sha256:bbc50df7945524aedd3dbc9773202d79f7b2ad8afc41c36b1d4667fa02e2c22c"
X-GitHub-Request-Id: D509:134E8:1C3CBC1:1D73A33:63CBB28E
docker-content-digest: sha256:bbc50df7945524aedd3dbc9773202d79f7b2ad8afc41c36b1d4667fa02e2c22c
docker-distribution-api-version: registry/2.0

{
    "manifests": [
        {
            "digest": "sha256:2028474f36889e7c6496377c072078daab7c5f442e661d36891862b9a3c80af1",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
...

[jannfis]

We just released v0.12.2 which should fix this issue. If it doesn't, feel free to reopen, please.

[jannfis]

And thanks a lot @dir01 for providing the public image to test against. Much appreciated!

[christian-schlichtherle]

@jannfis The image was updated and the Helm chart was updated, but unfortunately the new Helm chart does not reference the new image in it's appVersion.

[jannfis]

@christian-schlichtherle The Image Updater never touches a Helm chart directly, so I'm not sure what you mean by

the new Helm chart does not reference the new image in it's appVersion

Could you elaborate a little on that, please?

[christian-schlichtherle]

@jannfis You need to update the file Chart.yaml and put in appVersion: v0.12.2 please.

[jannfis]

Ah. You mean the Helm chart for Argo CD Image Updater. Now I understand. Sorry :)

Unfortunately, I'm not the one managing this chart. Please contact the kind folks over at https://github.com/argoproj/argo-helm