Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cosign sign does not use local image registry credentials #587

Closed
adambkaplan opened this issue Aug 27, 2021 · 29 comments
Closed

cosign sign does not use local image registry credentials #587

adambkaplan opened this issue Aug 27, 2021 · 29 comments
Labels
no-issue-activity question Further information is requested

Comments

@adambkaplan
Copy link

Question

I'm trying to sign a container image that I pushed to docker.io. This seems simple enough:

$ cosign sign -key cosign.key docker.io/adambkaplan/ruby-ex

However, when trying to pull the image, cosign does not appear to use my local credentials. Instead cosign appears to do the following:

  1. Try to GET the root of the container registry
  2. With the UNAUTHORIZED response, then it requests a "pull" scoped token
  3. After receiving the token, it then tries to pull the container image. For whatever reason the token is invalid, and the pull fails.

I was able to verify that my ~/.docker/config.json contains valid auth credentials for docker.io, and that I can pull images using those credentials with my container engine (podman).

This could be related to #337.

Here's the debug output:

cosign -d sign -key cosign.key docker.io/adambkaplan/ruby-ex
2021/08/27 10:46:56 --> GET https://index.docker.io/v2/
2021/08/27 10:46:56 GET /v2/ HTTP/1.1
Host: index.docker.io
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


2021/08/27 10:46:56 <-- 401 https://index.docker.io/v2/ (201.76705ms)
2021/08/27 10:46:56 HTTP/1.1 401 Unauthorized
Content-Length: 87
Content-Type: application/json
Date: Fri, 27 Aug 2021 14:46:56 GMT
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

2021/08/27 10:46:56 --> GET https://auth.docker.io/token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io [body redacted: basic token response contains credentials]
2021/08/27 10:46:56 GET /token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io HTTP/1.1
Host: auth.docker.io
User-Agent: go-containerregistry/v0.6.0
Accept-Encoding: gzip


2021/08/27 10:46:57 <-- 200 https://auth.docker.io/token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io (179.251563ms) [body redacted: basic token response contains credentials]
2021/08/27 10:46:57 HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json
Date: Fri, 27 Aug 2021 14:46:56 GMT
Strict-Transport-Security: max-age=31536000


2021/08/27 10:46:57 --> GET https://index.docker.io/v2/adambkaplan/ruby-ex/manifests/latest
2021/08/27 10:46:57 GET /v2/adambkaplan/ruby-ex/manifests/latest HTTP/1.1
Host: index.docker.io
User-Agent: go-containerregistry/v0.6.0
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization: <redacted>
Accept-Encoding: gzip


2021/08/27 10:46:57 <-- 401 https://index.docker.io/v2/adambkaplan/ruby-ex/manifests/latest (43.68461ms)
2021/08/27 10:46:57 HTTP/1.1 401 Unauthorized
Content-Length: 162
Content-Type: application/json
Date: Fri, 27 Aug 2021 14:46:57 GMT
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:adambkaplan/ruby-ex:pull",error="insufficient_scope"

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"adambkaplan/ruby-ex","Action":"pull"}]}]}

2021/08/27 10:46:57 --> GET https://auth.docker.io/token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io [body redacted: basic token response contains credentials]
2021/08/27 10:46:57 GET /token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io HTTP/1.1
Host: auth.docker.io
User-Agent: go-containerregistry/v0.6.0
Accept-Encoding: gzip


2021/08/27 10:46:57 <-- 200 https://auth.docker.io/token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io (38.934215ms) [body redacted: basic token response contains credentials]
2021/08/27 10:46:57 HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json
Date: Fri, 27 Aug 2021 14:46:57 GMT
Strict-Transport-Security: max-age=31536000


2021/08/27 10:46:57 --> GET https://index.docker.io/v2/adambkaplan/ruby-ex/manifests/latest
2021/08/27 10:46:57 GET /v2/adambkaplan/ruby-ex/manifests/latest HTTP/1.1
Host: index.docker.io
User-Agent: go-containerregistry/v0.6.0
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization: <redacted>
Accept-Encoding: gzip


2021/08/27 10:46:57 <-- 401 https://index.docker.io/v2/adambkaplan/ruby-ex/manifests/latest (40.460773ms)
2021/08/27 10:46:57 HTTP/1.1 401 Unauthorized
Content-Length: 162
Content-Type: application/json
Date: Fri, 27 Aug 2021 14:46:57 GMT
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:adambkaplan/ruby-ex:pull",error="insufficient_scope"

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"adambkaplan/ruby-ex","Action":"pull"}]}]}

error: signing docker.io/adambkaplan/ruby-ex: getting remote image: GET https://index.docker.io/v2/adambkaplan/ruby-ex/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:adambkaplan/ruby-ex Type:repository]]

@adambkaplan adambkaplan added the question Further information is requested label Aug 27, 2021
@dlorenc
Copy link
Member

dlorenc commented Aug 27, 2021

What if you leave off the "docker.io"? cc @jonjohnsonjr any ideas here?

@jonjohnsonjr
Copy link
Contributor

I was able to verify that my ~/.docker/config.json contains valid auth credentials for docker.io

Can you share this file (with anything sensitive redacted)?

@adambkaplan
Copy link
Author

Looks like a pretty standard dockerconfigjson:

{
	"auths": {
		"cloud.openshift.com": {
			"auth": "<TOKEN-REDACTED>"
		},
		"docker.io": {
			"auth": "<TOKEN-REDACTED>"
		},
		"quay.io": {
			"auth": "<TOKEN-REDACTED>"
		},
		"registry.connect.redhat.com": {
			"auth": "<TOKEN-REDACTED>"
		},
		"registry.redhat.io": {
			"auth": "<TOKEN-REDACTED>"
		},
		"registry.svc.ci.openshift.org": {
			"auth": "<TOKEN-REDACTED>"
		}
}

@adambkaplan
Copy link
Author

What if you leave off the "docker.io"? cc @jonjohnsonjr any ideas here?

Same result.

@adambkaplan
Copy link
Author

adambkaplan commented Aug 30, 2021

Note for quay.io authentication isn't the issue - cosign is able to log in just fine, however uploading signatures fails because quay.io doesn't accept the cosign layer type.

See log: https://gist.github.com/adambkaplan/0b8b4bfdaf8771d57730eface258d5a8

@jonjohnsonjr
Copy link
Contributor

It's stupid but can you try changing the "docker.io" key to "https://index.docker.io/v1/"?

Some version of docker would set this as the key, and I haven't found a reasonable way to work around this without breaking someone: https://github.com/docker/cli/blob/25eee83d6b8c475548254b2decc9c8e0490d229c/cli/config/configfile/file.go#L23

@adambkaplan
Copy link
Author

It's stupid but can you try changing the "docker.io" key to "https://index.docker.io/v1/"?

Sadly same result 😞

@dekkagaijin
Copy link
Member

Hm. You're right, it's not sending creds in the initial auth handshake. I would have guessed that https://index.docker.io/v1/ was the issue, as well.

Looks like ggcr does recognize docker.io and does translate it to index.docker.io under the hood, which means that it uses https://index.docker.io/v1/ as its auth key: https://github.com/google/go-containerregistry/blob/7e0ed51a7bb1930bfb6859f319975b578ed04de2/pkg/authn/keychain.go#L73-L76

@jonjohnsonjr
Copy link
Contributor

Sadly same result

😭

You're certain you don't have a credential store or credential helper configured?

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

$ cat config.json 
{
        "auths": {
                "docker.io": {
                        "auth": "QXp1cmVEaWFtb25kOmh1bnRlcjI="
                }
        }
}
$ echo 'docker.io' | DOCKER_CONFIG=$PWD crane auth get
credentials not found in native keychain

$ cat config.json 
{
        "auths": {
                "https://index.docker.io/v1/": {
                        "auth": "QXp1cmVEaWFtb25kOmh1bnRlcjI="
                }
        }
}
$ echo 'docker.io' | DOCKER_CONFIG=$PWD crane auth get
{"Username":"AzureDiamond","Secret":"hunter2"}

This would match the failure mode I expect from your config file :/

@adambkaplan
Copy link
Author

I am running Fedora 34 Silverblue - no credential store as far as I am aware of, and assuming the presence of anything Docker is not a safe assumption!

@jonjohnsonjr
Copy link
Contributor

I don't want to say I don't believe you, but I'm having a hard time imagining what could be going wrong here 😄

Can you verify that cosign is reading the right file (via strace or similar)?

Are you executing cosign as a different user, maybe?

Is DOCKER_CONFIG set to point to a different file?

@adambkaplan
Copy link
Author

Can you verify that cosign is reading the right file (via strace or similar)?

I can verify it is reading what I think is the right file. From strace, it's clear the config.json file is read:

read(6, "{\n\t\"auths\": {\n\t\t\"cloud.openshift"..., 512) = 512
read(6, "<redacted>=\"\n\t\t"..., 1024) = 1024
read(6, "<redacted>"..., 2048) = 1378

Are you executing cosign as a different user, maybe?

I'm running cosign in a Silverblue toolbox. It's a container-like environment where the user looks and feels the same as the user on the host system. So in the toolbox I'm my normal username with my normal permissions.

Is DOCKER_CONFIG set to point to a different file?

No - this is empty.

@jonjohnsonjr
Copy link
Contributor

and that I can pull images using those credentials with my container engine (podman).

So I know that podman tries to read a different file by default from here:

The path of the authentication file can be specified by the user by setting the authfile flag. The default path for reading and writing credentials is ${XDG_RUNTIME_DIR}/containers/auth.json. Podman will use existing credentials if the user does not pass in a username. Podman will first search for the username and password in the ${XDG_RUNTIME_DIR}/containers/auth.json, if they are not valid, Podman will then use any existing credentials found in $HOME/.docker/config.json.

Are you certain that podman is reading the same credentials file as cosign?

It's possible that fixing this issue gets you some credentials, but not valid credentials?

In your original debug logs, we see that cosign isn't sending any credentials to Docker Hub:

2021/08/27 10:46:56 --> GET https://auth.docker.io/token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io [body redacted: basic token response contains credentials]
2021/08/27 10:46:56 GET /token?scope=repository%3Aadambkaplan%2Fruby-ex%3Apull&service=registry.docker.io HTTP/1.1
Host: auth.docker.io
User-Agent: go-containerregistry/v0.6.0
Accept-Encoding: gzip

If we're sending invalid credentials, we would expect to see this line as well:

Authorization: <redacted>

@adambkaplan
Copy link
Author

@jonjohnsonjr it seems the root issue here is that it is very hard to identify which credentials are being used to push/pull from the container registry. Next chance I have some free "hacking" time I can try submitting a PR to surface this in debug logs.

Any initial guidance/pointers would be appreciated - thanks!

@tommyreilly
Copy link

I'm hitting this same issue when trying to integrate cosign into our teams CI flow. Any use of buildah, podman, skopeo based tools where the credentials are stored in a different location than ~/.docker, seems to have cosign not recognise the use of a different container build tool and it's already authenticated information creds.

As these tools are commonly used in automated flows, it would be great if anyone has any ideas to work round the current situation.

@DazWilkin
Copy link

DazWilkin commented Oct 8, 2021

I skimmed the issue but had issues (403) POST'ing to GHCR.

I'm running Docker under a Snap (I assume I'll have the same issue w/ e.g. Podman) and the Snap maintains ./docker/config.json in a Snap subdir (${HOME}/snap/docker/current/.docker/config.json).

The solution was to copy (I guess I could have ln -s too?) the file to Docker's default config location (${HOME}.docker/config.json).

I assume the issue is that cosign expects a Docker daemon with a default config (or env vars set).

I think it would be preferable either to be able to configure cosign sign with an auth token for the registry referenced by the image (i.e. replacing the value from .docker/config.json) or to implement the registry API and support e.g. cosign registry login directly.

@jonjohnsonjr
Copy link
Contributor

I assume the issue is that cosign expects a Docker daemon with a default config (or env vars set).

Not a daemon, just the normal docker config file behavior; i.e. $HOME/.docker/config.json but it can be overridden by DOCKER_CONFIG. If Snap is moving this around but not setting that environment variable, I'd argue it's a bug in Snap?

or to implement the registry API and support e.g. cosign registry login directly.

There's not really a registry API for logging in. We could support a login command, but it just manipulates a local configuration file and wouldn't support configuring credential helpers...

I think the cosign tool and docs could probably do a better job describing how auth works, regardless.

@dekkagaijin
Copy link
Member

We could support a login command, but it just manipulates a local configuration file and wouldn't support configuring credential helpers...

Veto :p

We should endeavor to figure out @adambkaplan's issue and identify how we've differed from Docker's credential resolution behavior. I might have volunteered myself for that 😢

I think the cosign tool and docs could probably do a better job describing how auth works, regardless.

Agreed.

@dekkagaijin
Copy link
Member

dekkagaijin commented Oct 9, 2021

Any use of buildah, podman, skopeo based tools where the credentials are stored in a different location than ~/.docker, seems to have cosign not recognise the use of a different container build tool and it's already authenticated information creds.

@tommyreilly you're right, it only supports docker's behavior right now.

If we end up supporting some alternative to Docker's configuration and authentication scheme, it should be one we maintain. I think documenting our docker mimicry and prescribing a manual way of setting things up for automation should suffice for now

@lfstm
Copy link

lfstm commented Oct 13, 2021

Hi, had auth issues (I use podman instead of docker), the following was a workaround for me:

cat ${XDG_RUNTIME_DIR}/containers/auth.json > ~/.docker/config.json

@ricardomaraschini
Copy link

ricardomaraschini commented Dec 30, 2021

My two cents: auth.json (podman) or config.json (docker) must contain an entry with authentications for https://index.docker.io/v1/ otherwise the authentication seems to fail. I have solved it by adding an extra entry like so (trailing / needed):

{
        "auths": {
                "docker.io": {
                        "auth": "redacted"
                },
                "https://index.docker.io/v1/": {
                        "auth": "redacted"
                }
        }
}

I found out that if the registry URL is index.docker.io go-containerregistry uses the full url when looking for creds.

@ibazulic
Copy link

ibazulic commented Feb 21, 2022

Sadly, I'm expirencing the same issue as others. cosign fails to sign an image on my local Quay installation, apparently due to lacking credentials:

# cosign -d sign --key cosign.key quay.skynet/ibazulic/quay:v2.9.5
an error occurred: no provider found for that key reference, will try to load key from disk...
Enter password for private key: 2022/02/21 14:04:19 --> GET https://quay.skynet/v2/
...

2022/02/21 14:04:19 --> GET https://quay.skynet/v2/auth?scope=repository%3Aibazulic%2Fquay%3Apull&service=quay.skynet [body redacted: basic token response contains credentials]
2022/02/21 14:04:19 GET /v2/auth?scope=repository%3Aibazulic%2Fquay%3Apull&service=quay.skynet HTTP/1.1
Host: quay.skynet
User-Agent: cosign/v1.5.2 (linux; amd64) go-containerregistry/v0.8.1-0.20220125170349-50dfc2733d10
Authorization: <redacted>
Accept-Encoding: gzip


2022/02/21 14:04:20 <-- 200 https://quay.skynet/v2/auth?scope=repository%3Aibazulic%2Fquay%3Apull&service=quay.skynet (193.918782ms) [body redacted: basic token response contains credentials]
2022/02/21 14:04:20 HTTP/1.1 200 OK
Content-Length: 977
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: application/json
Date: Mon, 21 Feb 2022 14:04:20 GMT
Server: nginx/1.14.1
X-Frame-Options: DENY


2022/02/21 14:04:20 --> GET https://quay.skynet/v2/ibazulic/quay/manifests/v2.9.5
2022/02/21 14:04:20 GET /v2/ibazulic/quay/manifests/v2.9.5 HTTP/1.1
Host: quay.skynet
User-Agent: cosign/v1.5.2 (linux; amd64) go-containerregistry/v0.8.1-0.20220125170349-50dfc2733d10
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization: <redacted>
Accept-Encoding: gzip


2022/02/21 14:04:20 <-- 401 https://quay.skynet/v2/ibazulic/quay/manifests/v2.9.5 (4.850918ms)
2022/02/21 14:04:20 HTTP/1.1 401 UNAUTHORIZED
Content-Length: 112
Content-Type: application/json
Date: Mon, 21 Feb 2022 14:04:20 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: nginx/1.14.1
Www-Authenticate: Bearer realm="https://quay.skynet/v2/auth",service="quay.skynet",scope="repository:ibazulic/quay:pull"

{"errors":[{"code":"UNAUTHORIZED","detail":{},"message":"access to the requested resource is not authorized"}]}

Error: signing [quay.skynet/ibazulic/quay:v2.9.5]: accessing entity: GET https://quay.skynet/v2/ibazulic/quay/manifests/v2.9.5: UNAUTHORIZED: access to the requested resource is not authorized; map[]
main.go:46: error during command execution: signing [quay.skynet/ibazulic/quay:v2.9.5]: accessing entity: GET https://quay.skynet/v2/ibazulic/quay/manifests/v2.9.5: UNAUTHORIZED: access to the requested resource is not authorized; map[]

Output of Docker's config.json file:

# cat ~/.docker/config.json 
{
	"auths": {
		"quay.skynet": {
			"auth": "SOMETHING"
		}
	}
}

Podman's auth.json also exists:

# cat /run/containers/0/auth.json 
{
	"auths": {
		"quay.skynet": {
			"auth": "SOMETHING"
		}
	}
}

Any ideas?

@jonjohnsonjr
Copy link
Contributor

Any ideas?

You're certain the credentials are valid?

@imjasonh
Copy link
Member

imjasonh commented Mar 3, 2022

Since google/go-containerregistry#1185 which should be in cosign afaik, cosign will check for podman's auth config as well. Does this help at all?

@ibazulic
Copy link

ibazulic commented Mar 3, 2022

Any ideas?

You're certain the credentials are valid?

Didn't see your response till now, sorry. Yes, credentials are fine, push/pulls against the registry function normally. This is my own local instance. I'll try again with a new release.

@funkypenguin
Copy link

In my case, I discovered that this .docker/config.json results in authentication failure:

{"auths":{"https://index.docker.io":{"auth":"<redacted>"}}}

But this one results in success:

{"auths":{"https://index.docker.io/v1/":{"auth":"<redacted>"}}}

Interestingly the authentication failure when /v1 is not appended clearly shows that /v2/ is attempted by default:

main.go:46: error during command execution: GET https://index.docker.io/v2/myrepo/curl/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:myrepo/curl Type:repository]]

@github-actions
Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions
Copy link

This issue was closed because it has been stalled for 5 days with no activity.

@devops-team-92
Copy link

Hi, had auth issues (I use podman instead of docker), the following was a workaround for me:

cat ${XDG_RUNTIME_DIR}/containers/auth.json > ~/.docker/config.json

working as expected , great 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no-issue-activity question Further information is requested
Projects
None yet
Development

No branches or pull requests