-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Token authentication error: token signed by untrusted key with ID: "xyz" #1143
Comments
I used the public key for the certificate used to sign the tokens and it worked. |
Indeed it's very misleading, that the registry's config setting is named "rootcertbundle" even if just the certificate (public key) is expected there. |
Is this still true? I tried using the pubkey generated with
but that causes
|
I have asked related question here, https://gitlab.com/gitlab-org/gitlab-ee/issues/1424 |
@CVTJNII the @gajus maybe ask your question again here? Seems like there is some configuration issue but it looks like it could be on the token server side and not the registry. The token server should be configured to sign tokens using the private key associated with a public key here |
@dmcgowan can you have a look at https://gitlab.com/gitlab-org/gitlab-ee/blob/e27ebba57dbf8195873cff678b38900084e766d5/app/services/auth/container_registry_authentication_service.rb#L19-31 ? This is the logic for generating the token. I am not much of a Ruby developer, but from I can tell, it is using Therefore, is there something wrong with how I generate the certificate? These are the only files that I generate: openssl req -nodes -newkey rsa:4096 -keyout registry-auth.key -out registry-auth.csr -subj "/CN=gitlab-issuer"
openssl x509 -in registry-auth.csr -out registry-auth.crt -req -signkey registry-auth.key -days 3650
I give Also, see response from GitLab staff, https://gitlab.com/gitlab-org/gitlab-ce/issues/25967#note_20369208. |
What is the relation between Understanding that would help to understand the issue better. |
If thats the case, then how is |
|
Understood. Thats what I thought from the start. Just wanted to make sure. This still leaves me completely baffled as to why my setup is not working. (The HTTPS bit is working as expected.) I have done: [root@gitlab /]# mkdir /etc/gitlab/registry-certs
[root@gitlab /]# cd /etc/gitlab/registry-certs
[root@gitlab registry-certs]# openssl req -nodes -newkey rsa:4096 -keyout registry-auth.key -out registry-auth.csr -subj "/CN=gitlab-issuer"
Generating a 4096 bit RSA private key
............
............................................................................................................++
..........................................++
writing new private key to 'registry-auth.key'
-----
[root@gitlab registry-certs]# openssl x509 -in registry-auth.csr -out registry-auth.crt -req -signkey registry-auth.key -days 3650
Signature ok
subject=/CN=gitlab-issuer
Getting Private key
[root@gitlab registry-certs]# openssl x509 -text -in /etc/gitlab/registry-certs/registry-auth.crt
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 9263121198918743687 (0x808d37a6fcf13687)
Signature Algorithm: sha1WithRSAEncryption
Issuer: CN=gitlab-issuer
Validity
Not Before: Dec 27 18:22:54 2016 GMT
Not After : Dec 25 18:22:54 2026 GMT
Subject: CN=gitlab-issuer
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
Modulus:
00:df:a5:b0:f3:bc:f7:a2:5d:54:cb:e0:3c:e4:f4:
31:24:53:37:84:ab:4e:42:d9:5e:0d:98:65:ca:e7:
0f:f7:0c:41:d5:2c:0e:b0:02:e1:57:3a:df:10:03:
c7:8a:d0:56:c9:55:11:0a:bd:f7:a7:55:1d:fe:76:
f4:80:74:68:a0:fa:dd:86:6a:29:0f:2a:63:e7:62:
7d:20:b9:a4:78:ae:07:80:16:54:a5:72:80:4c:6b:
e6:62:4d:c7:46:21:14:16:45:c6:57:2b:b9:29:79:
c5:3c:cb:66:41:45:18:06:fb:48:62:ba:15:6f:66:
d0:e7:f3:5a:4f:3b:41:18:12:72:5f:08:b5:2f:6b:
3e:ac:39:ee:0f:69:1e:0e:33:35:e8:d1:d9:9c:c7:
e0:0b:5c:9b:d9:1c:9e:2d:f8:48:30:26:4d:b9:3a:
b8:94:c9:4f:45:45:95:11:0a:d8:07:26:ca:6e:94:
dd:4a:8a:6c:0c:2d:ea:2d:cd:fa:0c:3b:f7:aa:ef:
33:bf:5d:f1:ca:d0:14:c9:76:b9:ac:ca:bb:7d:d6:
99:fc:5f:aa:39:de:bb:e5:71:d4:80:4e:39:82:2b:
f0:35:a1:5e:88:26:c8:58:8a:37:91:c4:1c:1f:4b:
f8:92:71:a6:b3:d9:5b:1f:22:1c:90:c5:5b:c6:32:
a1:ef:1a:a0:a1:67:ee:89:b6:27:88:1f:9b:cd:e2:
cf:c8:af:9b:a8:86:8e:ae:2d:36:b7:71:ec:7a:40:
df:29:df:09:1d:d6:46:95:29:3d:00:6c:08:b7:dd:
56:b9:65:f8:1a:fc:ad:58:df:72:8d:4d:6b:68:67:
c0:c6:25:42:8b:06:65:99:8c:2f:76:ea:e2:c1:5a:
e3:04:9b:c1:1c:d2:96:f0:b2:f2:34:1e:87:a9:de:
29:50:35:9a:05:c9:e6:d8:4f:84:28:f5:60:51:a8:
aa:8e:93:b0:0f:a8:1e:52:c0:1b:42:27:70:1d:d6:
5b:8f:1e:df:59:82:0f:28:87:d4:2c:f2:39:3d:1e:
37:c9:23:75:e4:84:97:cb:a3:8a:81:c3:9d:0f:4b:
15:d2:d2:57:fa:f6:63:8c:fa:da:bc:5d:63:5b:19:
70:9f:a4:de:db:67:00:f8:a6:1a:e4:91:f0:38:ba:
e3:1c:c0:68:fd:cd:4a:86:55:74:32:d1:ba:30:8a:
a7:66:ee:50:a3:59:d1:9f:78:fa:a3:c1:4b:7a:5e:
61:72:a6:15:b5:98:f0:10:e7:b3:c1:d5:6f:e0:35:
da:98:ef:b2:da:78:a7:5b:79:e7:1e:ba:c6:45:b2:
51:28:00:1a:4c:3e:82:2d:0d:8e:1a:61:a2:5b:0e:
4f:17:55
Exponent: 65537 (0x10001)
Signature Algorithm: sha1WithRSAEncryption
6e:36:42:3d:ca:ab:4b:e3:bd:7a:be:49:af:f8:2b:27:5a:99:
87:ab:31:22:fa:25:c2:43:2d:c7:cf:28:26:00:a2:95:bf:43:
d8:94:cd:f6:a5:4d:d0:43:ff:c2:ad:82:38:fb:98:b1:63:31:
32:6c:9b:8c:4e:f2:30:b7:cf:c8:88:ef:ce:de:6f:8e:8a:b3:
62:32:e9:31:62:a6:e3:89:bb:9a:3c:77:e0:c4:f9:8b:da:b9:
94:e9:86:14:8b:54:38:b6:df:9a:0d:dc:c6:51:cb:c7:ae:57:
fb:95:4a:21:aa:ca:ec:bf:4b:b6:4e:3f:f4:2a:29:f9:af:24:
aa:fd:ac:f9:5a:47:e2:62:29:8e:5c:b5:89:69:9f:9e:82:5f:
3f:ec:4f:72:e0:aa:72:03:5f:74:4c:01:a8:0b:f7:00:bd:a6:
47:43:c5:11:d7:c6:02:a8:89:66:d8:ca:68:6b:a8:71:df:29:
40:a1:76:35:17:3a:b2:1a:cb:3e:59:55:1b:fc:f4:a9:5c:a4:
1f:eb:cc:7f:e8:c0:cd:92:3c:53:57:50:54:36:b6:25:3a:e6:
43:d4:a4:cb:93:2d:5b:d1:8b:37:57:98:5f:2e:5f:4e:cf:a4:
a4:e5:7a:ef:dc:be:d5:bc:3a:41:3a:21:ea:f3:01:06:c5:20:
8b:be:67:c3:c9:4a:02:94:91:82:96:fa:25:ed:e9:a9:87:d3:
25:e3:27:a5:1a:b0:64:dc:62:b3:f6:80:12:25:0a:13:35:ec:
44:e8:d0:82:0d:aa:8d:67:a1:cf:5b:4f:88:23:f2:94:2f:78:
18:04:07:25:29:6b:79:59:37:3b:db:8c:7e:c2:92:f7:6f:0e:
45:96:1e:63:ab:93:24:ce:46:4d:e2:be:5a:4a:f6:0a:4a:80:
3b:a3:2e:cf:a9:89:b4:2b:a6:94:c4:83:bd:77:d5:ae:3b:e6:
87:4b:7b:06:4e:86:aa:0f:8c:0f:a8:e4:e0:75:d5:27:2e:ef:
8f:ff:36:46:f4:14:32:55:63:91:e4:c6:7c:e8:da:61:31:f3:
f3:ab:db:d8:ac:b6:40:b4:db:73:ea:3a:c7:7d:6b:2c:18:42:
28:7e:01:0d:64:80:d6:21:62:fa:5c:b0:32:a5:d6:82:d4:c8:
09:c3:94:ea:0d:f6:10:85:b0:73:b4:1e:cb:3d:67:1a:e4:58:
ac:88:ea:86:b6:fc:fc:03:d5:4d:20:62:de:c5:d0:b3:3a:da:
11:dd:a9:99:e6:36:9f:d6:ed:7c:ae:a5:2b:22:01:dc:6a:32:
bd:11:35:78:0b:cf:f8:c2:78:85:bb:b7:b9:a9:d9:0c:71:8e:
6b:d5:64:d5:07:e1:1d:8e
-----BEGIN CERTIFICATE-----
MIIErDCCApQCCQCAjTem/PE2hzANBgkqhkiG9w0BAQUFADAYMRYwFAYDVQQDDA1n
aXRsYWItaXNzdWVyMB4XDTE2MTIyNzE4MjI1NFoXDTI2MTIyNTE4MjI1NFowGDEW
MBQGA1UEAwwNZ2l0bGFiLWlzc3VlcjCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCC
AgoCggIBAN+lsPO896JdVMvgPOT0MSRTN4SrTkLZXg2YZcrnD/cMQdUsDrAC4Vc6
3xADx4rQVslVEQq996dVHf529IB0aKD63YZqKQ8qY+difSC5pHiuB4AWVKVygExr
5mJNx0YhFBZFxlcruSl5xTzLZkFFGAb7SGK6FW9m0OfzWk87QRgScl8ItS9rPqw5
7g9pHg4zNejR2ZzH4Atcm9kcni34SDAmTbk6uJTJT0VFlREK2Acmym6U3UqKbAwt
6i3N+gw796rvM79d8crQFMl2uazKu33Wmfxfqjneu+Vx1IBOOYIr8DWhXogmyFiK
N5HEHB9L+JJxprPZWx8iHJDFW8Yyoe8aoKFn7om2J4gfm83iz8ivm6iGjq4tNrdx
7HpA3ynfCR3WRpUpPQBsCLfdVrll+Br8rVjfco1Na2hnwMYlQosGZZmML3bq4sFa
4wSbwRzSlvCy8jQeh6neKVA1mgXJ5thPhCj1YFGoqo6TsA+oHlLAG0IncB3WW48e
31mCDyiH1CzyOT0eN8kjdeSEl8ujioHDnQ9LFdLSV/r2Y4z62rxdY1sZcJ+k3ttn
APimGuSR8Di64xzAaP3NSoZVdDLRujCKp2buUKNZ0Z94+qPBS3peYXKmFbWY8BDn
s8HVb+A12pjvstp4p1t55x66xkWyUSgAGkw+gi0NjhpholsOTxdVAgMBAAEwDQYJ
KoZIhvcNAQEFBQADggIBAG42Qj3Kq0vjvXq+Sa/4KydamYerMSL6JcJDLcfPKCYA
opW/Q9iUzfalTdBD/8Ktgjj7mLFjMTJsm4xO8jC3z8iI787eb46Ks2Iy6TFipuOJ
u5o8d+DE+YvauZTphhSLVDi235oN3MZRy8euV/uVSiGqyuy/S7ZOP/QqKfmvJKr9
rPlaR+JiKY5ctYlpn56CXz/sT3LgqnIDX3RMAagL9wC9pkdDxRHXxgKoiWbYymhr
qHHfKUChdjUXOrIayz5ZVRv89KlcpB/rzH/owM2SPFNXUFQ2tiU65kPUpMuTLVvR
izdXmF8uX07PpKTleu/cvtW8OkE6IerzAQbFIIu+Z8PJSgKUkYKW+iXt6amH0yXj
J6UasGTcYrP2gBIlChM17ETo0IINqo1noc9bT4gj8pQveBgEByUpa3lZNzvbjH7C
kvdvDkWWHmOrkyTORk3ivlpK9gpKgDujLs+pibQrppTEg7131a475odLewZOhqoP
jA+o5OB11Scu74//Nkb0FDJVY5Hkxnzo2mEx8/Or29istkC023PqOsd9aywYQih+
AQ1kgNYhYvpcsDKl1oLUyAnDlOoN9hCFsHO0Hss9ZxrkWKyI6oa2/PwD1U0gYt7F
0LM62hHdqZnmNp/W7XyupSsiAdxqMr0RNXgLz/jCeIW7t7mp2QxxjmvVZNUH4R2O
-----END CERTIFICATE----- I have copied Docker logs: WARN[0045] error authorizing context: authorization token required go.version=go1.6.3 http.request.host=registry.anuary.com http.request.id=a9ac3795-d9ac-40ce-9cd4-07f29b77843c http.request.method=GET http.request.remoteaddr=78.61.210.113:57715 http.request.uri=/v2/ http.request.useragent=docker/1.12.5 go/go1.6.4 git-commit/7392c3b kernel/4.4.39-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \(darwin\)) instance.id=57d8be55-05d3-4323-8e79-0314386459af version=v2.5.1
78.61.210.113 - - [27/Dec/2016:18:29:31 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.12.5 go/go1.6.4 git-commit/7392c3b kernel/4.4.39-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(darwin\\))"
- ERRO[0046] token signed by untrusted key with ID: "7K7M:VDNO:YAOL:X4ST:E2XK:HYLF:ROY4:7L7L:SCLH:46TJ:W3LZ:FJCH"
- WARN[0046] error authorizing context: invalid token go.version=go1.6.3 http.request.host=registry.anuary.com http.request.id=3c23c782-643e-4ddc-9871-533db0f11d51 http.request.method=GET http.request.remoteaddr=78.61.210.113:57717 http.request.uri=/v2/ http.request.useragent=docker/1.12.5 go/go1.6.4 git-commit/7392c3b kernel/4.4.39-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \(darwin\)) instance.id=57d8be55-05d3-4323-8e79-0314386459af version=v2.5.1
78.61.210.113 - - [27/Dec/2016:18:29:32 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.12.5 go/go1.6.4 git-commit/7392c3b kernel/4.4.39-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.5 \\(darwin\\))" Auth server logs:
What am I possibly missing? How do I further debug this issue? |
I chased my tail on this for days as well. ROOTCERTBUNDLE is not the certificate bundle bundle, it's confusingly named. It needs to be the certificate of a signed key pair, in my experience CA certificates won't work. Who signs the certificate doesn't matter and isn't checked, but it must be signed. I went with a self signed certificate because the CA doesn't matter. @gajus If I'm reading your post correctly I think the signing might be the issue. This is the command I used to generate the key/cert I'm using for Portus:
Give that a shot. It would be nice if Docker could use an existing JWT library instead of rolling their own, I believe most of this difficulty is coming from the implementation. |
It works. First of all, thank you @CVTJNII for contributing the solution. However, I don't understand whats the difference between: openssl req -nodes -newkey rsa:4096 -keyout registry-auth.key -out registry-auth.crt -days 3650 -x509 -subj "/CN=gitlab-issuer" and openssl req -nodes -newkey rsa:4096 -keyout registry-auth.key -out registry-auth.csr -subj "/CN=gitlab-issuer"
openssl x509 -in registry-auth.csr -out registry-auth.crt -req -signkey registry-auth.key -days 3650 Aren't they supposed to generate exactly the same thing? |
Comparing the output of those two sets of commands the only thing that jumps out at me is the first does produce a CA certificate. So it looks like to work the certificate needs to be a CA cert signed by the key but not an intermediary CA cert? After figuring this out myself I started to look into why this is so fussy, and in my opinion it's due to the Docker JWT implementation handling SSL itself and not using a JWT library resulting in implementation eccentricities. If nothing else the registry seems to be requiring a very specific type of certificate (too specific, in my opinion) and it isn't clearly documented what those requirements are. |
Docker distribution (at least as of v2.3.1) requires the full bundle (not just the public certificate). This is unclear in the documentation, as well as, I believe, an unnecessary requirement, since the public certificate alone is sufficient to verify a token's signature. Playing with a running v2 registry using token authentication, a signed token from the token server (in this case, Portus), and the jwt debugger at https://jwt.io I spent some time trying to understand how the token signature was verified. The registry and portus server both use a provided crt file for verification/signing when deploying Portus with docker-compose. However, I was initially unable to verify the signature with the tool, using both the portus.crt file linked above, as well as the public key generated from it with openssl tools. Here's a sample token:
and decoded:
After some digging, I came across https://docs.docker.com/registry/spec/auth/jwt/#getting-a-bearer-token, which outlines the requirements for "kid" header generation, and it occurred to me to generate the public key in DER format: However, attempting to start the registry using a
I also tried this with a pem-formatted pubkey as well as a der-formatted crt file, but neither worked. I'm struggling to see why the full crt should be required here when just the pubkey would be sufficient. |
I am trying to setup token authentication using the
cesanta/docker_auth
token authentication server. I am constantly getting an error in the log for my registry that indicates the token is signed by an untrusted key:I believe I have all of the configuration settings correct when running the registry:
The HTTP_TLS_CERTIFICATE and HTTP_TLS_KEY are issued by GoDaddy.com, and are working correctly for HTTPS. I have specified their root certificate bundle in AUTH_TOKEN_ROOTCERTBUNDLE, which is what is supposed to be necessary to validate the token issued correct?
For the authentication server, this is the configuration I am using (https://github.com/cesanta/docker_auth/blob/master/examples/simple.yml). Referencing the same certificate and private key that I am using for HTTPS:
Does AUTH_TOKEN_ROOTCERTBUNDLE need to be something else?
The text was updated successfully, but these errors were encountered: