Verifying a Kubernetes serviceaccount token with step

[olix0r]

I'm trying to use step to verify a Kubernetes ServiceAccount token.

Provision a Service Account

:; kubectl create serviceaccount trixrabbit
serviceaccount "trixrabbit" created

:; kubectl get -o json sa/trixrabbit
{
    "apiVersion": "v1",
    "kind": "ServiceAccount",
    "metadata": {
        "creationTimestamp": "2019-01-23T23:52:59Z",
        "name": "trixrabbit",
        "namespace": "default",
        "resourceVersion": "3745182",
        "selfLink": "/api/v1/namespaces/default/serviceaccounts/trixrabbit",
        "uid": "033b7edb-1f6a-11e9-98ce-80fa5b5b38db"
    },
    "secrets": [
        {
            "name": "trixrabbit-token-7ln2l"
        }
    ]
}

Obtain the credentials

:; token=$(kubectl get -o json sa/trixrabbit |jq -r '.secrets[0].name')
:; kubectl get -o json secret $token |jq -r '.data.token' |base64 -d >trixrabbit.jwt
:; kubectl get -o json secret $token |jq -r '.data["ca.crt"]' |base64 -d >trixrabbit.ca.crt

Inspect the credentials

; step crypto jwt inspect --insecure <trixrabbit.jwt |jq .                              
{
  "header": {
    "alg": "RS256",
    "kid": ""
  },
  "payload": {
    "iss": "kubernetes/serviceaccount",
    "kubernetes.io/serviceaccount/namespace": "default",
    "kubernetes.io/serviceaccount/secret.name": "trixrabbit-token-7ln2l",
    "kubernetes.io/serviceaccount/service-account.name": "trixrabbit",
    "kubernetes.io/serviceaccount/service-account.uid": "033b7edb-1f6a-11e9-98ce-80fa5b5b38db",
    "sub": "system:serviceaccount:default:trixrabbit"
  },
  "signature": "AgDNyeFlVlQBmBC32iCCMUSF1AGU_GxehYQmY_P-Jh4N3rD33MUSoWe0SLUa7Sl7yPXK-HpdKMLuerlmghgVp9SAnuSO1QQD9Hh0jHfJZCEmfWtEVItHeBEbV6on15wRAEs75HFfjGHnShrdgwZHYxupMUNt84f5KHYC3U3VfMtKx9X7xkwqll9oC0q-bGsY4X4Jxq45zFi7AXBwMbmcnu1U00M9dHfUBHQ2Z2Lti8N3Wt5PrpGC3-zSyfsw3U__VJDt_N7sMDkq7YmppL9IWYVeWbrFO2kl4xRXGuEEYfidTizJghOQ4adFFK5juKwbwvs7WrO7qOkFDg0_jDsQjQ"
}

:; step certificate inspect trixrabbit.ca.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12509686381743968844 (0xad9b557c2b64124c)
    Signature Algorithm: SHA256-RSA
        Issuer: CN=127.0.0.1
        Validity
            Not Before: Dec 3 23:00:39 2018 UTC
            Not After : Apr 20 23:00:39 2046 UTC
        Subject: CN=127.0.0.1
        Subject Public Key Info:
            Public Key Algorithm: RSA
                Public-Key: (2048 bit)
                Modulus:
                    b1:44:71:70:4d:96:ba:49:87:3d:5c:be:a0:61:e9:
                    40:e8:90:fa:f1:de:67:c8:6f:af:db:60:60:75:12:
                    96:94:70:1c:ce:51:9f:c0:05:4c:46:b9:b2:ed:59:
                    75:7f:53:7c:2a:bf:0e:ab:20:d9:ae:c2:d7:8a:a9:
                    ff:49:86:96:f2:0b:94:d1:b0:fc:d9:f4:25:8f:b2:
                    3d:cc:8b:4e:be:bf:36:35:24:47:da:18:4f:70:16:
                    73:8e:f3:a2:f0:f6:26:ae:d0:c5:db:40:2f:9c:60:
                    ba:e7:44:b5:66:99:48:a9:09:7e:d6:ab:a8:22:0f:
                    9b:de:cc:17:1f:2e:b6:74:83:48:8a:ee:60:f4:7d:
                    71:d5:f6:21:2f:9d:d1:7c:87:37:f1:63:f2:af:24:
                    6e:5a:fd:1c:a2:4e:af:d9:9b:85:ee:f3:66:6c:c8:
                    2b:dc:3e:d1:fe:61:70:07:ff:8b:4e:83:fb:73:a9:
                    44:98:81:b8:27:c2:57:a2:9f:a6:80:29:61:80:b9:
                    e5:e6:f1:d0:00:ec:d8:4b:ca:9b:67:83:53:1e:66:
                    b0:f1:06:69:23:56:e6:1b:91:25:79:73:85:9f:e0:
                    cd:48:14:67:51:9c:a8:e2:3d:c9:8d:96:fd:c1:97:
                    9b:4a:cd:89:4f:8e:b0:e0:d6:14:7c:a4:21:cd:fd:
                    fb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                E4:12:02:D3:F4:17:10:FA:63:EC:A3:11:DF:B0:A3:2B:35:E8:CF:F0
            X509v3 Authority Key Identifier:
                keyid:E4:12:02:D3:F4:17:10:FA:63:EC:A3:11:DF:B0:A3:2B:35:E8:CF:F0
            X509v3 Basic Constraints:
                CA:TRUE

    Signature Algorithm: SHA256-RSA
         42:72:61:72:2a:c8:a2:b7:33:1d:a5:32:f4:fa:8d:2b:de:9e:
         ff:7e:d3:a4:95:9b:96:aa:75:c5:3e:a0:4b:c0:d3:cd:7d:5f:
         60:a2:b5:1d:98:c9:1f:a1:f8:fe:3b:18:6d:b4:00:b4:87:97:
         cc:5b:c1:94:5e:8d:f8:87:81:d6:91:c7:eb:22:38:d2:f4:8c:
         4d:9a:ca:51:83:c9:c6:17:52:a4:0f:8e:c8:7e:84:eb:bb:ab:
         20:7e:22:e3:ee:c1:c3:95:10:13:ed:2e:59:01:57:60:78:3e:
         59:53:ff:53:0d:2d:c7:22:42:cf:5a:76:7b:38:ff:ba:28:2f:
         d8:54:06:38:62:23:5b:04:9b:cc:3a:3f:cf:e2:a9:a0:32:03:
         12:f2:74:38:11:22:29:92:ba:f4:93:67:11:81:8d:6a:29:fb:
         68:9e:37:60:f9:64:bc:9f:34:65:4f:1f:e7:34:19:f2:6c:02:
         aa:06:14:e1:e2:bc:c0:62:f9:c8:3a:9a:f0:20:cf:85:ec:4e:
         f7:50:fb:0a:36:94:c5:f1:9c:17:d5:02:18:fd:6f:d4:b6:65:
         98:88:cb:89:be:4a:f5:6a:e3:82:2f:3a:a8:c4:67:84:ec:11:
         8b:f4:b3:4b:94:bc:b5:94:f1:c4:19:62:88:be:0d:a0:d1:97:
         d1:ea:ce:b9

Problem: How to verify credentials?

I can't figure out how to use the step CLI to validate that the given jwt was signed such that it can be trusted via the ca.crt...

:; step crypto jwt verify --subtle --key trixrabbit.ca.crt --alg RS256 <trixrabbit.jwt 
alg 'RS256' is not compatible with the given key

(The --subtle flag isn't documented in --help output, so I don't know what I'm exactly specifying here)

Any suggestions?

Thanks.

[olix0r]

:; step version
Smallstep CLI/0.8.2 (darwin/amd64)
Release Date: 2019-01-07 12:05 UTC

[sourishkrout]

Thank you, @olix0r. I believe step crypto jwt verify expects the keys in JWK format. I'll get back to you shortly.

[maraino]

@olix0r right now we don't have a way to extract the public key from a certificate, but it's actually pretty simple to add, we will add it.

After my tests, I checked that that key is not the one that has to be used to verify the token. But you can find it in the kube-apiserver

$ kubectl -n kube-system -o json get pod kube-apiserver-docker-for-desktop | jq .spec.containers[0].command | grep service-account-key-file
  "--service-account-key-file=/run/config/pki/sa.pub",
$ kubectl -n kube-system exec -it kube-apiserver-docker-for-desktop -- cat /run/config/pki/sa.pub
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----

And then:

$ step crypto jwt verify --subtle --key /tmp/key --alg RS256 < /tmp/trixrabbit.jwt
{
  "header": {
    "alg": "RS256",
    "kid": ""
  },
  "payload": {
    "iss": "kubernetes/serviceaccount",
    "kubernetes.io/serviceaccount/namespace": "default",
    "kubernetes.io/serviceaccount/secret.name": "trixrabbit-token-c4kxv",
    "kubernetes.io/serviceaccount/service-account.name": "trixrabbit",
    "kubernetes.io/serviceaccount/service-account.uid": "41558afc-1f70-11e9-bff5-025000000001",
    "sub": "system:serviceaccount:default:trixrabbit"
  },
  "signature": "yCeSgyMAv8uOHrPNznl_jL2yRqhIl3I2-EZekbYNnfgewPUqt0jNq2KSGJY-6KazNbCvOaIFLPwqYySDPmAE_sB6mO41jrqYgf03EPQELpUnm-hcy6E1ebpqgK3mxf1mlrUAMtJYA2HnF4-CLEDVicwOI25s2Qaypm_1P0N2FDgOnB5r7mhGaNQtd04ndbo_7tjD4n4dY3xsTVIjGbBSm_OOR6Utx7cpY_7p_vjSPeMvDeTOztFcESy0NctCqqADHQW-8PP9cQqVD5Unt1sA6wq_l5hZAMIzvPlS5bCdX9A8PDFuSr8igih6K_H9_KIuy6f8nM5x7Cy9bnt23cdclg"
}

[mmalone]

The --subtle flag makes iss (issuer) and aud (audience) checks optional and allows JWTs that don't have an exp (expiration). Kubernetes doesn't include aud or exp (pretty sure this violates spec; definitely violates best practices) so you need to pass --subtle. It only makes flags optional and only disables checks if flags are omitted. So if you do pass --iss with --subtle, for example, it'll still be checked.

We punted on documenting this for the moment because it's really hard to explain when it's safe to skip these checks and when it's not. You have to go pretty deep on the semantics of JWT and JWS and various sorts of attacks in a particular context. The thinking is that you should only be skipping this stuff if you already know what you're doing. Unfortunately, k8s kinda screws with this product decision by making it a fairly common occurrence :/.

FWIW the TokenSigning API is supposed to fix this stuff.

[olix0r]

Thanks for the quick explanations!

[olix0r]

One more question. It's entirely possible this is pebkac or... a kubernetes issue?

For some reason, I'm still unable to use step to validate the token:

Setup, as before, but using new version

:; step version
Smallstep CLI/0.8.4-rc.2-6-ged3e464 (linux/amd64)
Release Date: 2019-01-26 16:54 UTC
:; kubectl delete sa/leaf ; kubectl create sa leaf                                                            
serviceaccount "leaf" deleted
serviceaccount "leaf" created
:; token=$(kubectl get -o json sa/leaf |jq -r '.secrets[0].name')
:; kubectl get -o json secrets $token |jq -r '.data.token' |base64 -d >leaf.jwt
:; kubectl get -o json secrets $token |jq -r '.data["ca.crt"]' |base64 -d >leaf.ca.crt
:; step certificate inspect leaf.ca.crt |head
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12509686381743968844 (0xad9b557c2b64124c)
    Signature Algorithm: SHA256-RSA
        Issuer: CN=127.0.0.1
        Validity
            Not Before: Dec 3 23:00:39 2018 UTC
            Not After : Apr 20 23:00:39 2046 UTC
        Subject: CN=127.0.0.1
:; step certificate key leaf.ca.crt >leaf.ca.pub
:; step crypto jwt inspect --insecure <leaf.jwt
{
  "header": {
    "alg": "RS256",
    "kid": ""
  },
  "payload": {
    "iss": "kubernetes/serviceaccount",
    "kubernetes.io/serviceaccount/namespace": "default",
    "kubernetes.io/serviceaccount/secret.name": "leaf-token-sh4ht",
    "kubernetes.io/serviceaccount/service-account.name": "leaf",
    "kubernetes.io/serviceaccount/service-account.uid": "a311fb16-218c-11e9-ae5e-80fa5b5b38db",
    "sub": "system:serviceaccount:default:leaf"
  },
  "signature": "Csa7PP_lx3moKUex5yOABMVT6GeIjdzC9Yt2nq-uounx5Lx32sdbfoaToPFdix1ATZ7nAg6MIdMBc3_vtnxWbqXIOMGoatBeJDKBvP0ZJ1LsVytUJnAfjZUtsnrW9nEe519bvLn_F6xnIO_JfGJnSZTofLHUq_9Gd7nq5c_a2kEFaVNZbpjJ0RbYN5B-rtvQ6IElRInZHNOUmW21QDDBnC-OyTEUf78fDvsFkQHGSd2oh5d5H0EHbTEwOtsCamRNh9Lhd9RBMdkjuoEZTj75xavQXB9PrHDapDv3hJzeqdLhwfl_SO0CzVpiUMRZTKtB38YEVg__muw6xEKKirpymA"
}

Invalid signature

:; step crypto jwt verify --subtle --alg RS256 --key leaf.ca.pub <leaf.jwt                                                       
validation failed: invalid signature

I observe this on both Linux (with minik8s) and Mac (with docker-desktop).

Am I doing something obviously wrong? Perhaps this ca.crt can't actually be used to validate the token's signature? I'll continue to investigate, but I'm open to suggestions as to how to debug this.

[olix0r]

Ah, now I understand @maraino's earlier comments after reading more about the Kubernetes configuration -- the key used to verify serviceaccount token is part of kube-api-server/kube-controller-manager's configuration; and, by default, the apiserver's private key is used. This has nothing to do with the ca.crt in the secret, which should be used to validate the apiserver's identity.