istioctl JWT tokens to contact istiod

[esnible]

Istioctl 1.8.x will still use --cert-dir but we will not work on improving the Makefile cert mining.

Instead, we will allow Istioctl to request and check JWT tokens from Kubernetes.
The tokens will be cached on the user's file system.

[esnible]

@howardjohn suggests using a token rather than a cert.

[myidpt]

Are you trying to build a tool in istioctl for debugging? If so, I think we should define the right way to do it. It would be better that istiod generates the key/cert and writes it to some K8s resource, and RBAC on that resource is in place to protect it. Using K8s token also works with RBAC protection. The most important thing is Istiod should authenticate and authorize the caller.

[esnible]

To get certificates that work with the latest master:

CTX=...
PREFIX=it
kubectl config use-context $CTX
BUILD_WITH_CONTAINER=0 make $PREFIX-certs-k8s
mkdir $PREFIX-istioctl
cp $PREFIX/root-cert.pem $PREFIX-istioctl/
cp $PREFIX/workload-cert-chain.pem $PREFIX-istioctl/cert-chain.pem
cp $PREFIX/key.pem $PREFIX-istioctl/key.pem
rm k8s-root-cert.pem k8s-root-cert.srl k8s-root-key.pem ${CTX/\//-}/*
rmdir ${CTX/\//-}

At this point, the certs should be good enough to use istioctl x version --cert-dir <user>.

[esnible]

In the future we want to use the mechanism proposed for #24008 and #22918 to get the cert.

[esnible]

@irisdingbj Is there an example of how VMs get and use tokens? I would like istioctl to be as close to a sidecar as possible in implementation.

[irisdingbj]

@esnible as for as I know current VM doc is based on client cert. There are some discussions to make VM use token to bootstrap but I am not keeping close track of it.

Back to token for istioctl, you can use the default token for a namespace directly or use TokenRequest from k8s to create one and use it.

[esnible]

The token request:

echo '{"kind":"TokenRequest","apiVersion":"authentication.k8s.io/v1","spec":{"audiences":["istio-ca"], "expirationSeconds":2592000}}' | \
   kubectl create --raw /api/v1/namespaces/default/serviceaccounts/default/token -f - | \
   jq -j '.status.token' > ./var/run/secrets/tokens/istio-token

gets a token. replace default and default with relevant NS and SA.

[howardjohn]

One note - the expiration time on that request is massive. If we are doing this in istioctl or something, should likely be much smaller. Great for testing though

…

On Tue, Aug 18, 2020 at 4:25 PM Ed Snible ***@***.***> wrote: The token request: echo '{"kind":"TokenRequest","apiVersion":"authentication.k8s.io/v1","spec":{"audiences":["istio-ca"], "expirationSeconds":2592000}}' | kubectl create --raw /api/v1/namespaces/default/serviceaccounts/default/token -f - | jq -j '.status.token' > ./var/run/secrets/tokens/istio-token gets a token. replace default and default with relevant NS and SA. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24905 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXKW2TIOYFZ7QDKJQ53SBMEWDANCNFSM4OF3TZYA> .

[esnible]

@howardjohn Understood about the expiration time. The example is 720 days. What do you think would be reasonable for Istio?

• Do you envision istioctl keeping the certificate around for re-use? If so, 12 hours seems reasonable.
• or should it be one-time? If so, the command --duration or 5 seconds seems reasonable.

Question about how to use it. The ADSC functions seem to need both a JWT Token and a istio.io/istio/pkg/security.SecretManager. I see _test.go examples and one in security.pkg.NodeAgent. How do I get started learning this piece of the code?

[howardjohn]

how kubectl works (at least with GKE) is it sticks the token in kubeconfig and it expires in 60 minutes. kubectl transparently refreshes it if its near expiration.

For ADSC: Currently it uses the jwt token and exchanges it for a certificate, which it then sends. This is because we used to only support cert auth. Now XDS can read jwt directly, so you can just pass the jwt as part of the gRPC request. In grpcurl this is like -insecure -rpc-header "authorization: Bearer $token", probably some similar option in go code

[esnible]

@howardjohn It isn't working for me.

NAMESPACE=default
SVCACCT=default
TOKEN=$(echo '{"kind":"TokenRequest","apiVersion":"authentication.k8s.io/v1","spec":{"audiences":["istio-ca"], "expirationSeconds":2592000}}' | kubectl create --raw /api/v1/namespaces/$NAMESPACE/serviceaccounts/$SVCACCT/token -f - | jq -j '.status.token')
grpcurl -d @ -insecure -rpc-header "authorization: Bearer $TOKEN" localhost:15012 envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources < /tmp/connreq.json
ERROR:
  Code: Unknown
  Message: authentication failure

Istiod logs 2020-08-19T15:57:33.353725Z error ads Failed to authenticate client from 127.0.0.1:33542 Authenticator ClientCertAuthenticator: no verified chain is found.

[howardjohn]

@esnible it works for me, are you sure you are using the latest code? it was merged a couple days ago

[esnible]

For 1.8, it has been suggested we use JWT tokens.

• create a medium-lived (hour) token and store it in ~/.istio/ when the user supplies neither --cert-dir nor --plaintext to an XDS-based command?

Probably we need a way to keep multiple tokens around for users with >1 mesh. I propose we use --token-file to override the token location. (The default for --token-file will be _$HOME/.istio/token`)