[istio.io testing] Need test for Provisioning Identity through SDS

[brian-avery]

Describe the feature request
We need to develop a test for the Provisioning Identity through SDS task in order to provide automated testing of the Istio.io website for future releases.

Example tests and the documentation for the framework can be found here: https://github.com/istio/istio/tree/master/tests/integration/istioio

A YouTube video describing the usage of the framework can be found here: https://www.youtube.com/watch?v=3y-z8NaVwr0

Please reach out to the Test and Release channel on the Istio Slack or Test and Release Discuss with any questions.

[ ] Configuration Infrastructure
[ x] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ x] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Additional context

[su225]

I'll take this one.

[brian-avery]

Thanks @su225

[Pawan-Bishnoi]

@su225 Have you started working on this. I would like to help here (or just follow along if you don't want help)
Looking for a place to start. Thanks!

[su225]

@Pawan-Bishnoi - Yes. I have started working on this. There are lots of [istio.io testing] issues here. You can pick any of them and start working.

[su225]

@brian-avery - Some issues regarding environment setup for SDS. I'm blocked. I dug through a lot, but keep finding new feature gates to enable. I found 4 so far. So any clue how to fix this one? This is blocking my way.

Here is what I did so far
I compared the default/demo profile with sds profile of Istio with istio profile diff and set those Helm values

func enableSDSIdProvisioning(cfg *istio.Config) {
	if cfg == nil {
		return
	}
	cfg.Values["global.sds.enabled"] = "true"
	cfg.Values["global.sds.udsPath"] = "unix:/var/run/sds/uds_path"
	cfg.Values["global.sds.token.aud"] = "istio-ca"

	cfg.Values["global.mtls.enabled"] = "true"

	cfg.Values["nodeagent.enabled"] = "true"
	cfg.Values["nodeagent.image"] = "node-agent-k8s"
	cfg.Values["nodeagent.env.CA_PROVIDER"] = "Citadel"
	cfg.Values["nodeagent.env.CA_ADDR"] = "istio-citadel.istio-system:8060"
	cfg.Values["nodeagent.env.VALID_TOKEN"] = "true"

	cfg.Values["security.components.nodeAgent.enabled"] = "true"
}

Pilot, Ingress-gateway and others were stuck in ContainerCreating state forever. Here is the error I found

Unable to attach or mount volumes: unmounted volumes=[istio-token], 
unattached volumes=[istio-pilot-service-account-token-xdzjz sds-uds-path 
istio-token config-volume istio-certs]: timed out waiting for the condition

Even before that it complained that TokenEndpoint feature gate was not enabled. So i enabled them by passing --feature-gates=TokenEndpoint=true,TokenEndpointProjection=true flags to minikube start. I can also see them passed to kube-apiserver. In spite of it no luck

kube-apiserver
  --advertise-address=192.168.0.122
  --allow-privileged=true
  --authorization-mode=Node,RBAC
  --client-ca-file=/var/lib/minikube/certs/ca.crt
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
  --enable-bootstrap-token-auth=true
  --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt
  --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt
  --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key
  --etcd-servers=https://127.0.0.1:2379
+ --feature-gates=TokenRequest=true,TokenRequestProjection=true
  --insecure-port=0
  --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt
  --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt
  --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key
  --requestheader-allowed-names=front-proxy-client
  --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt
  --requestheader-extra-headers-prefix=X-Remote-Extra-
  --requestheader-group-headers=X-Remote-Group
  --requestheader-username-headers=X-Remote-User
  --secure-port=8443
  --service-account-key-file=/var/lib/minikube/certs/sa.pub
  --service-cluster-ip-range=10.96.0.0/12
  --tls-cert-file=/var/lib/minikube/certs/apiserver.crt
  --tls-private-key-file=/var/lib/minikube/certs/apiserver.key

I then started examining the secret on which projection is being done. An example JWT looks like this

{
  "alg": "RS256",
  "kid": "yo-oRLZMTR6xrPyE8cFCuAiMiSVwssAgiF6YbgRAs2M"
}
{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "istio-system",
  "kubernetes.io/serviceaccount/secret.name": "istio-pilot-service-account-token-stxdp",
  "kubernetes.io/serviceaccount/service-account.name": "istio-pilot-service-account",
  "kubernetes.io/serviceaccount/service-account.uid": "ce257494-652f-4f04-b4ac-e85dffc9a66e",
  "sub": "system:serviceaccount:istio-system:istio-pilot-service-account"
}

It does not look like what the projection YAML expects

But the projection YAML for pilot looks like this

- name: istio-token
  projected:
    sources:
    - serviceAccountToken:
        audience: istio-ca
        expirationSeconds: 43200
        path: istio-token

So I started digging into "service account token projection" and Google took me to Kubernetes documentation page here

To summarize, it mentions one more feature gate called ServiceAccountTokenVolumeProjection to be enabled. Besides that, it also mentions some more parameter to be passed to kube-apiserver namely

• --service-account-issuer
• --service-account-signing-key-file
• --service-account-api-audiences

There is yet another link to Istio blog titled Change in Secret Discovery Service in Istio 1.3 which talks about introducing audience fields to JWT tokens to limit its scope. I see that this is biting me. While digging further in that blog, I ran into another feature gate named BoundServiceAccountTokenVolume which is still in alpha and is off by default.

[su225]

Ah! Found something here - https://github.com/istio/istio/blob/master/prow/config/trustworthy-jwt.yaml
Let me try and come back to you.

[su225]

Finally it worked. I had to add the following extra configuration to Minikube to make it work

--extra-config=apiserver.service-account-signing-key-file="/var/lib/minikube/certs/sa.key" \
--extra-config=apiserver.service-account-issuer="kubernetes.default.svc" \
--extra-config=kubelet.resolv-conf="/run/systemd/resolve/resolv.conf"

I guess for most people only the first two are relevant. My CoreDNS was stuck in forwarding loop and hence the third one.

My bad! Never knew that the fix is easy. 🤦‍♂️

[su225]

@brian-avery - It seems that the link is broken. I checked in https://preliminary.istio.io/docs/tasks/security/ and it does not seem to exist. I heard that SDS is enabled by default in after Istio moved to using Operators instead of Helm charts. Is that correct?

By the way, if this issue is outdated, can we close it?

[brian-avery]

It looks like this was removed Friday as part of #6644. Unfortunately, I don't think this test is needed anymore and that this issue should be closed.