Use ServiceAccountToken volumes #3260

grampelberg · 2019-08-14T21:11:40Z

What problem are you trying to solve?

Currently, service account tokens are used to validate identity for pods. These are passed as part of the CSR to identity. In situations where the token is not mounted automatically, a pod's identity cannot be verified and identity must be disabled. This token also is needlessly over permissive and shouldn't be shipped around.

How should the problem be solved?

A new type of volume - ServiceAccountToken reached beta in k8s 1.12. This is mounted on a per-container basis, can have a specific expiration and allows restriction by audience. Instead of relying on the default service account token to be auto-mounted, injection should add the volume to the proxy's pod and restrict to exactly the audience required.

This should not be a configuration option and instead be the only way to view service account tokens moving forward.

Concerns

While this feature reached beta in k8s 1.12, it is unclear that it is available in most cloud providers and local solutions (docker desktop, minikube). Before implementing, support for at least GKE, AKS, EKS, minikube and docker desktop should be validated.

The text was updated successfully, but these errors were encountered:

grampelberg · 2019-08-14T21:17:28Z

Related to #2843 and #3183.

alpeb · 2019-08-26T22:52:48Z

As for provider support, this is supported wherever we can specify the --service-account-issuer, service-account-signing-key-file and --service-account-api-audiences flags to the kube-apiserver:

kind

Supported, although not enabled by default. One needs to pass a config file such as:

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatches:
  - |
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterConfiguration
    metadata:
      name: config
    apiServer:
      extraArgs:
        "service-account-issuer": "kubernetes.default.svc"
        "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"

Minikube

Supported, although not enabled by default. Minikube needs to be started with those flags enabled:

minikube start --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key --extra-config=apiserver.service-account-issuer=kubernetes/serviceaccount --extra-config=apiserver.service-account-api-audiences=api

Microk8s

Supported, although not enabled by default. The flags need to be set in /var/snap/microk8s/current/args/kube-apiserver:

...
--service-account-issuer=api
--service-account-signing-key-file=${SNAP_DATA}/certs/server.key
--service-account-api-audiences=api

Docker Desktop

Supported, although not enabled by default. To enable first open a session against Docker's tty:

screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

Then edit the kube-apiserver config:

vi /etc/kubernetes/manifests/kube-apiserver.yaml

and append the flags to the spec.containers.command section:

    - --service-account-issuer=api
    - --service-account-signing-key-file=/run/config/pki/apiserver.key
    - --service-account-api-audiences=api

Close the session with ctrl-a :quit and restart Docker.

GKE

Supported, and enabled by default!

AKS

Not supported and there's no way to set it up.

EKS

Not supported and there's no way to set it up. Not through the console nor through eksctl.

Edit: Completed Docker Desktop section
Edit: Updated command for Minikube, according to this comment

grampelberg · 2019-08-27T16:38:35Z

Whelp, this sucks. Should we add a config option just for GKE or shelve this for now?

grampelberg · 2019-08-27T20:23:34Z

I'm moving out of the release for now. Once this is more widely adopted it'd be great to do.

reegnz · 2019-10-29T15:55:39Z

AWS EKS now supports this by default since early september with kubernetes 1.14 and they encourage using projected service account tokens for IAM Role assumption.
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

As for AKS, let's see how this issue gets resolved: Azure/AKS#1288 Azure/AKS#1208

stale · 2020-01-27T16:45:42Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

reegnz · 2020-01-28T21:14:01Z

AKS ETA is early February Azure/AKS#1208 (comment) So is support in GKE+EKS+AKS enough to have a go at this feature?

grampelberg · 2020-01-28T21:15:35Z

@reegnz AKS has support now?

Given the lack of support in minikube/microk8s, we'll need to degrade gracefully no matter what. Interested in taking up the contribution?

reegnz · 2020-01-28T21:17:19Z

@grampelberg I'm not that proficient in go yet, so I'll have someone else pick this up. :)
I am very enthusiastic about having this as a feature though!

irizzant · 2020-03-26T12:51:47Z

Minikube won't start with:
minikube start --extra-config=apiserver.service-account-issuer=api --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/apiserver.key --extra-config=apiserver.service-account-api-audiences=api

at least not version 1.8.2
You need to use:
minikube start --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key --extra-config=apiserver.service-account-issuer=kubernetes/serviceaccount --extra-config=apiserver.service-account-api-audiences=api

alpeb · 2020-03-26T14:42:54Z

Thanks @irizzant 👍
I've updated the comment above.

stale · 2020-06-26T03:39:29Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

Olshansk · 2020-09-21T02:18:27Z

I recently came by the following solution for minikube:

export GOOGLE_APPLICATION_CREDENTIALS=<creds-path>.json 
minikube addons enable gcp-auth

I'm not sure it completely address the problems here but figured it's worth mentioning.

Reference: https://minikube.sigs.k8s.io/docs/handbook/addons/gcp-auth/

taman9333 · 2021-01-13T21:12:59Z

Hello @alpeb there's a type in Docker Desktop which will make cluster stuck starting...
this

- --service-account-signing-key-file=/var/run/config/pki/apiserver.key

should be

- --service-account-signing-key-file=/run/config/pki/apiserver.key

alpeb · 2021-01-14T15:59:38Z

Thanks @taman9333! I've updated the comment above 👍

Pothulapati · 2021-10-14T11:59:53Z

Thanks for the patience everyone!

With this being available by default in all the major cloud providers i.e GKE, AKS, EKS and also in kind, We can start building this for Linkerd 2.12.0 IMO.

I started reading up more on this, and me or someone else will post up a separate design issue in the very near future!

Fixes #3260 ## Summary Currently, Linkerd uses a service Account token to validate a pod during the `Certify` request with identity, through which identity is established on the proxy. This works well and good, as Kubernetes attaches the `default` service account token of a namespace as a volume (unless overridden with a specific service account by the user). Catch here being that this token is aimed at the application to talk to the kubernetes API and not specifically for Linkerd. This means that there are [controls outside of Linkerd](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server), to manage this service token, which users might want to use, [causing problems with Linkerd](#3183) as Linkerd might expect it to be present. To have a more granular control over the token, and not rely on the service token that can be managed externally, [Bound Service Tokens](https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/1205-bound-service-account-tokens) can be used to generate tokens that are specifically for Linkerd, that are bound to a specific pod, along with an expiry. ## Background on Bounded Service Tokens This feature has been GA’ed in Kubernetes 1.20, and is enabled by default in most cloud provider distributions. Using this feature, Kubernetes can be asked to issue specific tokens for linkerd usage (through audience bound configuration), with a specific expiry time (as the validation happens every 24 hours when establishing identity, we can follow the same), bounded to a specific pod (meaning verification fails if the pod object isn’t available). Because of all these bounds, and not being able to use this token for anything else, This feels like the right thing to rely on to validate a pod to issue a certificate. ### Pod Identity Name We still use the same service account name as the pod identity (used with metrics, etc) as these tokens are all generated from the same base service account attached to the pod (could be defualt, or the user overriden one). This can be verified by looking at the `user` field in the `TokenReview` response. <details> <summary>Sample TokenReview response</summary> Here, The new token was created for the vault audience for a pod which had a serviceAccount token volume projection and was using the `mine` serviceAccount in the default namespace. ```json "kind": "TokenReview", "apiVersion": "authentication.k8s.io/v1", "metadata": { "creationTimestamp": null, "managedFields": [ { "manager": "curl", "operation": "Update", "apiVersion": "authentication.k8s.io/v1", "time": "2021-10-19T19:21:40Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec":{"f:audiences":{},"f:token":{}}} } ] }, "spec": { "token": "....", "audiences": [ "vault" ] }, "status": { "authenticated": true, "user": { "username": "system:serviceaccount:default:mine", "uid": "889a81bd-e31c-4423-b542-98ddca89bfd9", "groups": [ "system:serviceaccounts", "system:serviceaccounts:default", "system:authenticated" ], "extra": { "authentication.kubernetes.io/pod-name": [ "nginx" ], "authentication.kubernetes.io/pod-uid": [ "ebf36f80-40ee-48ee-a75b-96dcc21466a6" ] } }, "audiences": [ "vault" ] } ``` </details> ## Changes - Update `proxy-injector` and install scripts to include the new projected Volume and VolumeMount. - Update the `identity` pod to validate the token with the linkerd audience key. - Added `identity.serviceAccountTokenProjection` to disable this feature. - Updated err'ing logic with `autoMountServiceAccount: false` to fail only when this feature is disabled. Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com>

grampelberg added priority/P0 Release Blocker area/identity Automatic transport identity area/inject labels Aug 14, 2019

ihcsim mentioned this issue Aug 21, 2019

Handle automountServiceAccountToken #3183

Closed

grampelberg removed the priority/P0 Release Blocker label Aug 27, 2019

stale bot added the wontfix label Jan 27, 2020

stale bot removed the wontfix label Jan 28, 2020

nepalineupane mentioned this issue Feb 28, 2020

MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource kubeflow/manifests#959

Closed

stale bot added the wontfix label Jun 26, 2020

olix0r added pinned needs/design and removed wontfix labels Jun 26, 2020

BenTheElder mentioned this issue Dec 3, 2020

TokenRequest feature not enabled by default kubernetes-sigs/kind#1954

Closed

adleong added this to the stable-2.12.0 milestone Oct 14, 2021

Pothulapati mentioned this issue Oct 19, 2021

core: use serviceAccountToken volume for pod authentication #7117

Merged

Pothulapati closed this as completed in #7117 Nov 2, 2021

github-actions bot locked as resolved and limited conversation to collaborators Dec 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use ServiceAccountToken volumes #3260

Use ServiceAccountToken volumes #3260

grampelberg commented Aug 14, 2019

grampelberg commented Aug 14, 2019

alpeb commented Aug 26, 2019 •

edited

Loading

grampelberg commented Aug 27, 2019

grampelberg commented Aug 27, 2019

reegnz commented Oct 29, 2019 •

edited

Loading

stale bot commented Jan 27, 2020

reegnz commented Jan 28, 2020 •

edited

Loading

grampelberg commented Jan 28, 2020

reegnz commented Jan 28, 2020

irizzant commented Mar 26, 2020 •

edited

Loading

alpeb commented Mar 26, 2020

stale bot commented Jun 26, 2020

Olshansk commented Sep 21, 2020

taman9333 commented Jan 13, 2021

alpeb commented Jan 14, 2021

Pothulapati commented Oct 14, 2021

Use ServiceAccountToken volumes #3260

Use ServiceAccountToken volumes #3260

Comments

grampelberg commented Aug 14, 2019

What problem are you trying to solve?

How should the problem be solved?

Concerns

grampelberg commented Aug 14, 2019

alpeb commented Aug 26, 2019 • edited Loading

kind

Minikube

Microk8s

Docker Desktop

GKE

AKS

EKS

grampelberg commented Aug 27, 2019

grampelberg commented Aug 27, 2019

reegnz commented Oct 29, 2019 • edited Loading

stale bot commented Jan 27, 2020

reegnz commented Jan 28, 2020 • edited Loading

grampelberg commented Jan 28, 2020

reegnz commented Jan 28, 2020

irizzant commented Mar 26, 2020 • edited Loading

alpeb commented Mar 26, 2020

stale bot commented Jun 26, 2020

Olshansk commented Sep 21, 2020

taman9333 commented Jan 13, 2021

alpeb commented Jan 14, 2021

Pothulapati commented Oct 14, 2021

alpeb commented Aug 26, 2019 •

edited

Loading

reegnz commented Oct 29, 2019 •

edited

Loading

reegnz commented Jan 28, 2020 •

edited

Loading

irizzant commented Mar 26, 2020 •

edited

Loading