ServiceAccounts #7101

liggitt · 2015-04-21T08:41:27Z

This is a first pass at ServiceAccount objects and associated tokens.

Overview:

ServiceAccount API objects (and client, printer, describer, etc)
ServiceAccountToken secret type (and describer)
ServiceAccountTokensController to manage ServiceAccountToken secrets
- ensures a ServiceAccountToken secret exists for every ServiceAccount
- ensures ServiceAccountToken secrets are deleted when the corresponding ServiceAccount is deleted
- ensures the "token" key has a value set for every ServiceAccountToken
ServiceAccountsController to ensure a ServiceAccount (named "default") exists in every namespace
Admission plugin:
- set "default" ServiceAccount on pods without a specified serviceAccount
- reject pods which reference a serviceAccount that does not exist
- optionally (enabled by default) mount the pod's service account's APISecret at a well-known location (/var/run/secrets/kubernetes.io/serviceaccount)
- optionally (disabled by default for now) reject pods that have SecretVolumeSources that reference secrets the pod's service account doesn't reference
JWT token generator. Generates the following claims:
- iss (issuer, kubernetes.io/serviceaccount)
- sub (subject/username, serviceaccount:[namespace]:[service account name])
- kubernetes.io/namespace (for authenticator to look up secret/service account)
- kubernetes.io/service-account.name
- kubernetes.io/service-account.uid
- kubernetes.io/secret.name (for looking up secret to make sure token hasn't been revoked)
JWT token authenticator.
- validates well-formedness
- validates signature (supports trying multiple keys to allow signing key rotation)
- validates issuer and presence of expected claims
- optionally looks up secret
  - validates its name/existence name (ensures the token hasn't been revoked)
- optionally looks up service account
  - validates the UID still matches
- uses sub, kubernetes.io/service-account.uid claims from token to build user.Info

Short-term follow-ups:

Put all service accounts in a namespace in a group for ease of reference? - Add groups to service account user.Info #8607
Move JWT generation/signing into APIServer?
Sweep secret usage in scripts and examples to account for service account's referencing secrets, and then enable the admission plugin to limit pod secret usage to the pod's service account secrets
Performance test service account/secret lookups in Authenticator, maybe enable by default with a store in front of the lookups (and/or with an authenticator decision cache)

Longer-term follow-ups:

Include kubeconfig and API CA in addition to the token (requires knowing the CA bundle and hostname clients will need to use to contact the API... do we know that?) - Put Root CA Cert for cluster into all(ish) pods #7648
Work with @pweil- on interaction with securitycontext
Work with @pmorie to update Kubelet to filter secrets for a pod to the ones referenced by the pod's ServiceAccount (might want to expose /pods/[pod]/secrets to make this easier)

Sample usage

To create a service account:

serviceaccount.json:
{
    "kind": "ServiceAccount",
    "metadata": {
        "name": "myserviceaccount"
    }
}

$ kubectl create -f serviceaccount.json

To delete a service account:

$ kubectl delete serviceaccount/myserviceaccount

To create additional API tokens

A controller loop ensures a secret with an API token exists for each service account. To create additional API tokens for a service account, create a secret of type ServiceAccountToken with an annotation referencing the service account, and the controller will update it with a generated token:

secret.json:
{
    "kind": "Secret",
    "metadata": {
        "name": "mysecretname",
        "annotations": {
            "kubernetes.io/service-account.name": "myserviceaccount"
        }
    }
    "type": "kubernetes.io/service-account-token"
}

$ kubectl create -f secret.json
$ kubectl describe secret mysecretname

To delete/invalidate a service account token:

kubectl delete secret mysecretname

To set a pod's service account

Create or update the pod with the spec.serviceAccount field set to the name of a ServiceAccount. That field is set to the name of the default ServiceAccount if empty.

To mount the APISecret in containers

Ha! Trick step. You don't have to do anything... the ServiceAccount admission plugin automatically mounts the pod's spec.serviceAccount's ServiceAccountToken secret at /var/run/secrets/kubernetes.io/serviceaccount. That means containers can count on an API token existing at /var/run/secrets/kubernetes.io/serviceaccount/token

googlebot · 2015-04-21T08:41:28Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project, in which case you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please let us know the company's name.

liggitt · 2015-04-21T20:57:59Z

sorry for the massive cc, but a lot of you might be interested in this:
@erictune @roberthbailey @smarterclayton @pmorie @pweil- @ncdc

smarterclayton · 2015-04-21T21:17:09Z

Really awesome

pmorie · 2015-04-21T21:23:15Z

Haven't reviewed yet but may intersect what's being discussed in #6578 cc
@markturansky
On Tue, Apr 21, 2015 at 5:17 PM Clayton Coleman notifications@github.com
wrote:

Really awesome

—
Reply to this email directly or view it on GitHub
#7101 (comment)
.

markturansky · 2015-04-22T13:04:18Z

cmd/kubernetes/kubernetes.go

 func startComponents(etcdClient tools.EtcdClient, cl *client.Client, addr net.IP, port int) {
 	machineList := []string{"localhost"}

 	runApiServer(etcdClient, addr, port, *masterServiceNamespace)
 	runScheduler(cl)
 	runControllerManager(machineList, cl, *nodeMilliCPU, *nodeMemory)
+	runServiceAccountController(cl)


The controller is also being run from controllermanager.go. Is that correct?

nm, i see that's the standalone kubernetes binary.

which means i need to put my PVC binder in the same place :) Glad I was reviewing this PR!

No, actually, I think this does get run twice.

You have this in controllermanager.go and that's run on line 165 above. Two controllers running.

I'm having a conversation with myself. runControllerManager runs the same controllers as controllermanager but they are not the same thing.

I suppose the question then is why a separate run method for your controller instead of rolling it into runControllerManager (which ideally wouldn't have duplication with controllermanager.go)?

erictune · 2015-04-24T21:35:10Z

pkg/api/types.go

+	// It must be of type SecretTypeServiceAccountToken
+	APISecret string `json:"apiSecret"`
+	// Secrets is the list of secrets allowed to be used by pods running using this ServiceAccount
+	Secrets []string `json:"secrets"`


So, apiSecret is for this service account to talk to kubernetes/openshift apiserver, and secret is to talk to other things, like, say, the GCE API or the github API?

Should we call it KubeSecret? Or do you not want to leak the k word into OS API (understandable)? Other ideas?

What are your thoughts on tradeoffs on this design versus just having one array of secret names, where one of the entries is by convention a secret of type SecretTypeServiceAccountToken? It does work around the naming problem.

Would some clusters ever have (I'm making this up but, hopefully you get the idea) a GithubSecretCrontroller which adds Github keys to pods based on a certain default Github project for the namespace. Or a GoogleCloudStorageController which adds GoogleCloudStorage bucket keys. Etc.

If that is the case, then those initializers will need to be allowed to edit the Secrets array. Which requires that the array have a patchStrategy:"merge" patchMergeKey:"foo".

What would foo be?

If we aren't sure what foo is, then I think this array needs to be a map.

The apiSecret field did feel messy... open to suggestions. My thinking in making it a special field:

It is sort of special... the secret it refers to has to be of a certain type, and two different service accounts shouldn't reference the same ServiceAccountToken secret.

Explicitly naming the secret that holds the ServiceAccount's token lets the admission controller that sets up the VolumeMount do so based only on the info in the ServiceAccount, without also listing/searching secrets.

Avoids ambiguity if there are more than one ServiceAccountToken secret for a ServiceAccount (which is intentionally possible)

As to the name, serviceAccountTokenSecret felt too long (though that might be better than apiToken now that I think about it), and tokenSecret not descriptive enough. If the token is usable against things running on top of Kube with the ability to identify the ServiceAccount from it, calling it a KubeSecret felt too constraining.

Would some clusters ever have ... which adds Github keys to pods ...

Sure. Controllers managing secrets (either to generate them, populate them, or rotate them) seems very reasonable to me.

Which requires that the array have a patchStrategy:"merge" patchMergeKey:"foo".

The patchStrategy stuff is new to me... I'll catch up and update

It is fairly new and explained somewhat in docs/api-conventions.md.

erictune · 2015-04-24T22:20:28Z

Excited about this PR.

erictune · 2015-04-24T22:36:36Z

I've been working on service accounts for system components (bootstrapping).
For that, I would like to create a service account and a token prior to when the controller has started, and have the admission controller attach the volume for me. Can you confirm that this will work?

erictune · 2015-04-24T22:39:47Z

This is a perfect use case for integration testing.
Suggest you start with a copy of test/integration/auth_test.go, add a controller-manager thread, as is done in cmd/integration/integration.go, and then do holistic tests of system behavior.
Much faster than an e2e test.

erictune · 2015-04-24T22:50:21Z

What is the story about updating a token and its dependent processes?

I guess the controller could just update the Secret when it is near expiry, and existing pods would keep running with their already mounted data, and newly created pods would use the new secret? Some other thing could make sure all the pods that used the old secret were re-made.

liggitt · 2015-04-24T22:51:58Z

The service account tokens have to be more structured in order for the authenticator to look in the right namespace and secret to validate them. Assuming you generated them the same way the controller would, nothing would prevent you from doing so proactively. The admission plugin is cleanly separated from the account/token-creating controllers.

erictune · 2015-04-24T22:52:46Z

pkg/serviceaccount/tokens_controller.go

+		e.deleteSecret(newSecret)
+	}
+
+	// Handle case where they changed the type or annotations


You need resourceVersion as well as UID to reliably detect changes, since these objects are not immutable.

I'm just making sure the old and new secret reference the same serviceAccount (both name and UID)... I don't think I care about the specific resourceVersion of the serviceaccount

erictune · 2015-04-24T23:01:29Z

To ponder:
In a multi-cluster environment, does a pods need credentials to communicate with other apiservers? Maybe that is a special case, and you use an Opaque token in that case? Or maybe the controller syncs with a global database of tokens, with a single token working in all clusters? Not critical to figure out for this PR.

erictune · 2015-04-24T23:01:34Z

@cjcullen

liggitt · 2015-04-24T23:10:09Z

If the source cluster was one of the claims in the token, you could still do cross cluster validation. Alternately, you could expire tokens (not my preference for tokens given to machine accounts) and just verify the token's signature on the remote clusters.

liggitt · 2015-04-24T23:16:34Z

The same mechanism that would let you have a single signing (private) key to sign the tokens and multiple verifying (public) keys would allow:

signing key rotation (keep the old public keys as verifying keys for some period of time)
third party verification of tokens without having the signing key
multi-cluster token sharing (each server has its own signer key for creating, and all public keys from all servers for verifying)

liggitt · 2015-05-12T17:27:29Z

@nikhiljindal I've run e2e locally, but hadn't completed a GCE run with this branch yet... working on that today

roberthbailey · 2015-05-12T17:35:41Z

Jenkins will probably beat you to it now that this is merged. :)

deads2k · 2015-05-12T17:39:52Z

Jenkins will probably beat you to it now that this is merged. :)

Can you comment here on pass/fail, so I know whether to rebase on top of it or wait for the revert?

liggitt · 2015-05-12T17:41:25Z

wait for the revert?

You're so supportive :)

nikhiljindal · 2015-05-12T18:01:43Z

aah Sorry. I saw the status/LGTM tag on the PR and assumed that it was ready to merge :)

roberthbailey · 2015-05-12T18:02:50Z

On GCE:

Ran 38 of 44 Specs in 1488.142 seconds
SUCCESS! -- 38 Passed | 0 Failed | 1 Pending | 5 Skipped I0512 10:53:34.645502    9079 driver.go:96] All tests pass

roberthbailey · 2015-05-12T18:03:47Z

On GKE-CI:

Summarizing 1 Failure:

[Fail] ServiceAccounts [It] should mount an API token into pods 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:374

Ran 39 of 45 Specs in 1096.853 seconds
FAIL! -- 38 Passed | 1 Failed | 1 Pending | 5 Skipped F0512 11:02:03.234970   29396 driver.go:94] At least one test failed
2015/05/12 11:02:03 Error running Ginkgo tests: exit status 255

roberthbailey · 2015-05-12T18:05:52Z

Looks like we need to add the ServiceAccount to ADMISSION_CONTROL for GKE.

pmorie · 2015-05-12T18:07:44Z

@liggitt ^^

liggitt · 2015-05-12T19:21:04Z

@roberthbailey I'm having a hard time telling where GKE-CI gets its ADMISSION_CONTROL env from... would that be fixed by #8127 or is that something you have to do on your side?

roberthbailey · 2015-05-12T20:11:57Z

Fixing it on our side.

liggitt · 2015-05-12T20:22:57Z

@roberthbailey ok... the two changes that were needed for GCE were:

Adding ServiceAccount to ADMISSION_CONTROL:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7101/files#diff-57380bb03c3598646d5a3cb66518fa0bR79
Adding a --service_account_private_key_file=<server-private-key> arg to kube-controller-manager:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7101/files#diff-65667c6e20fad19143da85de765ee1e3

Not sure how GKE starts up kube-controller-manager, so I thought I'd mention it

roberthbailey · 2015-05-12T20:31:45Z

The second one gets picked up automatically. Changes to the bash scripts/config to create a cluster need to be mirrored in our provisioning code.

roberthbailey · 2015-05-12T22:05:13Z

All fixed on our end: GKE-CI is now happy as well:

Ran 39 of 45 Specs in 1073.010 seconds
SUCCESS! -- 39 Passed | 0 Failed | 1 Pending | 5 Skipped I0512 14:34:45.343409    5177 driver.go:96] All tests pass

nikhiljindal · 2015-05-13T21:03:40Z

./hack/local-up-cluster broke for me after syncing this change.

/tmp/kube-apiserver.log shows:

F0513 14:00:41.636446   23797 server.go:285] Invalid Authentication Config: open /var/run/kubernetes/serviceaccount.key: no such file or directory

After some debugging I found that
openssl genrsa -out /var/run/kubernetes/serviceaccount.key 2048 was failing with the following output:

/var/run/kubernetes/serviceaccount.key: No such file or directory
139781268178592:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/var/run/kubernetes/serviceaccount.key','w')
139781268178592:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:

I ran the following commands to fix it:

sudo mkdir /var/run/kubernetes
sudo touch /var/run/kubernetes/serviceaccount.key
sudo openssl genrsa -out /var/run/kubernetes/serviceaccount.key 2048

./hack/local-up-cluster works fine for me now.
Is this expected?

liggitt · 2015-05-13T21:06:16Z

@nikhiljindal nope, opened #8174

liggitt · 2015-05-13T21:07:27Z

worked for me locally because that folder existed from a previous run (it gets created later by the apiserver to hold self-signed certs/keys)

pires · 2015-05-21T15:56:18Z

Was this merged in 0.17.0?

roberthbailey · 2015-05-21T16:32:39Z

Nope. It missed the release by a day or so.

pires · 2015-05-21T17:48:01Z

OK, tagging 0.17.1 has just fixed that. Thanks

googlebot added the cla: no label Apr 21, 2015

liggitt force-pushed the service_account branch 7 times, most recently from 467a6ce to 0b75524 Compare April 21, 2015 20:25

markturansky reviewed Apr 22, 2015
View reviewed changes

davidopp assigned erictune Apr 24, 2015

erictune reviewed Apr 24, 2015
View reviewed changes

cjcullen mentioned this pull request Apr 24, 2015

Securing kubernetes-ro use cases #5921

Closed

liggitt force-pushed the service_account branch 2 times, most recently from 9656370 to b9ab5ce Compare April 27, 2015 13:36

nikhiljindal merged commit d75bd8b into kubernetes:master May 12, 2015

liggitt mentioned this pull request May 12, 2015

Add ServiceAccount admission plugin #8127

Merged

yujuhong mentioned this pull request May 13, 2015

Kubelet log: error creating mirror pod #8180

Closed

nikhiljindal mentioned this pull request May 13, 2015

Terminate child process by cleanup() in local-up-cluster.sh #8040

Merged

liggitt deleted the service_account branch May 14, 2015 14:11

brendandburns mentioned this pull request May 18, 2015

Update the hazelcast discovery to use the service account and https. pires/hazelcast-kubernetes-bootstrapper#2

Merged

erictune mentioned this pull request Jun 5, 2015

add dockercfg to secret describer #9321

Closed

timothysc mentioned this pull request Jun 5, 2015

Secrets and EmptyDir E2E are broken in master #9208

Closed

devurandom mentioned this pull request Jul 1, 2015

kube2sky can not resolve kubernetes service DNS name #10265

Closed

karlkfi mentioned this pull request Jul 4, 2015

mesos/docker automated local cluster deployment #10049

Merged

21 tasks

ServiceAccounts #7101

ServiceAccounts #7101

Conversation

liggitt commented Apr 21, 2015

Overview:

Sample usage

To create a service account:

To delete a service account:

To create additional API tokens

To delete/invalidate a service account token:

To set a pod's service account

To mount the APISecret in containers

googlebot commented Apr 21, 2015

liggitt commented Apr 21, 2015

smarterclayton commented Apr 21, 2015

pmorie commented Apr 21, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erictune commented Apr 24, 2015

erictune commented Apr 24, 2015

erictune commented Apr 24, 2015

erictune commented Apr 24, 2015

liggitt commented Apr 24, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erictune commented Apr 24, 2015

erictune commented Apr 24, 2015

liggitt commented Apr 24, 2015

liggitt commented Apr 24, 2015

liggitt commented May 12, 2015

roberthbailey commented May 12, 2015

deads2k commented May 12, 2015

liggitt commented May 12, 2015

nikhiljindal commented May 12, 2015

roberthbailey commented May 12, 2015

roberthbailey commented May 12, 2015

roberthbailey commented May 12, 2015

pmorie commented May 12, 2015

liggitt commented May 12, 2015

roberthbailey commented May 12, 2015

liggitt commented May 12, 2015

roberthbailey commented May 12, 2015

roberthbailey commented May 12, 2015

nikhiljindal commented May 13, 2015

liggitt commented May 13, 2015

liggitt commented May 13, 2015

pires commented May 21, 2015

roberthbailey commented May 21, 2015

pires commented May 21, 2015