Developers use kubectl
to access Kubernetes clusters. By default kubectl
uses a certificate to authenticate to the Kubernetes API. This means that when multiple developers need to access a cluster, the certificate needs to be shared. Sharing the credentials to access a Kubernetes cluster presents a significant security problem. Compromise of the certificate is very easy and the consequences can be catastrophic.
In this tutorial, we walk through how to set up your Kubernetes cluster to add Single Sign-On support for kubectl
using OpenID Connect (OIDC) and Keycloak. Instead of using a shared certificate, users will be able to use their own personal credentials to use kubectl
with kubelogin
.
This tutorial relies on
Note This guide was designed and validated using an Azure AKS Cluster. It's possible that this procedure will work with other cloud providers, but there is a lot of variance in the Authentication mechanisms for the Kubernetes API. See the troubleshooting note at the bottom for more info.
In this section, we'll configure your Kubernetes cluster for single-sign on.
-
Delete the openapi mapping from the Ambassador namespace
kubectl delete -n ambassador ambassador-devportal-api
. (this mapping can conflict withkubectl
commands) -
Create a new private key using
openssl genrsa -out aes-key.pem 4096
. -
Create a file
aes-csr.cnf
and paste the following config.[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ dn ] CN = ambassador-kubeapi # Required [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=keyEncipherment,dataEncipherment extendedKeyUsage=serverAuth,clientAuth
-
Create a certificate signing request with the config file we just created.
openssl req -config ./aes-csr.cnf -new -key aes-key.pem -nodes -out aes-csr.csr
. -
Create and apply the following YAML for a CertificateSigningRequest. Replace {{BASE64_CSR}} with the value from
cat aes-csr.csr | base64
. Note that this isaes-csr.csr
, and notaes-csr.cnf
.apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: aes-csr spec: groups: - system:authenticated request: {{BASE64_CSR}} # Base64 encoded aes-csr.csr usages: - digital signature - key encipherment - server auth - client auth
-
Check csr was created:
kubectl get csr
(it will be in pending state). After confirmation, runkubectl certificate approve aes-csr
. You can checkkubectl get csr
again to see that it's in theApproved, Issued
state. -
Get the resulting certificate and put it into a pem file.
kubectl get csr aes-csr -o jsonpath='{.status.certificate}' | base64 -d > aes-cert.pem
. -
Create a TLS
Secret
using our private key and public certificate.kubectl create secret tls -n ambassador aes-kubeapi --cert ./aes-cert.pem --key ./aes-key.pem
-
Create a
Mapping
andTLSContext
for the Kube API.--- apiVersion: getambassador.io/v3alpha1 kind: TLSContext metadata: name: aes-kubeapi-context namespace: ambassador spec: hosts: - "*" secret: aes-kubeapi --- apiVersion: getambassador.io/v3alpha1 kind: Mapping metadata: name: aes-kubeapi-mapping namespace: ambassador spec:
hostname: "*" prefix: / allow_upgrade: - spdy/3.1 service: https://kubernetes.default.svc timeout_ms: 0 tls: aes-kubeapi-context ```
-
Create RBAC for the "aes-kubeapi" user by applying the following YAML.
--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: aes-impersonator-role rules: - apiGroups: [""] resources: ["users", "groups", "serviceaccounts"] verbs: ["impersonate"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: aes-impersonator-rolebinding subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: aes-kubeapi roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: aes-impersonator-role
As a quick check, you should be able to curl https://<ambassador-domain>/api
and get a response similar to the following:
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "\"<some-kubernetes-service-address>\":443"
}
]
}%
- Create a new Realm and Client (e.g. ambassador, ambassador)
- Make sure that
http://localhost:8000
andhttp://localhost:18000
are valid Redirect URIs - Set access type to confidential and Save
- Go to the Credentials tab and note down the secret
- Go to the user tab and create a user with the first name "john"
-
Add the following RBAC to create a user "john" that only allowed to perform
kubectl get services
in the cluster.--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: john-binding subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: john-role --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: john-role rules: - apiGroups: [""] resources: ["services"] verbs: ["get", "list"]
-
Test the API again with the following 2
curls
:curl https://<ambassador-domain>/api/v1/namespaces/default/services?limit=500 -H "Impersonate-User: "john"
andcurl https://<ambassador-domain>/api/v1/namespaces/default/pods?limit=500 -H "Impersonate-User: "john"
. You will find that the first curl should succeeds and the second curl should fail with the following response.
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "pods is forbidden: User \"john\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
"reason": "Forbidden",
"details": {
"kind": "pods"
},
"code": 403
}
-
Create the following JWT
Filter
andFilterPolicy
based on this template:--- apiVersion: getambassador.io/v3alpha1 kind: Filter metadata: name: "kubeapi-jwt-filter" namespace: "ambassador" spec: JWT: jwksURI: https://<keycloak-domain>/auth/realms/<my-realm>/protocol/openid-connect/certs # If the keycloak instance is internal, you may want to use the internal k8s endpoint (e.g. http://keycloak.keycloak) instead of figuring out how to exclude JWKS requests from the FilterPolicy injectRequestHeaders: - name: "Impersonate-User" # Impersonate-User is mandatory, you can also add an Impersonate-Groups if you want to do group-based RBAC value: "{{ .token.Claims.given_name }}" # This uses the first name we specified in the Keycloak user account --- apiVersion: getambassador.io/v3alpha1 kind: FilterPolicy metadata: name: "kubeapi-filter-policy" namespace: "ambassador" spec: rules: - host: "*" path: "*" filters: - name: kubeapi-jwt-filter
Now, we need to set up the client. Each user who needs to access the Kubernetes cluster will need to follow these steps.
-
Install kubelogin. Kubelogin is a
kubectl
plugin that enables OpenID Connect login withkubectl
. -
Edit your local Kubernetes config file (either
~/.kube/config
, or your$KUBECONFIG
file) to include the following, making sure to replace the templated values.apiVersion: v1 kind: Config clusters: - name: azure-ambassador cluster: server: https://<ambassador-domain> contexts: - name: azure-ambassador-kube-api context: cluster: azure-ambassador user: azure-ambassador users: - name: azure-ambassador user: exec: apiVersion: client.authentication.k8s.io/v1beta1 command: kubectl args: - oidc-login - get-token - --oidc-issuer-url=https://<keycloak-domain>/auth/realms/<my-realm> - --oidc-client-id=<client-id> - --oidc-client-secret=<client-secret>
-
Switch to the context set above (in the example it's
azure-ambassador-kube-api
). -
Run
kubectl get svc
. This should open a browser page to the Keycloak login. Type in the credentials for "john" and, on success, return to the terminal to see the kubectl response. Congratulations, you've set up Single Sign-On with Kubernetes! -
Now try running
kubectl get pods
, and notice we get anError from server (Forbidden): pods is forbidden: User "john" cannot list resource "pods" in API group "" in the namespace "default"
. This is expected because we explicitly set up "john" to only have access to viewService
resources, and notPods
.
- Delete the token cache with
rm -r ~/.kube/cache/oidc-login
- You may also have to remove session cookies in your browser or do a remote logout in the keycloak admin page.
- Why isn't this process working in my
<insert-cloud-provider-here>
cluster? Authentication to the Kubernetes API is highly cluster specific. Many use x509 certificates, but as a notable exception, Amazon's Elastic Kubernetes Service, for example, uses an Authenticating Webhook that connects to their IAM solution for Authentication, and so is not compatible specifically with this guide. - What if I want to use RBAC Groups?
User impersonation allows you to specify a Group using the
Impersonate-Group
header. As such, if you wanted to use any kind of custom claims for the ID token, they can be mapped to theImpersonate-Group
header. Note that you always have to use anImpersonate-Name
header, even if you're relying solely on the Group for Authorization. - I keep getting a 401
Failure
,Unauthorized
message, even forhttps://<ambassador-domain>/api
. This likely means that there is either something wrong with the Certificate that was issued, or there's something wrong with yourTLSContext
orMapping
config.$AESproductName$ must present the correct certificate to the Kubernetes API and the RBAC usernames and the CN of the certificate have to be consistent with one another. - Do I have to use
kubelogin
? Technically no. Any method of obtaining an ID or Access token from an Identity Provider will work. You can then pass the token using--token <jwt-token>
when runningkubectl
.kubelogin
simply automates the process of getting the ID token and attaching it to akubectl
request.
In this tutorial, we set up Filter
.
The general flow of the kubectl
command is as follows: On making an unauthenticated kubectl command, kubelogin
does a browser open/redirect in order to do OIDC token negotiation. kubelogin
obtains an OIDC Identity Token (notice this is not an access token) and sends it to Impersonate-XXX
headers.