Kubernetes provides a mechanism for pods to obtain bound service account JWTs issued by the API server. However, this JWT is not directly useful outside the Kubernetes cluster. In general, applications require credentials issued by an organization specific IAM provider to authenticate with external systems. This project aims to provide a way for pods to exchange service account JWTs with a credential provider for a unique identity.
Athenz is an IAM provider which allows users to define role based access control (RBAC) for users and services. It also acts as a certificate authority (CA) by provisioning instances with unique identities through X.509 certificates.
Athenz Copper Argos is a callback mechanism for identity providers to validate certificate requests from instances running in a variety of platforms such as Kubernetes pods, openstack vms, AWS EC2s, etc. This allows any orchestration layer to implement platform specific instance verification checks.
The architecture section dives into the details of the various components built to create an Athenz Copper Argos identity provider.
Service Identity Agent (SIA) is a container which is bundled as a sidecar for an application pod. It is primarily responsible for creating a CSR for the application and requesting a certificate from Athenz ZTS. SIA is mounted with a bound Kubernetes service account JWT issued with an audience specific to the identity provider, "athenz-identityd" (can be the Athenz service of identity provider).
These are the steps the SIA container follows to retrieve an identity:
- Creates a new private key and signs a CSR with the subject common name as "athenz-domain.athenz-service", required SANs, etc.
- Constructs an Athenz InstanceRegisterInformation object with the CSR and the bound service account JWT as attestation data.
- Makes a request to ZTS to the postInstanceRegister API call.
- ZTS forwards the attestation data and CSR details to the identity provider for validation.
- Certificate is minted by Athenz ZTS and returned to the SIA container.
- SIA writes the certificates to a directory for the application to consume.
Identity provider is an Athenz Copper Argos callback provider which validates requests for new identities. It runs as a deployment in the cluster and has an in-memory cache of all running pods.
These are the steps the Identity provider follows to validate an identity:
- Validate the bound service account JWT of the attestation data is valid, this involves either making a request to the Kubernetes TokenReview API or using public key validation.
- Validate the pod which is requesting the identity is actually running within the cluster.
- Validate the CSR details including IP, SANS, common name, etc.
The checks are implemented as rego policies evaluated by an OPA server.
Using Open Policy Agent (OPA) for the Identity Provider
OPA provides a policy evaluation engine and a Rego language to write the policies. The most common use case of OPA in Kubernetes is for admission control where the checks are written as rego policies and loaded to OPA through configmaps. If the admission checks require to lookup any kubernetes object, OPA provides replication (cache) setup for cluster & namespace scoped resources. This replication mechanism also allows updating policy checks dynamically.
The idea is to run the identity provider pod as an OPA instance and define the identity verification checks as Rego policies written in configmaps. OPA’s rego evaluation engine also has built-in support for JWT verification and making HTTP requests. This helps us to verify the attestation data JWT either by making Kubernetes API TokenReview requests or by locally validating the signature using the pre-loaded kubernetes API public key. Other field verifications that depend on the pod metadata can use the resource replication feature and cache the pod resources.
The identity provider pod contains two containers:
- Opa - Evaluates the identity verification checks with information provided with from the policy configmaps and replicated pods resource information
- Mgmt - Fetches the configmap policies from the configured system namespace and the cluster pod resources by watching the Kubernetes API and storing them on OPA through localhost REST API calls.
There are a couple of prerequisites required to run this identity provider, they are specified below.
- Kubernetes cluster - A running Kubernetes cluster, version 1.13 or higher, is required with admin access. More information on how to setup a cluster can be found in the official documentation here.
- Athenz - A complete Athenz setup including ZMS, ZTS, and an optional UI server installed.
git clone https://github.com/yahoo/k8s-athenz-identity.git kubectl apply -f k8s-athenz-identity/k8s
Create an Athenz domain and a service using zms-cli or ZMS UI corresponding to the Identityd k8s service
# Build the zms-cli utility to interact with the Athenz ZMS server go get github.com/yahoo/athenz/utils/zms-cli # Create an Athenz domain associated with the Kubernetes cluster zms-cli add-domain k8s-cluster-name # Create an Athenz service on the cluster domain to associate it with the Identityd Kubernetes service zms-cli -d k8s-cluster-name add-service identityd # Set the Identityd service endpoint to point to the kubernetes identityd service zms-cli -d k8s-cluster-name set-service-endpoint identityd identityd.default.svc.cluster.local
The application deployments that require Athenz identity certs need the SIA container and the kubernetes bound service account JWT volume mount added to the pod template using the sample patch
kubectl patch deploy <app.yaml> -p k8s-athenz-identity/k8s/patch/sia.yaml
The generated Athenz cert and key is stored under the
tls-certs volume mount.
The following table outlines the configuration parameters for the SIA. Environment variables can be used instead of the parameters
|mode||MODE||Mode, must be one of init or refresh||init|
|endpoint||ENDPOINT||Athenz ZTS endpoint|
|provider-service||PROVIDER_SERVICE||Identity Provider service|
|dns-suffix||DNS_SUFFIX||DNS Suffix for the certs|
|refresh-interval||REFRESH_INTERVAL||Interval to refresh the certs||24h|
|out-cert||CERT_FILE||Path to output the certificate||/var/run/athenz/service.cert.pem|
|out-key||KEY_FILE||Path to output the key||/var/run/athenz/service.key.pem|
|out-ca-cert||CA_CERT_FILE||Path to output the ca certificate||/var/run/athenz/ca.cert.pem|
|log-dir||LOG_DIR||Directory to store the server log files||/var/log/athenz-sia|
|sa-token-file||SA_TOKEN_FILE||Bound sa token file location||/var/run/secrets/kubernetes.io/bound-serviceaccount/token|
|server-ca-cert||SERVER_CA_CERT||Path to the ca cert file to verify ZTS server certs|
$ athenz-sia --mode=init --out-cert=/var/run/athenz/service.cert.pem --out-key=/var/run/athenz/service.key.pem --out-ca-cert=/var/run/athenz/ca.cert.pem ... $ athenz-sia --mode=refresh --refresh-interval=720h --out-cert=/var/run/athenz/service.cert.pem --out-key=/var/run/athenz/service.key.pem --out-ca-cert=/var/run/athenz/ca.cert.pem ...
Please refer to the contributing file for information about how to get involved. We welcome issues, questions, and pull requests.
Core Team : firstname.lastname@example.org
Copyright 2019 Verizon Media Inc. Licensed under the terms of the 3-Clause BSD License.