title | sidebar_position |
---|---|
Ratify on Azure |
2 |
Signing container images ensure their authenticity and integrity. By deploying only signed images on Azure Kubernetes Service (AKS), you can ensure that the images come from a trusted origin and have not been altered since they were created.
With Azure Container Registry (ACR), you can store and distribute images with signatures together. You can use Azure Key Vault (AKV) to keep your signing keys and certificates safe, and then use tools like Notation or Cosign to sign your container images with them.
This article walks you through an end-to-end workflow of deploying only signed images on AKS with Ratify.
In this article:
- Prerequisites
- Prepare container images in ACR
- Sign container images in ACR
- Set up an Azure workload identity
- Set up your AKS cluster
- Install OPA Gatekeeper and Ratify
- Configure Ratify
- Deploy container images in AKS
Please note that the examples and commands provided in this document are specifically designed for the Linux operating system.
- Create or use an ACR for storing container images and signatures
- Create or use an AKV for storing keys and certificates
- Create or use an AKS for deploying container images
- Install Cosign for signing container images with Cosign signatures
- Install Notation and Notation AKV plugin for signing container images with Notary Project signatures
- Install and configure the latest Azure CLI, or run commands in the Azure Cloud Shell.
# Azure related variables
export TENANT_ID=<your Tenant ID>
export SUB_ID=<your subscription id>
# AKV related variables
export AKV_RG=<your AKV resource group>
export AKV_NAME=<your AKV name>
# ACR related variables
export ACR_RG=<your ACR resource group>
export ACR_NAME=<your ACR name>
export IMAGE_SIGNED="$ACR_NAME.azurecr.io/ratify-demo/net-monitor:v1"
export IMAGE_SIGNED_SOURCE=https://github.com/wabbit-networks/net-monitor.git#main
export IMAGE_UNSIGNED="$ACR_NAME.azurecr.io/ratify-demo/net-watcher:v1"
export IMAGE_UNSIGNED_SOURCE=https://github.com/wabbit-networks/net-watcher.git#main
# Workload identity used by Ratify to access ACR and AKV
export IDENTITY_NAME=<name of the identity to be created>
export IDENTITY_RG=<your identity resource group name>
# AKS related variables
export AKS_RG=<your AKS resource group>
export AKS_NAME=<your AKS name>
export RATIFY_NAMESPACE="gatekeeper-system"
You can skip this section if you already built and pushed images to ACR.
-
Sign in with Azure CLI
az login
To learn more about Azure CLI and how to sign in with it, see Sign in with Azure CLI.
-
Ensure the logged-in identity has both
acrpush
andacrpull
roles assigned.export USER_OBJECT_ID="$(az ad signed-in-user show --query id -o tsv)" az role assignment create \ --assignee-object-id ${USER_OBJECT_ID} \ --role acrpull --role acrpush \ --scope subscriptions/${SUB_ID}/resourceGroups/${ACR_RG}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}
-
Log into ACR
az acr login --name ${ACR_NAME}
-
Build and push an image that will be signed in later steps.
az acr build -r ${ACR_NAME} -t ${IMAGE_SIGNED} ${IMAGE_SIGNED_SOURCE} --no-logs
-
Build and push an image that will not be signed
az acr build -r ${ACR_NAME} -t ${IMAGE_UNSIGNED} ${IMAGE_UNSIGNED_SOURCE} --no-logs
Notation and Cosign are two options for signing container images. They produce different kinds of signatures and store them in the ACR. Depending on the tool you use, you need to configure Ratify accordingly to verify the signatures from each tool.
You can skip this section if you have already used Notation or Cosign to sign your container images in ACR.
Depending on the type of certificates you use, you can refer to different documents to sign container images with Notation and AKV.
For self-signed certificates, see Sign container images with Notation and Azure Key Vault using a self-signed certificate.
For CA issued certificates, see Sign container images with Notation and Azure Key Vault using a CA issued certificate.
As a result, the image named ${IMAGE_SIGNED}
should be signed successfully with Notation and AKV. Configure the following environment variables for later usage.
export CERT_NAME=<name of signing/leaf certificate>
export SUBJECT_DN=<subject DN of the signing/leaf certificate>
export CERT_KEY_ID=<key identity for the signing/leaf certificate>
- Configure environment variables
# The key used for Cosign signing
export KEY_NAME=<name of the key to be created for Cosign signing>
-
Log into Azure
az login
-
Assign role
Key Vault Crypto Officer
to logged in identity for creating a keyexport USER_OBJECT_ID="$(az ad signed-in-user show --query id -o tsv)" az role assignment create --role "Key Vault Crypto Officer" --assignee ${USER_OBJECT_ID} \ --scope "/subscriptions/${SUB_ID}/resourceGroups/${AKV_RG}/providers/Microsoft.KeyVault/vaults/${AKV_NAME}"
-
Create a key in your AKV
az keyvault key create --vault-name ${AKV_NAME} -n ${KEY_NAME} --protection software
Get the key id and retrieve the key version:
az keyvault key show --name ${KEY_NAME} --vault-name ${AKV_NAME} --query "key.kid"
An example output:
https://<your akv name>.vault.azure.net/keys/<your key name>/<your key version>
Configure an environment variable for the version for later usage.
export KEY_VER=<your key version>
-
Sign an image stored in your ACR
Confirm no signatures before signing
cosign tree ${IMAGE_SIGNED}
Sign the image
cosign sign --key azurekms://$AKV_NAME.vault.azure.net/${KEY_NAME}/${KEY_VER} --tlog-upload=false ${IMAGE_SIGNED}
In this article, use flag
--tlog-upload=false
to skip upload the signature to the transparent log (Rekor by default).Sign using a key in AKV does not necessarily require the role
Key Vault Crypto Officer
, you can use another identity and assign the roleKey Vault Crypto User
for signing action only.Confirm the signature is pushed and associated with the image in ACR
cosign tree ${IMAGE_SIGNED}
Ratify pulls artifacts from an ACR using Workload Federated Identity in an AKS cluster. For an overview on how workload identity operates in Azure, refer to the documentation. You can use workload identity federation to configure an Azure AD app registration or user-assigned managed identity. The following workflow includes the Workload Identity configuration.
az identity create --name "${IDENTITY_NAME}" --resource-group "${IDENTITY_RG}" --location "${LOCATION}" --subscription "${SUB_ID}"
export IDENTITY_OBJECT_ID="$(az identity show --name "${IDENTITY_NAME}" --resource-group "${IDENTITY_RG}" --query 'principalId' -otsv)"
export IDENTITY_CLIENT_ID=$(az identity show --name ${IDENTITY_NAME} --resource-group ${IDENTITY_RG} --query 'clientId' -o tsv)
Configure the user-assigned managed identity and assign AcrPull
role to the workload identity.
az role assignment create \
--assignee-object-id ${IDENTITY_OBJECT_ID} \
--role acrpull \
--scope subscriptions/${SUB_ID}/resourceGroups/${ACR_RG}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}
Ratify requires secret permissions to retrieve the root CA certificate from the entire certificate chain, please set private keys to Non-exportable at certificate creation time to avoid security risk. Learn more about non-exportable keys here
For security or other reasons (such as you are from a different organization), you may not be able to access AKV and get the root CA certificates. In that case, you can use the inline certificate provider to specify the root CA certificate value directly, without needing AKV.
Assign Key Vault Secrets User
role to this identity for accessing AKV
az role assignment create --role "Key Vault Secrets User" --assignee ${IDENTITY_OBJECT_ID} \
--scope "/subscriptions/${SUB_ID}/resourceGroups/${AKV_RG}/providers/Microsoft.KeyVault/vaults/${AKV_NAME}"
Assign Key Vault Crypto User
role to this identity for accessing AKV
az role assignment create --role "Key Vault Crypto User" --assignee $IDENTITY_OBJECT_ID \
--scope "/subscriptions/${SUB_ID}/resourceGroups/${AKV_RG}/providers/Microsoft.KeyVault/vaults/${AKV_NAME}"
-
Create an OIDC enabled AKS cluster. You can skip this step if you have an AKS cluster with both OIDC and workload identity enabled.
# Install the aks-preview extension az extension add --name aks-preview # Register the 'EnableWorkloadIdentityPreview' feature flag az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview" az provider register --namespace Microsoft.ContainerService az aks create \ --resource-group "${AKS_RG}" \ --name "${AKS_NAME}" \ --node-vm-size Standard_DS3_v2 \ --node-count 1 \ --generate-ssh-keys \ --enable-workload-identity \ --attach-acr ${ACR_NAME} \ --enable-oidc-issuer # Connect to the AKS cluster: az aks get-credentials --resource-group ${AKS_RG} --name ${AKS_NAME} export AKS_OIDC_ISSUER="$(az aks show -n ${AKS_NAME} -g ${AKS_RG} --query "oidcIssuerProfile.issuerUrl" -otsv)"
The official steps for setting up Workload Identity on AKS can be found here.
This step above may take around 10 minutes to complete. The registration status can be checked by running the following command:
az feature show --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview" -o table Name RegistrationState -------------------------------------------------------- ------------------- Microsoft.ContainerService/EnableWorkloadIdentityPreview Registered
-
Update an existing AKS cluster with OIDC and workload identity enabled. You can skip this step if you have an AKS cluster with both OIDC and workload identity enabled.
az aks update -g ${AKS_RG} -n ${AKS_NAME} --enable-oidc-issuer --enable-workload-identity export AKS_OIDC_ISSUER="$(az aks show -n ${AKS_NAME} -g ${AKS_RG} --query "oidcIssuerProfile.issuerUrl" -otsv)"
-
Establish federated identity credential for namespace
${RATIFY_NAMESPACE}
where you deploy Ratify:az identity federated-credential create \ --name ratify-federated-credential \ --identity-name "${IDENTITY_NAME}" \ --resource-group "${IDENTITY_RG}" \ --issuer "${AKS_OIDC_ISSUER}" \ --subject system:serviceaccount:"${RATIFY_NAMESPACE}":"ratify-admin"
Run az aks show -g "${AKS_RG}" -n "${AKS_NAME}" --query addonProfiles.azurepolicy
to verify if the AKS cluster has azure policy addon enabled, learn more at use azure policy
-
Install Gatekeeper from helm chart:
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts helm install gatekeeper/gatekeeper \ --name-template=gatekeeper \ --namespace gatekeeper-system --create-namespace \ --set enableExternalData=true \ --set validatingWebhookTimeoutSeconds=5 \ --set mutatingWebhookTimeoutSeconds=2 \ --set externaldataProviderResponseCacheTTL=10s
-
Install Ratify on AKS from helm chart:
# Add a Helm repo helm repo add ratify https://ratify-project.github.io/ratify # Install Ratify helm install ratify \ ratify/ratify --atomic \ --namespace ${RATIFY_NAMESPACE} --create-namespace \ --set featureFlags.RATIFY_CERT_ROTATION=true \ --set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID}
-
Enforce Gatekeeper policy to allow only signed images can be deployed on AKS:
kubectl apply -f https://ratify-project.github.io/ratify/library/default/template.yaml kubectl apply -f https://ratify-project.github.io/ratify/library/default/samples/constraint.yaml
-
Ensure your AKS cluster is 1.26+
-
Run
az feature register -n AKS-AzurePolicyExternalData --namespace Microsoft.ContainerService
-
Install Ratify on AKS from helm chart:
# Add a Helm repo helm repo add ratify https://ratify-project.github.io/ratify helm repo update # Install Ratify helm install ratify \ ratify/ratify --atomic \ --namespace gatekeeper-system --create-namespace \ --set provider.enableMutation=false \ --set featureFlags.RATIFY_CERT_ROTATION=true \ --set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID}
-
Create and assign azure policy on your cluster:
export CUSTOM_POLICY=$(curl -L https://raw.githubusercontent.com/deislabs/ratify/main/library/default/customazurepolicy.json) export DEFINITION_NAME="ratify-default-custom-policy" export POLICY_SCOPE=$(az aks show -g "${AKS_RG}" -n "${AKS_NAME}" --query id -o tsv) export DEFINITION_ID=$(az policy definition create --name "${DEFINITION_NAME}" --rules "$(echo "${CUSTOM_POLICY}" | jq .policyRule)" --params "$(echo "${CUSTOM_POLICY}" | jq .parameters)" --mode "Microsoft.Kubernetes.Data" --query id -o tsv) export ASSIGNMENT_ID=$(az policy assignment create --policy "${DEFINITION_ID}" --name "${DEFINITION_NAME}" --scope "${POLICY_SCOPE}" --query id -o tsv) echo "Please wait policy assignment with id ${ASSIGNMENT_ID} taking effect" echo "It often requires 15 min" echo "You can run 'kubectl get constraintTemplate ratifyverification' to verify the policy takes effect"
-
Create a configuration file for a
Store
custom resource namedstore-oras
:cat <<EOF > store_config.yaml apiVersion: config.ratify.deislabs.io/v1beta1 kind: Store metadata: name: store-oras spec: name: oras parameters: authProvider: name: azureWorkloadIdentity clientID: $IDENTITY_CLIENT_ID cosignEnabled: true EOF
-
Apply the configuration
kubectl apply -f store_config.yaml
-
Confirm the configuration is applied successful.
kubectl get Store store-oras
Make sure the
ISSUCCESS
value is true in the results of above three commands. If it is not, you need to check the detailed error logs by usingkubectl describe
commands. For example,kubectl describe Store store-oras
-
Create a configuration file for a
keymanagementprovider
custom resource namedkeymanagementprovider-akv
:For verifying images signed with Notation using certificates in AKV, create the following configuration:
cat <<EOF > kmp_config.yaml apiVersion: config.ratify.deislabs.io/v1beta1 kind: KeyManagementProvider metadata: name: keymanagementprovider-akv spec: type: azurekeyvault parameters: vaultURI: https://${AKV_NAME}.vault.azure.net/ certificates: - name: ${CERT_NAME} version: ${CERT_KEY_ID} tenantID: ${TENANT_ID} clientID: ${IDENTITY_CLIENT_ID} EOF
For verifying images signed with Cosign using keys in AKV, create the following configuration:
cat <<EOF > kmp_config.yaml apiVersion: config.ratify.deislabs.io/v1beta1 kind: KeyManagementProvider metadata: name: keymanagementprovider-akv spec: type: azurekeyvault parameters: vaultURI: https://${AKV_NAME}.vault.azure.net/ keys: - name: ${KEY_NAME} version: ${KEY_VER} tenantID: ${TENANT_ID} clientID: ${IDENTITY_CLIENT_ID} EOF
You may combine the configuration into one
KeyManagementProvider
resource for both keys and certificates if they are stored in the same AKV.
-
Apply the configuration
kubectl apply -f kmp_config.yaml
-
Confirm the configuration is applied successful.
kubectl get KeyManagementProvider KeyManagementProvider-akv
Make sure the
ISSUCCESS
value is true in the results of above three commands. If it is not, you need to check the detailed error logs by usingkubectl describe
commands. For example,kubectl describe KeyManagementProvider KeyManagementProvider-akv
-
Create a configuration file for a
Verifier
custom resource namedverifier-notation
:cat <<EOF > notation_config.yaml apiVersion: config.ratify.deislabs.io/v1beta1 kind: Verifier metadata: name: verifier-notation spec: name: notation artifactTypes: application/vnd.cncf.notary.signature parameters: verificationCertStores: certs: - keymanagementprovider-akv trustPolicyDoc: version: "1.0" trustPolicies: - name: default registryScopes: - "*" signatureVerification: level: strict trustStores: - ca:certs trustedIdentities: - "x509.subject: ${SUBJECT_DN}" EOF
-
Apply the configuration
kubectl apply -f notation_config.yaml
-
Confirm the configuration is applied successful.
kubectl get Verifier verifier-notation
Make sure the
ISSUCCESS
value is true in the results of above three commands. If it is not, you need to check the detailed error logs by usingkubectl describe
commands. For example,kubectl describe Verifier verifier-notation
-
Create a configuration file for a
Verifier
custom resource namedverifier-cosign
:cat <<EOF > cosign_config.yaml apiVersion: config.ratify.deislabs.io/v1beta1 kind: Verifier metadata: name: verifier-cosign spec: name: cosign artifactTypes: application/vnd.dev.cosign.artifact.sig.v1+json parameters: trustPolicies: - name: default scopes: - "*" keys: - provider: keymanagementprovider-akv EOF
-
Apply the verification configuration
kubectl apply -f cosign_config.yaml
-
Confirm the configuration is applied successful.
kubectl get Verifier verifier-cosign
Make sure the
ISSUCCESS
value is true in the results of above three commands. If it is not, you need to check the detailed error logs by usingkubectl describe
commands. For example,kubectl describe Verifier verifier-cosign
Run the following command, since $IMAGE_SIGNED is signed with the key configured in Ratify, so this image was allowed for deployment after signature verification succeeded.
kubectl run demo-signed --image=$IMAGE_SIGNED
Run the following command, since $IMAGE_UNSIGNED is not signed, so this image was NOT allowed for deployment.
kubectl run demo-unsigned --image=$IMAGE_UNSIGNED
You can configure different trust policies for images from various registry scope for the Cosign verifier. For example, you have two ACR: $ACR_NAME1
and $ACR_NAME2
. For $ACR_NAME1
, you want to use keymanagementprovider-akv1
resource, For $ACR_NAME2
, you want to use keymanagementprovider-akv2
resource. You can update Cosign Verifier resource as the following:
apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
name: verifier-cosign
spec:
name: cosign
artifactTypes: application/vnd.dev.cosign.artifact.sig.v1+json
parameters:
trustPolicies:
- name: $ACR_NAME1
scopes:
- "$ACR_NAME1.azurecr.io/*"
keys:
- provider: keymanagementprovider-akv1
- name: $ACR_NAME2
scopes:
- "$ACR_NAME2.azurecr.io/*"
keys:
- provider: keymanagementprovider-akv2
For more information, please refer to the scopes of Cosign verifier .
Keys in AKV may be rotated regularly as security best practice. If the key is rotated with a new version, you can update the KeyManagementProvider
resource by adding the new version of the key, as Ratify (v1.2.0 or before) does not support reconciling key resources regularly. For example, if the new version of key is set to environment variable $KEY_VER_NEW
, you can do the following:
-
Add the new key
$KEY_VER_NEW
forKeyManagementProvider
resourceapiVersion: config.ratify.deislabs.io/v1beta1 kind: KeyManagementProvider metadata: name: keymanagementprovider-akv spec: type: azurekeyvault parameters: vaultURI: https://$AKV_NAME.vault.azure.net/ keys: - name: $KEY_NAME version: $KEY_VER version: $KEY_VER_NEW tenantID: $TENANT_ID clientID: $CLIENT_ID
-
Apply the new configuration
kubectl apply -f verification_config.yaml
-
Confirm the new configuration is applied successfully
kubectl get KeyManagementProvider keymanagementprovider-akv
In some cases, you may need to disable a specific version of key. For example, the specific version of key is leaked. So, images signed using the specific version of key should not be trusted and the deployment of those images should be denied. As Ratify (v1.2.0 or before) does not support reconciling key resources regularly, so you need to manually remove the version of key from KeyManagementProvider
resource. For example, if version $KEY_VER
is leaked, what you need to do is:
-
Disable the specific version from AKV, in this case, version
$KEY_VER
is disabled. -
Rotate the key to a new version
$KEY_VER_NEW
, so that your images will be signed with new version. -
Update
KeyManagementProvider
resource to add the new version of key and remove the disabled versionThe
KeyManagementProvider
resource will look like the following as disabled version$KEY_VER
was removed.apiVersion: config.ratify.deislabs.io/v1beta1 kind: KeyManagementProvider metadata: name: keymanagementprovider-akv spec: type: azurekeyvault parameters: vaultURI: https://$AKV_NAME.vault.azure.net/ keys: - name: $KEY_NAME version: $KEY_VER_NEW tenantID: $TENANT_ID clientID: $CLIENT_ID
Apply the new configuration
kubectl apply -f verification_config.yaml
Confirm the new configuration is applied successfully
kubectl get KeyManagementProvider keymanagementprovider-akv