title | description | author | ms.service | ms.subservice | ms.topic | ms.date | ms.author | ms.custom |
---|---|---|---|---|---|---|---|---|
Quickstart: Deploy an AKS cluster with Enclave Confidential Container Intel SGX nodes by using the Azure CLI |
Learn how to create an Azure Kubernetes Service (AKS) cluster with enclave confidential containers a Hello World app by using the Azure CLI. |
angarg05 |
virtual-machines |
confidential-computing |
quickstart |
11/06/2023 |
ananyagarg |
devx-track-azurecli, mode-api |
Quickstart: Deploy an AKS cluster with confidential computing Intel SGX agent nodes by using the Azure CLI
In this quickstart, you'll use the Azure CLI to deploy an Azure Kubernetes Service (AKS) cluster with enclave-aware (DCsv2/DCSv3) VM nodes. You'll then run a simple Hello World application in an enclave. You can also provision a cluster and add confidential computing nodes from the Azure portal, but this quickstart focuses on the Azure CLI.
AKS is a managed Kubernetes service that enables developers or cluster operators to quickly deploy and manage clusters. To learn more, read the AKS introduction and the overview of AKS confidential nodes.
Features of confidential computing nodes include:
- Linux worker nodes supporting Linux containers.
- Generation 2 virtual machine (VM) with Ubuntu 18.04 VM nodes.
- Intel SGX capable CPU to help run your containers in confidentiality protected enclave leveraging Encrypted Page Cache (EPC) memory. For more information, see Frequently asked questions for Azure confidential computing.
- Intel SGX DCAP Driver preinstalled on the confidential computing nodes. For more information, see Frequently asked questions for Azure confidential computing.
Note
DCsv2/DCsv3 VMs use specialized hardware that's subject region availability. For more information, see the available SKUs and supported regions.
This quickstart requires:
-
A minimum of eight DCsv2/DCSv3/DCdsv3 cores available in your subscription.
By default, there is no pre-assigned quota for Intel SGX VM sizes for your Azure subscriptions. You should follow these instructions to request for VM core quota for your subscriptions.
Use the following instructions to create an AKS cluster with the Intel SGX add-on enabled, add a node pool to the cluster, and verify what you created with hello world enclave application.
Note
If you already have an AKS cluster that meets the prerequisite criteria listed earlier, skip to the next section to add a confidential computing node pool.
Intel SGX AKS Addon "confcom" exposes the Intel SGX device drivers to your containers to avoid added changes to your pod yaml.
First, create a resource group for the cluster by using the az group create command. The following example creates a resource group named myResourceGroup in the eastus2 region:
az group create --name myResourceGroup --location eastus2
Now create an AKS cluster, with the confidential computing add-on enabled, by using the az aks create command:
az aks create -g myResourceGroup --name myAKSCluster --generate-ssh-keys --enable-addons confcom
The above command will deploy a new AKS cluster with system node pool of non confidential computing node. Confidential computing Intel SGX nodes are not recommended for system node pools.
Run the following command to add a user node pool of Standard_DC4s_v3
size with three nodes to the AKS cluster. You can choose another larger sized SKU from the list of supported DCsv2/DCsv3 SKUs and regions.
az aks nodepool add --cluster-name myAKSCluster --name confcompool1 --resource-group myResourceGroup --node-vm-size Standard_DC4s_v3 --node-count 2
After you run the command, a new node pool with DCsv3 should be visible with confidential computing add-on DaemonSets (SGX device plug-in).
Get the credentials for your AKS cluster by using the az aks get-credentials command:
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
Use the kubectl get pods
command to verify that the nodes are created properly and the SGX-related DaemonSets are running on DCsv2 node pools:
kubectl get pods --all-namespaces
kube-system sgx-device-plugin-xxxx 1/1 Running
If the output matches the preceding code, your AKS cluster is now ready to run confidential applications.
You can go to the Deploy Hello World from an isolated enclave application section in this quickstart to test an app in an enclave. Or use the following instructions to add more node pools on AKS. (AKS supports mixing SGX node pools and non-SGX node pools.)
This section assumes you're already running an AKS cluster that meets the prerequisite criteria listed earlier in this quickstart.
Run the following command to enable the confidential computing add-on:
az aks enable-addons --addons confcom --name MyManagedCluster --resource-group MyResourceGroup
Note
To use the confidential computing capability, your existing AKS cluster needs to have a minimum of one node pool that's based on a DCsv2/DCsv3 VM SKU. To learn more about DCs-v2/Dcs-v3 VMs SKUs for confidential computing, see the available SKUs and supported regions.
Run the following command to create a node pool:
az aks nodepool add --cluster-name myAKSCluster --name confcompool1 --resource-group myResourceGroup --node-count 2 --node-vm-size Standard_DC4s_v3
Verify that the new node pool with the name confcompool1 has been created:
az aks nodepool list --cluster-name myAKSCluster --resource-group myResourceGroup
Sign in to your existing AKS cluster to perform the following verification:
kubectl get nodes
The output should show the newly added confcompool1 pool on the AKS cluster. You might also see other DaemonSets.
kubectl get pods --all-namespaces
kube-system sgx-device-plugin-xxxx 1/1 Running
If the output matches the preceding code, your AKS cluster is now ready to run confidential applications.
You're now ready to deploy a test application.
Create a file named hello-world-enclave.yaml and paste in the following YAML manifest. You can find this sample application code in the Open Enclave project. This deployment assumes that you've deployed the confcom add-on.
Note
The following example pulls a public container image from Docker Hub. We recommend that you set up a pull secret to authenticate using a Docker Hub account instead of making an anonymous pull request. To improve reliability when working with public content, import and manage the image in a private Azure container registry. Learn more about working with public images.
apiVersion: batch/v1
kind: Job
metadata:
name: oe-helloworld
namespace: default
spec:
template:
metadata:
labels:
app: oe-helloworld
spec:
containers:
- name: oe-helloworld
image: mcr.microsoft.com/acc/samples/oe-helloworld:latest
resources:
limits:
sgx.intel.com/epc: "10Mi"
requests:
sgx.intel.com/epc: "10Mi"
volumeMounts:
- name: var-run-aesmd
mountPath: /var/run/aesmd
restartPolicy: "Never"
volumes:
- name: var-run-aesmd
hostPath:
path: /var/run/aesmd
backoffLimit: 0
Alternatively you can also do a node pool selection deployment for your container deployments as shown below
apiVersion: batch/v1
kind: Job
metadata:
name: oe-helloworld
namespace: default
spec:
template:
metadata:
labels:
app: oe-helloworld
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: agentpool
operator: In
values:
- acc # this is the name of your confidential computing nodel pool
- acc_second # this is the name of your confidential computing nodel pool
containers:
- name: oe-helloworld
image: mcr.microsoft.com/acc/samples/oe-helloworld:latest
resources:
limits:
sgx.intel.com/epc: "10Mi"
requests:
sgx.intel.com/epc: "10Mi"
volumeMounts:
- name: var-run-aesmd
mountPath: /var/run/aesmd
restartPolicy: "Never"
volumes:
- name: var-run-aesmd
hostPath:
path: /var/run/aesmd
backoffLimit: 0
Now use the kubectl apply
command to create a sample job that will open in a secure enclave, as shown in the following example output:
kubectl apply -f hello-world-enclave.yaml
job "oe-helloworld" created
You can confirm that the workload successfully created a Trusted Execution Environment (enclave) by running the following commands:
kubectl get jobs -l app=oe-helloworld
NAME COMPLETIONS DURATION AGE
oe-helloworld 1/1 1s 23s
kubectl get pods -l app=oe-helloworld
NAME READY STATUS RESTARTS AGE
oe-helloworld-rchvg 0/1 Completed 0 25s
kubectl logs -l app=oe-helloworld
Hello world from the enclave
Enclave called into host to print: Hello World!
To remove the confidential computing node pool that you created in this quickstart, use the following command:
az aks nodepool delete --cluster-name myAKSCluster --name confcompool1 --resource-group myResourceGroup
To delete the AKS cluster, use the following command:
az aks delete --resource-group myResourceGroup --cluster-name myAKSCluster
-
Run Python, Node, or other applications through confidential containers using ISV/OSS SGX wrapper software. Review confidential container samples in GitHub.
-
Run enclave-aware applications by using the enclave-aware Azure container samples in GitHub.