Deploy custom policy as part of cluster deployment

01-prerequisites.md

@@ -15,6 +15,7 @@ This is the starting point for the instructions on deploying the [AKS Baseline r
    >
    > * [Contributor role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#contributor) is _required_ at the subscription level to have the ability to create resource groups and perform deployments.
    > * [User Access Administrator role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator) is _required_ at the subscription level since you'll be performing role assignments to managed identities across various resource groups.
+   > * [Resource Policy Contributor role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#resource-policy-contributor) is _required_ at the subscription level since you'll be creating custom Azure policy definitions to govern resources in your AKS cluster.
 
 1. An Azure AD tenant to associate your Kubernetes RBAC Cluster API authentication to.
 

08-workload-prerequisites.md

@@ -47,7 +47,9 @@ The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping
 
 ## Check Azure Policies are in place
 
-> :book: The app team wants to apply Azure Policy over their cluster like they do other Azure resources. Their pods will be covered using the [Azure Policy add-on for AKS](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy). Some of these audits might end up in the denial of a specific Kubernetes API request operation to ensure the pod's specification is compliance with the organization's security best practices. Moreover [data is generated by Azure Policy](https://learn.microsoft.com/azure/governance/policy/how-to/get-compliance-data) to assist the app team in the process of assessing the current compliance state of the AKS cluster. The app team is going to assign at the resource group level the [Azure Policy for Kubernetes built-in restricted initiative](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy#built-in-policy-initiatives) as well as five more [built-in individual Azure policies](https://learn.microsoft.com/azure/aks/policy-samples#microsoftcontainerservice) that enforce that pods perform resource requests, define trusted container registries, mandate that root filesystem access is read-only, enforce the usage of internal load balancers, and enforce https-only Kubernetes Ingress objects.
+> :book: The app team wants to apply Azure Policy over their cluster like they do other Azure resources. Their pods will be covered using the [Azure Policy add-on for AKS](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy). Some of these audits might end up in the denial of a specific Kubernetes API request operation to ensure the pod's specification is compliant with the organization's security best practices. Moreover [data is generated by Azure Policy](https://learn.microsoft.com/azure/governance/policy/how-to/get-compliance-data) to assist the app team in the process of assessing the current compliance state of the AKS cluster. The app team is going to assign at the resource group level the [Azure Policy for Kubernetes built-in restricted initiative](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy#built-in-policy-initiatives) as well as five more [built-in individual Azure policies](https://learn.microsoft.com/azure/aks/policy-samples#microsoftcontainerservice) that enforce that pods perform resource requests, define trusted container registries, mandate that root filesystem access is read-only, enforce the usage of internal load balancers, and enforce https-only Kubernetes Ingress objects.
+>
+> Beyond that, internal governance requires the team to ensure that any public endpoint is exposed through a full-qualified domain name ends with a company-owned domain suffix. To enforce this for all endpoints exposed by the cluster's ingress controller, they define a custom policy using [Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/docs/) and leverage the capability to [deploy it via Azure Policy](https://learn.microsoft.com/azure/aks/use-azure-policy#create-and-assign-a-custom-policy-definition) to their cluster.
 
 1. Confirm policies are applied to the AKS cluster
 
@@ -58,12 +60,13 @@ The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping
    A similar output as the one showed below should be returned
 
    ```output
-   NAME                                     AGE
-   k8sazureallowedcapabilities              21m
-   k8sazureallowedseccomp                   21m
-   … more …
-   k8sazurereadonlyrootfilesystem           21m
-   k8sazurevolumetypes                      21m
+   NAME                                                 AGE
+   k8sazureallowedcapabilities                          21m
+   k8sazureallowedseccomp                               21m
+   … more …            
+   k8sazurereadonlyrootfilesystem                       21m
+   k8sazurevolumetypes                                  21m
+   k8scustomingresstlshostshavedefineddomainsuffix      21m
    ```
 
 ### Save your work in-progress

11-validation.md

@@ -40,6 +40,49 @@ If instead Kubernetes RBAC is backed directly by Azure AD, then you'll need to e
 
 No matter which backing store you use, the user assigned to the group will then be able to `az aks get-credentials` to the cluster and you can validate that user is limited to a _read only_ view of the a0008 namespace.
 
+## Validate Azure Policy
+
+Built-in as well as custom policies are applied to the cluster as part of the [cluster deployment step](./06-aks-cluster.md) to ensure that workloads deployed to the cluster comply with the team's governance rules. Policy assignments with effect [`audit`](https://learn.microsoft.com/azure/governance/policy/concepts/effects#audit) will create a warning in the activity log and show violations in the Azure Policy blade in the portal, providing an aggregated view of the compliance state and the option to identify violating resources. Policy assignments with effect [`deny`](https://learn.microsoft.com/azure/governance/policy/concepts/effects#deny) will be enforced with the help of [Gatekeeper's admission controller webhook](https://open-policy-agent.github.io/gatekeeper/website/docs/) by denying API requests that would violate a policy otherwise.
+
+:bulb: Gatekeeper policies are implemented in the [policy language 'Rego'](https://learn.microsoft.com/azure/governance/policy/concepts/policy-for-kubernetes#policy-language). To deploy the policy of this reference architecture with the Azure platform, the Rego specification is Base64-encoded and stored in a field of the Azure Policy resource defined in `nested_K8sCustomIngressTlsHostsHaveDefinedDomainSuffix.bicep`. It might be insightful to decode the string with an Base64 decoder of your choice and investigate the declarative implementation. 
+
+### Steps
+
+1. Try to add a second `Ingress` resource to your workload namespace with the following command.
+
+    Notice that the host value specified in the `rules` and the `tls` sections defines a domain name with suffix `invalid-domain.com` rather than the domain suffix you defined for your setup when you [created your certificates](./02-ca-certificates.md)).
+
+   ```bash
+   cat <<EOF | kubectl create -f -
+   apiVersion: networking.k8s.io/v1
+   kind: Ingress
+   metadata:
+     name: aspnetapp-ingress-violating
+     namespace: a0008
+   spec:
+     tls:
+     - hosts:
+       - bu0001a0008-00.aks-ingress.invalid-domain.com
+     rules:
+     - host: bu0001a0008-00.aks-ingress.invalid-domain.com
+       http:
+         paths:
+         - path: /
+           pathType: Prefix
+           backend:
+             service:
+               name: aspnetapp-service
+               port:
+                 number: 80
+   EOF
+   ```
+
+2. Inspect the error message and remark that Gatekeeper's admission webhook rejects `bu0001a0008-00.aks-ingress.invalid-domain.com` as incompliant host.
+
+   ```output
+   Error from server (Forbidden): error when creating "STDIN": admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8scustomingresstlshostshavede-e64871e795ce3239cd99] TLS host must have one of defined domain suffixes. Valid domain names are ["contoso.com"]; defined TLS hosts are {"bu0001a0008-00.aks-ingress.invalid-domain.com"}; incompliant hosts are {"bu0001a0008-00.aks-ingress.invalid-domain.com"}.
+   ```
+
 ## Validate Web Application Firewall functionality
 
 Your workload is placed behind a Web Application Firewall (WAF), which has rules designed to stop intentionally malicious activity. You can test this by triggering one of the built-in rules with a request that looks malicious.

README.md

@@ -103,7 +103,6 @@ Kubernetes and, by extension, AKS are fast-evolving products. The [AKS roadmap](
 This implementation will not include every preview feature, but instead only those that add significant value to a general-purpose cluster. There are some additional preview features you may wish to evaluate in pre-production clusters that augment your posture around security, manageability, etc. As these features come out of preview, this reference implementation may be updated to incorporate them. Consider trying out and providing feedback on the following:
 
 - [BYO Kubelet Identity](https://learn.microsoft.com/azure/aks/use-managed-identity#bring-your-own-kubelet-mi)
-- [Custom Azure Policy for Kubernetes support](https://techcommunity.microsoft.com/t5/azure-governance-and-management/azure-policy-for-kubernetes-releases-support-for-custom-policy/ba-p/2699466)
 - [Planned maintenance window](https://learn.microsoft.com/azure/aks/planned-maintenance)
 - [BYO CNI (`--network-plugin none`)](https://learn.microsoft.com/azure/aks/use-byo-cni)
 - [Simplified application autoscaling with Kubernetes Event-driven Autoscaling (KEDA) add-on](https://learn.microsoft.com/azure/aks/keda)

cluster-stamp.bicep

@@ -1411,6 +1411,38 @@ resource paManagedIdentitiesEnabled 'Microsoft.Authorization/policyAssignments@2
   }
 }
 
+// Deploying and applying the custom policy 'Kubernetes cluster ingress TLS hosts must have defined domain suffix' as defined in nested_K8sCustomIngressTlsHostsHaveDefinedDomainSuffix.bicep
+// Note: Policy definition must be deployed as module since policy definitions require a targetScope of 'subscription'.
+
+module modK8sIngressTlsHostsHaveDefinedDomainSuffix 'nested_K8sCustomIngressTlsHostsHaveDefinedDomainSuffix.bicep' = {
+  name: 'modK8sIngressTlsHostsHaveDefinedDomainSuffix'
+  scope: subscription()
+}
+
+resource paK8sIngressTlsHostsHaveSpecificDomainSuffix 'Microsoft.Authorization/policyAssignments@2021-06-01' = {
+  name: guid('K8sCustomIngressTlsHostsHaveDefinedDomainSuffix', resourceGroup().id, clusterName)
+  location: 'global'
+  scope: resourceGroup()
+  properties: {
+    displayName: take('[${clusterName}] ${modK8sIngressTlsHostsHaveDefinedDomainSuffix.outputs.policyName}', 120)
+    description: modK8sIngressTlsHostsHaveDefinedDomainSuffix.outputs.policyDescription
+    policyDefinitionId: modK8sIngressTlsHostsHaveDefinedDomainSuffix.outputs.policyId
+    parameters: {
+      excludedNamespaces: {
+        value: []
+      }
+      effect: {
+        value: 'deny'
+      }
+      allowedDomainSuffixes: {
+        value: [
+          domainName
+        ]
+      }
+    }
+  }
+}
+
 // The control plane identity used by the cluster. Used for networking access (VNET joining and DNS updating)
 resource miClusterControlPlane 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = {
   name: 'mi-${clusterName}-controlplane'
@@ -1817,6 +1849,7 @@ resource mc 'Microsoft.ContainerService/managedClusters@2022-03-02-preview' = {
     paEnforceInternalLoadBalancers
     paEnforceResourceLimits
     paRoRootFilesystem
+    paK8sIngressTlsHostsHaveSpecificDomainSuffix
 
     // Azure Resource Provider policies that we'd like to see in place before the cluster is deployed
     // They are not technically a dependency, but logically they would have existed on the resource group

nested_K8sCustomIngressTlsHostsHaveDefinedDomainSuffix.bicep

@@ -0,0 +1,153 @@
+targetScope = 'subscription'
+
+/*** RESOURCES ***/
+
+resource pdK8sCustomIngressTlsHostsHaveDefinedDomainSuffix 'Microsoft.Authorization/policyDefinitions@2021-06-01' = {
+  scope: subscription()
+  name: 'K8sCustomIngressTlsHostsHaveDefinedDomainSuffix'
+  properties: {
+    policyType: 'Custom'
+    mode: 'Microsoft.Kubernetes.Data'
+    displayName: 'Kubernetes cluster ingress TLS hosts must have defined domain suffix'
+    description: 'Kubernetes cluster ingress TLS hosts must have defined domain suffix'
+    policyRule: {
+      if: {
+        field: 'type'
+        in: [
+          'Microsoft.ContainerService/managedClusters'
+        ]
+      }
+      then: {
+        effect: '[parameters(\'effect\')]'
+        details: {
+          templateInfo: {
+            sourceType: 'Base64Encoded'
+            content: 'YXBpVmVyc2lvbjogdGVtcGxhdGVzLmdhdGVrZWVwZXIuc2gvdjFiZXRhMQpraW5kOiBDb25zdHJhaW50VGVtcGxhdGUKbWV0YWRhdGE6CiAgbmFtZTogazhzY3VzdG9taW5ncmVzc3Rsc2hvc3RzaGF2ZWRlZmluZWRkb21haW5zdWZmaXgKc3BlYzoKICBjcmQ6CiAgICBzcGVjOgogICAgICBuYW1lczoKICAgICAgICBraW5kOiBrOHNjdXN0b21pbmdyZXNzdGxzaG9zdHNoYXZlZGVmaW5lZGRvbWFpbnN1ZmZpeAogICAgICB2YWxpZGF0aW9uOgogICAgICAgIG9wZW5BUElWM1NjaGVtYToKICAgICAgICAgIHR5cGU6IG9iamVjdAogICAgICAgICAgcHJvcGVydGllczoKICAgICAgICAgICAgYWxsb3dlZERvbWFpblN1ZmZpeGVzOgogICAgICAgICAgICAgIHR5cGU6IGFycmF5CiAgICAgICAgICAgICAgaXRlbXM6CiAgICAgICAgICAgICAgICB0eXBlOiBzdHJpbmcKICB0YXJnZXRzOgogICAgLSB0YXJnZXQ6IGFkbWlzc2lvbi5rOHMuZ2F0ZWtlZXBlci5zaAogICAgICByZWdvOiB8CiAgICAgICAgcGFja2FnZSBrOHNjdXN0b21pbmdyZXNzdGxzaG9zdHNoYXZlZGVmaW5lZGRvbWFpbnN1ZmZpeAoKICAgICAgICB2aW9sYXRpb25beyJtc2ciOiBtc2csICJkZXRhaWxzIjoge319XSB7CiAgICAgICAgICBhbGxvd2VkRG9tYWluU3VmZml4ZXMgOj0gaW5wdXQucGFyYW1ldGVycy5hbGxvd2VkRG9tYWluU3VmZml4ZXMKICAgICAgICAgIGFsbERlZmluZWRIb3N0cyA6PSB7biB8IG4gOj0gaW5wdXQucmV2aWV3Lm9iamVjdC5zcGVjLnRsc1tfXS5ob3N0c1tfXX0KICAgICAgICAgIG1hdGNoZWRIb3N0cyA6PSB7IHsiaG9zdCI6IGgsICJtYXRjaGVkRG9tYWlucyI6IGR9IHwKICAgICAgICAgICAgaCA6PSBhbGxEZWZpbmVkSG9zdHNbX10gOwogICAgICAgICAgICBkIDo9IHsgbiB8IG4gOj0gYWxsb3dlZERvbWFpblN1ZmZpeGVzW19dIDsgZW5kc3dpdGgoaCwgbil9CiAgICAgICAgICB9CiAgICAgICAgICB1bm1hdGNoZWRIb3N0cyA6PSB7IGggfAogICAgICAgICAgICBoIDo9IG1hdGNoZWRIb3N0c1t4XS5ob3N0IDsgCiAgICAgICAgICAgIGQgOj0gbWF0Y2hlZEhvc3RzW3hdLm1hdGNoZWREb21haW5zIDsgCiAgICAgICAgICAgIGNvdW50KGQpID09IDAKICAgICAgICAgIH0KICAgICAgICAgIGNvdW50KHVubWF0Y2hlZEhvc3RzKSA+IDAKICAgICAgICAgIG1zZyA6PSBzcHJpbnRmKCJUTFMgaG9zdCBtdXN0IGhhdmUgb25lIG9mIGRlZmluZWQgZG9tYWluIHN1ZmZpeGVzLiBWYWxpZCBkb21haW4gbmFtZXMgYXJlICV2OyBkZWZpbmVkIFRMUyBob3N0cyBhcmUgJXY7IGluY29tcGxpYW50IGhvc3RzIGFyZSAldi4iLCBbYWxsb3dlZERvbWFpblN1ZmZpeGVzLCBhbGxEZWZpbmVkSG9zdHMsIHVubWF0Y2hlZEhvc3RzXSkKICAgICAgICB9'
+          }
+          apiGroups: [
+            'networking.k8s.io'
+          ]
+          kinds: [
+            'Ingress'
+          ]
+          namespaces: '[parameters(\'namespaces\')]'
+          excludedNamespaces: '[parameters(\'excludedNamespaces\')]'
+          labelSelector: '[parameters(\'labelSelector\')]'
+          values: {
+            allowedDomainSuffixes: '[parameters(\'allowedDomainSuffixes\')]'
+          }
+        }
+      }
+    }
+    parameters: {
+      effect: {
+        type: 'String'
+        metadata: {
+          displayName: 'Effect'
+          description: '\'audit\' allows a non-compliant resource to be created or updated, but flags it as non-compliant. \'deny\' blocks the non-compliant resource creation or update. \'disabled\' turns off the policy.'
+        }
+        allowedValues: [
+          'audit'
+          'Audit'
+          'deny'
+          'Deny'
+          'disabled'
+          'Disabled'
+        ]
+        defaultValue: 'audit'
+      }
+      excludedNamespaces: {
+        type: 'Array'
+        metadata: {
+          displayName: 'Namespace exclusions'
+          description: 'List of Kubernetes namespaces to exclude from policy evaluation.'
+        }
+        defaultValue: [
+          'kube-system'
+          'gatekeeper-system'
+          'azure-arc'
+        ]
+      }
+      namespaces: {
+        type: 'Array'
+        metadata: {
+          displayName: 'Namespace inclusions'
+          description: 'List of Kubernetes namespaces to only include in policy evaluation. An empty list means the policy is applied to all resources in all namespaces.'
+        }
+        defaultValue: []
+      }
+      labelSelector: {
+        type: 'Object'
+        metadata: {
+          displayName: 'Kubernetes label selector'
+          description: 'Label query to select Kubernetes resources for policy evaluation. An empty label selector matches all Kubernetes resources.'
+        }
+        defaultValue: {
+        }
+        schema: {
+          description: 'A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all resources.'
+          type: 'object'
+          properties: {
+            matchLabels: {
+              description: 'matchLabels is a map of {key,value} pairs.'
+              type: 'object'
+              additionalProperties: {
+                type: 'string'
+              }
+              minProperties: 1
+            }
+            matchExpressions: {
+              description: 'matchExpressions is a list of values, a key, and an operator.'
+              type: 'array'
+              items: {
+                type: 'object'
+                properties: {
+                  key: {
+                    description: 'key is the label key that the selector applies to.'
+                    type: 'string'
+                  }
+                  operator: {
+                    description: 'operator represents a key\'s relationship to a set of values.'
+                    type: 'string'
+                    enum: [
+                      'In'
+                      'NotIn'
+                      'Exists'
+                      'DoesNotExist'
+                    ]
+                  }
+                  values: {
+                    description: 'values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty.'
+                    type: 'array'
+                    items: {
+                      type: 'string'
+                    }
+                  }
+                }
+                required: [
+                  'key'
+                  'operator'
+                ]
+                additionalProperties: false
+              }
+              minItems: 1
+            }
+          }
+          additionalProperties: false
+        }
+      }
+      allowedDomainSuffixes: {
+        type: 'Array'
+        metadata: {
+          displayName: 'List of compliant domain suffixes'
+          description: 'List of compliant domain suffixes'
+        }
+      }
+    }
+  }
+}
+
+
+output policyId string = pdK8sCustomIngressTlsHostsHaveDefinedDomainSuffix.id
+output policyName string = pdK8sCustomIngressTlsHostsHaveDefinedDomainSuffix.properties.displayName
+output policyDescription string = pdK8sCustomIngressTlsHostsHaveDefinedDomainSuffix.properties.description