From 21b723d9691ce0d60b85c4077de2a551426a15a5 Mon Sep 17 00:00:00 2001 From: Chad Kittel Date: Fri, 16 Sep 2022 09:51:05 -0500 Subject: [PATCH] url updates --- 01-prerequisites.md | 18 ++++---- 03-aad.md | 2 +- 04-networking.md | 2 +- 06-aks-cluster.md | 2 +- 08-workload-prerequisites.md | 6 +-- 11-validation.md | 10 ++--- README.md | 32 +++++++------- acr-stamp.bicep | 2 +- cluster-manifests/README.md | 2 +- .../container-azm-ms-agentconfig.yaml | 2 +- cluster-stamp.bicep | 2 +- contoso-bicycle/README.md | 2 +- github-workflow/README.md | 2 +- github-workflow/aks-deploy.yaml | 2 +- networking/README.md | 2 +- networking/hub-default.bicep | 2 +- networking/hub-regionA.bicep | 2 +- networking/spoke-BU0001A0008.bicep | 4 +- networking/topology.md | 42 +++++++++---------- workload/aspnetapp.yaml | 2 +- 20 files changed, 70 insertions(+), 70 deletions(-) diff --git a/01-prerequisites.md b/01-prerequisites.md index 0eb9737c..af3e95c9 100644 --- a/01-prerequisites.md +++ b/01-prerequisites.md @@ -10,25 +10,25 @@ This is the starting point for the instructions on deploying the [AKS Baseline r > :warning: The user or service principal initiating the deployment process _must_ have the following minimal set of Azure Role-Based Access Control (RBAC) roles: > - > * [Contributor role](https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#contributor) is _required_ at the subscription level to have the ability to create resource groups and perform deployments. - > * [User Access Administrator role](https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator) is _required_ at the subscription level since you'll be performing role assignments to managed identities across various resource groups. + > * [Contributor role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#contributor) is _required_ at the subscription level to have the ability to create resource groups and perform deployments. + > * [User Access Administrator role](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator) is _required_ at the subscription level since you'll be performing role assignments to managed identities across various resource groups. 1. An Azure AD tenant to associate your Kubernetes RBAC Cluster API authentication to. > :warning: The user or service principal initiating the deployment process _must_ have the following minimal set of Azure AD permissions assigned: > - > * Azure AD [User Administrator](https://docs.microsoft.com/azure/active-directory/users-groups-roles/directory-assign-admin-roles#user-administrator-permissions) is _required_ to create a "break glass" AKS admin Active Directory Security Group and User. Alternatively, you could get your Azure AD admin to create this for you when instructed to do so. - > * If you are not part of the User Administrator group in the tenant associated to your Azure subscription, please consider [creating a new tenant](https://docs.microsoft.com/azure/active-directory/fundamentals/active-directory-access-create-new-tenant#create-a-new-tenant-for-your-organization) to use while evaluating this implementation. The Azure AD tenant backing your cluster's API RBAC does NOT need to be the same tenant associated with your Azure subscription. + > * Azure AD [User Administrator](https://learn.microsoft.com/azure/active-directory/users-groups-roles/directory-assign-admin-roles#user-administrator-permissions) is _required_ to create a "break glass" AKS admin Active Directory Security Group and User. Alternatively, you could get your Azure AD admin to create this for you when instructed to do so. + > * If you are not part of the User Administrator group in the tenant associated to your Azure subscription, please consider [creating a new tenant](https://learn.microsoft.com/azure/active-directory/fundamentals/active-directory-access-create-new-tenant#create-a-new-tenant-for-your-organization) to use while evaluating this implementation. The Azure AD tenant backing your cluster's API RBAC does NOT need to be the same tenant associated with your Azure subscription. -1. Latest [Azure CLI installed](https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest) (must be at least 2.37), or you can perform this from Azure Cloud Shell by clicking below. +1. Latest [Azure CLI installed](https://learn.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest) (must be at least 2.37), or you can perform this from Azure Cloud Shell by clicking below. - [![Launch Azure Cloud Shell](https://docs.microsoft.com/azure/includes/media/cloud-shell-try-it/launchcloudshell.png)](https://shell.azure.com) + [![Launch Azure Cloud Shell](https://learn.microsoft.com/azure/includes/media/cloud-shell-try-it/launchcloudshell.png)](https://shell.azure.com) 1. While the following feature(s) are still in _preview_, please enable them in your target subscription. - 1. [Register the Azure Event Grid preview feature - `EventgridPreview`](https://docs.microsoft.com/azure/aks/quickstart-event-grid#register-the-eventgridpreview-preview-feature) + 1. [Register the Azure Event Grid preview feature - `EventgridPreview`](https://learn.microsoft.com/azure/aks/quickstart-event-grid#register-the-eventgridpreview-preview-feature) - 1. [Register the OIDC Issuer preview feature = `EnableOIDCIssuerPreview`](https://docs.microsoft.com/azure/aks/cluster-configuration#oidc-issuer-preview) + 1. [Register the OIDC Issuer preview feature = `EnableOIDCIssuerPreview`](https://learn.microsoft.com/azure/aks/cluster-configuration#oidc-issuer-preview) ```bash az feature register --namespace "Microsoft.ContainerService" -n "EventgridPreview" @@ -55,7 +55,7 @@ This is the starting point for the instructions on deploying the [AKS Baseline r cd aks-baseline ``` - > :bulb: The steps shown here and elsewhere in the reference implementation use Bash shell commands. On Windows, you can use the [Windows Subsystem for Linux](https://docs.microsoft.com/windows/wsl/about) to run Bash. + > :bulb: The steps shown here and elsewhere in the reference implementation use Bash shell commands. On Windows, you can use the [Windows Subsystem for Linux](https://learn.microsoft.com/windows/wsl/about) to run Bash. 1. Ensure [OpenSSL is installed](https://github.com/openssl/openssl#download) in order to generate self-signed certs used in this implementation. _OpenSSL is already installed in Azure Cloud Shell._ diff --git a/03-aad.md b/03-aad.md index ef919245..b3811161 100644 --- a/03-aad.md +++ b/03-aad.md @@ -92,7 +92,7 @@ AKS supports backing Kubernetes with Azure AD in two different modalities. One i If you are using a single tenant for this walk-through, the cluster deployment step later will take care of the necessary role assignments for the groups created above. Specifically, in the above steps, you created the Azure AD security group `cluster-ns-a0008-readers-bu0001a000800` that is going to be a namespace reader in namespace `a0008` and the Azure AD security group `cluster-admins-bu0001a000800` is going to contain cluster admins. Those group Object IDs will be associated to the 'Azure Kubernetes Service RBAC Reader' and 'Azure Kubernetes Service RBAC Cluster Admin' RBAC role respectively, scoped to their proper level within the cluster. -Using Azure RBAC as your authorization approach is ultimately preferred as it allows for the unified management and access control across Azure Resources, AKS, and Kubernetes resources. At the time of this writing there are four [Azure RBAC roles](https://docs.microsoft.com/azure/aks/manage-azure-rbac#create-role-assignments-for-users-to-access-cluster) that represent typical cluster access patterns. +Using Azure RBAC as your authorization approach is ultimately preferred as it allows for the unified management and access control across Azure Resources, AKS, and Kubernetes resources. At the time of this writing there are four [Azure RBAC roles](https://learn.microsoft.com/azure/aks/manage-azure-rbac#create-role-assignments-for-users-to-access-cluster) that represent typical cluster access patterns. ### Direct Kubernetes RBAC management _[Alternative]_ diff --git a/04-networking.md b/04-networking.md index 345a39a1..04a7eceb 100644 --- a/04-networking.md +++ b/04-networking.md @@ -4,7 +4,7 @@ The prerequisites for the [AKS Baseline cluster](./) are now completed with [Azu ## Subscription and resource group topology -This reference implementation is split across several resource groups in a single subscription. This is to replicate the fact that many organizations will split certain responsibilities into specialized subscriptions (e.g. regional hubs/vwan in a _Connectivity_ subscription and workloads in landing zone subscriptions). We expect you to explore this reference implementation within a single subscription, but when you implement this cluster at your organization, you will need to take what you've learned here and apply it to your expected subscription and resource group topology (such as those [offered by the Cloud Adoption Framework](https://docs.microsoft.com/azure/cloud-adoption-framework/decision-guides/subscriptions/).) This single subscription, multiple resource group model is for simplicity of demonstration purposes only. +This reference implementation is split across several resource groups in a single subscription. This is to replicate the fact that many organizations will split certain responsibilities into specialized subscriptions (e.g. regional hubs/vwan in a _Connectivity_ subscription and workloads in landing zone subscriptions). We expect you to explore this reference implementation within a single subscription, but when you implement this cluster at your organization, you will need to take what you've learned here and apply it to your expected subscription and resource group topology (such as those [offered by the Cloud Adoption Framework](https://learn.microsoft.com/azure/cloud-adoption-framework/decision-guides/subscriptions/).) This single subscription, multiple resource group model is for simplicity of demonstration purposes only. ## Expected results diff --git a/06-aks-cluster.md b/06-aks-cluster.md index 7e667083..20f4f432 100644 --- a/06-aks-cluster.md +++ b/06-aks-cluster.md @@ -17,7 +17,7 @@ Now that your [ACR instance is deployed and ready to support cluster bootstrappi ``` 1. Deploy the cluster ARM template. - :exclamation: By default, this deployment will allow unrestricted access to your cluster's API Server. You can limit access to the API Server to a set of well-known IP addresses (i.,e. a jump box subnet (connected to by Azure Bastion), build agents, or any other networks you'll administer the cluster from) by setting the `clusterAuthorizedIPRanges` parameter in all deployment options. This setting will also impact traffic originating from within the cluster trying to use the API server, so you will also need to include _all_ of the public IPs used by your egress Azure Firewall. For more information, see [Secure access to the API server using authorized IP address ranges](https://docs.microsoft.com/azure/aks/api-server-authorized-ip-ranges#create-an-aks-cluster-with-api-server-authorized-ip-ranges-enabled). + :exclamation: By default, this deployment will allow unrestricted access to your cluster's API Server. You can limit access to the API Server to a set of well-known IP addresses (i.,e. a jump box subnet (connected to by Azure Bastion), build agents, or any other networks you'll administer the cluster from) by setting the `clusterAuthorizedIPRanges` parameter in all deployment options. This setting will also impact traffic originating from within the cluster trying to use the API server, so you will also need to include _all_ of the public IPs used by your egress Azure Firewall. For more information, see [Secure access to the API server using authorized IP address ranges](https://learn.microsoft.com/azure/aks/api-server-authorized-ip-ranges#create-an-aks-cluster-with-api-server-authorized-ip-ranges-enabled). **Option 1 - Deploy from the command line** diff --git a/08-workload-prerequisites.md b/08-workload-prerequisites.md index 6eb025f7..fff23976 100644 --- a/08-workload-prerequisites.md +++ b/08-workload-prerequisites.md @@ -27,7 +27,7 @@ The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping 1. Import the AKS ingress controller's wildcard certificate for `*.aks-ingress.contoso.com`. - :warning: If you already have access to an [appropriate certificate](https://docs.microsoft.com/azure/key-vault/certificates/certificate-scenarios#formats-of-import-we-support), or can procure one from your organization, consider using it for this step. For more information, please take a look at the [import certificate tutorial using Azure Key Vault](https://docs.microsoft.com/azure/key-vault/certificates/tutorial-import-certificate#import-a-certificate-to-key-vault). + :warning: If you already have access to an [appropriate certificate](https://learn.microsoft.com/azure/key-vault/certificates/certificate-scenarios#formats-of-import-we-support), or can procure one from your organization, consider using it for this step. For more information, please take a look at the [import certificate tutorial using Azure Key Vault](https://learn.microsoft.com/azure/key-vault/certificates/tutorial-import-certificate#import-a-certificate-to-key-vault). :warning: Do not use the certificate created by this script for actual deployments. The use of self-signed certificates are provided for ease of illustration purposes only. For your cluster, use your organization's requirements for procurement and lifetime management of TLS certificates, _even for development purposes_. @@ -38,7 +38,7 @@ The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping 1. Remove Azure Key Vault import certificates permissions and network access for current user. - > The Azure Key Vault RBAC assignment for your user and network allowance was temporary to allow you to upload the certificate for this walkthrough. In actual deployments, you would manage these any RBAC policies via your ARM templates using [Azure RBAC for Key Vault data plane](https://docs.microsoft.com/azure/key-vault/general/secure-your-key-vault#data-plane-and-access-policies) and only network-allowed traffic would access your Key Vault. + > The Azure Key Vault RBAC assignment for your user and network allowance was temporary to allow you to upload the certificate for this walkthrough. In actual deployments, you would manage these any RBAC policies via your ARM templates using [Azure RBAC for Key Vault data plane](https://learn.microsoft.com/azure/key-vault/general/secure-your-key-vault#data-plane-and-access-policies) and only network-allowed traffic would access your Key Vault. ```bash az keyvault network-rule remove -n $KEYVAULT_NAME_AKS_BASELINE --ip-address "${CURRENT_IP_ADDRESS}/32" @@ -47,7 +47,7 @@ The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping ## Check Azure Policies are in place -> :book: The app team wants to apply Azure Policy over their cluster like they do other Azure resources. Their pods will be covered using the [Azure Policy add-on for AKS](https://docs.microsoft.com/azure/aks/use-pod-security-on-azure-policy). Some of these audits might end up in the denial of a specific Kubernetes API request operation to ensure the pod's specification is compliance with the organization's security best practices. Moreover [data is generated by Azure Policy](https://docs.microsoft.com/azure/governance/policy/how-to/get-compliance-data) to assist the app team in the process of assessing the current compliance state of the AKS cluster. The app team is going to assign at the resource group level the [Azure Policy for Kubernetes built-in restricted initiative](https://docs.microsoft.com/azure/aks/use-pod-security-on-azure-policy#built-in-policy-initiatives) as well as five more [built-in individual Azure policies](https://docs.microsoft.com/azure/aks/policy-samples#microsoftcontainerservice) that enforce that pods perform resource requests, define trusted container registries, mandate that root filesystem access is read-only, enforce the usage of internal load balancers, and enforce https-only Kubernetes Ingress objects. +> :book: The app team wants to apply Azure Policy over their cluster like they do other Azure resources. Their pods will be covered using the [Azure Policy add-on for AKS](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy). Some of these audits might end up in the denial of a specific Kubernetes API request operation to ensure the pod's specification is compliance with the organization's security best practices. Moreover [data is generated by Azure Policy](https://learn.microsoft.com/azure/governance/policy/how-to/get-compliance-data) to assist the app team in the process of assessing the current compliance state of the AKS cluster. The app team is going to assign at the resource group level the [Azure Policy for Kubernetes built-in restricted initiative](https://learn.microsoft.com/azure/aks/use-pod-security-on-azure-policy#built-in-policy-initiatives) as well as five more [built-in individual Azure policies](https://learn.microsoft.com/azure/aks/policy-samples#microsoftcontainerservice) that enforce that pods perform resource requests, define trusted container registries, mandate that root filesystem access is read-only, enforce the usage of internal load balancers, and enforce https-only Kubernetes Ingress objects. 1. Confirm policies are applied to the AKS cluster diff --git a/11-validation.md b/11-validation.md index 2e2aebe2..18ae5e4c 100644 --- a/11-validation.md +++ b/11-validation.md @@ -63,7 +63,7 @@ Your workload is placed behind a Web Application Firewall (WAF), which has rules ## Validate Cluster Azure Monitor Insights and Logs -Monitoring your cluster is critical, especially when you're running a production cluster. Therefore, your AKS cluster is configured to send [diagnostic information](https://docs.microsoft.com/azure/aks/monitor-aks) of categories _cluster-autoscaler_, _kube-controller-manager_, _kube-audit-admin_ and _guard_ to the Log Analytics Workspace deployed as part of the [bootstrapping step](./05-bootstrap-prep.md). Additionally, [Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-overview) is configured on your cluster to capture metrics and logs from your workload containers. Azure Monitor is configured to surface cluster logs, here you can see those logs as they are generated. +Monitoring your cluster is critical, especially when you're running a production cluster. Therefore, your AKS cluster is configured to send [diagnostic information](https://learn.microsoft.com/azure/aks/monitor-aks) of categories _cluster-autoscaler_, _kube-controller-manager_, _kube-audit-admin_ and _guard_ to the Log Analytics Workspace deployed as part of the [bootstrapping step](./05-bootstrap-prep.md). Additionally, [Azure Monitor for containers](https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-overview) is configured on your cluster to capture metrics and logs from your workload containers. Azure Monitor is configured to surface cluster logs, here you can see those logs as they are generated. :bulb: If you need to inspect the behavior of the Kubernetes scheduler, enable the log category _kube-scheduler_ (either through the _Diagnostic Settings_ blade of your AKS cluster or by enabling the category in your `cluster-stamp.bicep` template). Note that this category is quite verbose and will impact the cost of your Log Analytics Workspace. @@ -72,7 +72,7 @@ Monitoring your cluster is critical, especially when you're running a production 1. In the Azure Portal, navigate to your AKS cluster resource. 1. Click _Insights_ to see captured data. -You can also execute [queries](https://docs.microsoft.com/azure/azure-monitor/log-query/get-started-portal) on the [cluster logs captured](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-log-search). +You can also execute [queries](https://learn.microsoft.com/azure/azure-monitor/log-query/get-started-portal) on the [cluster logs captured](https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-log-search). 1. In the Azure Portal, navigate to your AKS cluster resource. 1. Click _Logs_ to see and query log data. @@ -80,7 +80,7 @@ You can also execute [queries](https://docs.microsoft.com/azure/azure-monitor/lo ## Validate Azure Monitor for containers (Prometheus Metrics) -Azure Monitor is configured to [scrape Prometheus metrics](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-prometheus-integration) in your cluster. This reference implementation is configured to collect Prometheus metrics from two namespaces, as configured in [`container-azm-ms-agentconfig.yaml`](./cluster-baseline-settings/container-azm-ms-agentconfig.yaml). There are two pods configured to emit Prometheus metrics: +Azure Monitor is configured to [scrape Prometheus metrics](https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-prometheus-integration) in your cluster. This reference implementation is configured to collect Prometheus metrics from two namespaces, as configured in [`container-azm-ms-agentconfig.yaml`](./cluster-baseline-settings/container-azm-ms-agentconfig.yaml). There are two pods configured to emit Prometheus metrics: - [Traefik](./workload/traefik.yaml) (in the `a0008` namespace) - [Kured](./cluster-baseline-settings/kured.yaml) (in the `cluster-baseline-settings` namespace) @@ -124,13 +124,13 @@ Azure will generate alerts on the health of your cluster and adjacent resources. ### Steps -An alert based on [Azure Monitor for containers information using a Kusto query](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-alerts) was configured in this reference implementation. +An alert based on [Azure Monitor for containers information using a Kusto query](https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-alerts) was configured in this reference implementation. 1. In the Azure Portal, navigate to your AKS cluster resource group (`rg-bu0001a0008`). 1. Select _Alerts_, then _Alert Rules_. 1. There is an alert titled "[your cluster name] Scheduled Query for Pod Failed Alert" that will be triggered based on the custom query response. -An [Azure Advisor Alert](https://docs.microsoft.com/azure/advisor/advisor-overview) was configured as well in this reference implementation. +An [Azure Advisor Alert](https://learn.microsoft.com/azure/advisor/advisor-overview) was configured as well in this reference implementation. 1. In the Azure Portal, navigate to your AKS cluster resource group (`rg-bu0001a0008`). 1. Select _Alerts_, then _Alert Rules_. diff --git a/README.md b/README.md index 30f452ad..e4ac7ab8 100644 --- a/README.md +++ b/README.md @@ -25,12 +25,12 @@ Finally, this implementation uses the [ASP.NET Core Docker sample web app](https #### Azure platform - AKS v1.24 - - System and User [node pool separation](https://docs.microsoft.com/azure/aks/use-system-pools) - - [AKS-managed Azure AD](https://docs.microsoft.com/azure/aks/managed-aad) + - System and User [node pool separation](https://learn.microsoft.com/azure/aks/use-system-pools) + - [AKS-managed Azure AD](https://learn.microsoft.com/azure/aks/managed-aad) - Azure AD-backed Kubernetes RBAC (_local user accounts disabled_) - Managed Identities - Azure CNI - - [Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-overview) + - [Azure Monitor for containers](https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-overview) - Azure Virtual Networks (hub-spoke) - Azure Firewall managed egress - Azure Application Gateway (WAF) @@ -40,11 +40,11 @@ Finally, this implementation uses the [ASP.NET Core Docker sample web app](https - [Flux GitOps Operator](https://fluxcd.io) _[AKS-managed extension]_ - [Traefik Ingress Controller](https://doc.traefik.io/traefik/v2.5/routing/providers/kubernetes-ingress/) -- [Azure AD Pod Identity](https://docs.microsoft.com/azure/aks/use-azure-ad-pod-identity) -- [Secrets Store CSI Driver for Kubernetes](https://docs.microsoft.com/azure/aks/csi-secrets-store-driver) _[AKS-managed add-on]_ -- [Kured](https://docs.microsoft.com/azure/aks/node-updates-kured) +- [Azure AD Pod Identity](https://learn.microsoft.com/azure/aks/use-azure-ad-pod-identity) +- [Secrets Store CSI Driver for Kubernetes](https://learn.microsoft.com/azure/aks/csi-secrets-store-driver) _[AKS-managed add-on]_ +- [Kured](https://learn.microsoft.com/azure/aks/node-updates-kured) -![Network diagram depicting a hub-spoke network with two peered VNets and main Azure resources used in the architecture.](https://docs.microsoft.com/azure/architecture/reference-architectures/containers/aks/images/secure-baseline-architecture.svg) +![Network diagram depicting a hub-spoke network with two peered VNets and main Azure resources used in the architecture.](https://learn.microsoft.com/azure/architecture/reference-architectures/containers/aks/images/secure-baseline-architecture.svg) ## Deploy the reference implementation @@ -102,11 +102,11 @@ Kubernetes and, by extension, AKS are fast-evolving products. The [AKS roadmap]( This implementation will not include every preview feature, but instead only those that add significant value to a general-purpose cluster. There are some additional preview features you may wish to evaluate in pre-production clusters that augment your posture around security, manageability, etc. As these features come out of preview, this reference implementation may be updated to incorporate them. Consider trying out and providing feedback on the following: -- [BYO Kubelet Identity](https://docs.microsoft.com/azure/aks/use-managed-identity#bring-your-own-kubelet-mi) +- [BYO Kubelet Identity](https://learn.microsoft.com/azure/aks/use-managed-identity#bring-your-own-kubelet-mi) - [Custom Azure Policy for Kubernetes support](https://techcommunity.microsoft.com/t5/azure-governance-and-management/azure-policy-for-kubernetes-releases-support-for-custom-policy/ba-p/2699466) -- [Planned maintenance window](https://docs.microsoft.com/azure/aks/planned-maintenance) -- [BYO CNI (`--network-plugin none`)](https://docs.microsoft.com/azure/aks/use-byo-cni) -- [Simplified application autoscaling with Kubernetes Event-driven Autoscaling (KEDA) add-on](https://docs.microsoft.com/azure/aks/keda) +- [Planned maintenance window](https://learn.microsoft.com/azure/aks/planned-maintenance) +- [BYO CNI (`--network-plugin none`)](https://learn.microsoft.com/azure/aks/use-byo-cni) +- [Simplified application autoscaling with Kubernetes Event-driven Autoscaling (KEDA) add-on](https://learn.microsoft.com/azure/aks/keda) ## Related Reference Implementations @@ -122,14 +122,14 @@ The AKS Baseline was used as the foundation for the following additional referen This reference implementation intentionally does not cover more advanced scenarios. For example topics like the following are not addressed: - Cluster lifecycle management with regard to SDLC and GitOps -- Workload SDLC integration (including concepts like [Bridge to Kubernetes](https://docs.microsoft.com/visualstudio/containers/bridge-to-kubernetes), advanced deployment techniques, [Draft](https://docs.microsoft.com/azure/aks/draft), etc) +- Workload SDLC integration (including concepts like [Bridge to Kubernetes](https://learn.microsoft.com/visualstudio/containers/bridge-to-kubernetes), advanced deployment techniques, [Draft](https://learn.microsoft.com/azure/aks/draft), etc) - Container security - Multiple (related or unrelated) workloads owned by the same team - Multiple workloads owned by disparate teams (AKS as a shared platform in your organization) - Cluster-contained state (PVC, etc) - Windows node pools - Scale-to-zero node pools and event-based scaling (KEDA) -- [Terraform](https://docs.microsoft.com/azure/developer/terraform/create-k8s-cluster-with-tf-and-aks) +- [Terraform](https://learn.microsoft.com/azure/developer/terraform/create-k8s-cluster-with-tf-and-aks) - [dapr](https://github.com/dapr/dapr) Keep watching this space, as we build out reference implementation guidance on topics such as these. Further guidance delivered will use this baseline AKS implementation as their starting point. If you would like to contribute or suggest a pattern built on this baseline, [please get in touch](./CONTRIBUTING.md). @@ -140,9 +140,9 @@ Kubernetes is a very flexible platform, giving infrastructure and application op ## Related documentation -- [Azure Kubernetes Service Documentation](https://docs.microsoft.com/azure/aks/) -- [Microsoft Azure Well-Architected Framework](https://docs.microsoft.com/azure/architecture/framework/) -- [Microservices architecture on AKS](https://docs.microsoft.com/azure/architecture/reference-architectures/containers/aks-microservices/aks-microservices) +- [Azure Kubernetes Service Documentation](https://learn.microsoft.com/azure/aks/) +- [Microsoft Azure Well-Architected Framework](https://learn.microsoft.com/azure/architecture/framework/) +- [Microservices architecture on AKS](https://learn.microsoft.com/azure/architecture/reference-architectures/containers/aks-microservices/aks-microservices) ## Contributions diff --git a/acr-stamp.bicep b/acr-stamp.bicep index dac318b3..f57ada3a 100644 --- a/acr-stamp.bicep +++ b/acr-stamp.bicep @@ -49,7 +49,7 @@ param location string = 'eastus2' 'japaneast' 'southeastasia' ]) -@description('For Azure resources that support native geo-redunancy, provide the location the redundant service will have its secondary. Should be different than the location parameter and ideally should be a paired region - https://docs.microsoft.com/azure/best-practices-availability-paired-regions. This region does not need to support availability zones.') +@description('For Azure resources that support native geo-redunancy, provide the location the redundant service will have its secondary. Should be different than the location parameter and ideally should be a paired region - https://learn.microsoft.com/azure/best-practices-availability-paired-regions. This region does not need to support availability zones.') param geoRedundancyLocation string = 'centralus' /*** VARIABLES ***/ diff --git a/cluster-manifests/README.md b/cluster-manifests/README.md index 786eb8cb..6300ce2b 100644 --- a/cluster-manifests/README.md +++ b/cluster-manifests/README.md @@ -15,7 +15,7 @@ This is the root of the GitOps configuration directory. These Kubernetes object ### Kured -Kured is included as a solution to handle occasional required reboots from daily OS patching. This open-source software component is only needed if you require a managed rebooting solution between weekly [node image upgrades](https://docs.microsoft.com/azure/aks/node-image-upgrade). Building a process around deploying node image upgrades [every week](https://github.com/Azure/AKS/releases) satisfies most organizational weekly patching cadence requirements. Combined with most security patches on Linux not requiring reboots often, this leaves your cluster in a well supported state. If weekly node image upgrades satisfies your business requirements, then remove Kured from this solution by deleting [`kured.yaml`](./cluster-baseline-settings/kured.yaml). If however weekly patching using node image upgrades is not sufficient and you need to respond to daily security updates that mandate a reboot ASAP, then using a solution like Kured will help you achieve that objective. **Kured is not supported by Microsoft Support.** +Kured is included as a solution to handle occasional required reboots from daily OS patching. This open-source software component is only needed if you require a managed rebooting solution between weekly [node image upgrades](https://learn.microsoft.com/azure/aks/node-image-upgrade). Building a process around deploying node image upgrades [every week](https://github.com/Azure/AKS/releases) satisfies most organizational weekly patching cadence requirements. Combined with most security patches on Linux not requiring reboots often, this leaves your cluster in a well supported state. If weekly node image upgrades satisfies your business requirements, then remove Kured from this solution by deleting [`kured.yaml`](./cluster-baseline-settings/kured.yaml). If however weekly patching using node image upgrades is not sufficient and you need to respond to daily security updates that mandate a reboot ASAP, then using a solution like Kured will help you achieve that objective. **Kured is not supported by Microsoft Support.** ## Private bootstrapping repository diff --git a/cluster-manifests/kube-system/container-azm-ms-agentconfig.yaml b/cluster-manifests/kube-system/container-azm-ms-agentconfig.yaml index bc7a44a9..fd4a02f8 100644 --- a/cluster-manifests/kube-system/container-azm-ms-agentconfig.yaml +++ b/cluster-manifests/kube-system/container-azm-ms-agentconfig.yaml @@ -5,7 +5,7 @@ metadata: namespace: kube-system data: # https://raw.githubusercontent.com/microsoft/Docker-Provider/ci_prod/kubernetes/container-azm-ms-agentconfig.yaml - # https://docs.microsoft.com/azure/azure-monitor/containers/container-insights-agent-config + # https://learn.microsoft.com/azure/azure-monitor/containers/container-insights-agent-config schema-version: #string.used by agent to parse config. supported versions are {v1}. Configs with other schema versions will be rejected by the agent. v1 diff --git a/cluster-stamp.bicep b/cluster-stamp.bicep index 3a0acc99..354c5ab8 100644 --- a/cluster-stamp.bicep +++ b/cluster-stamp.bicep @@ -931,7 +931,7 @@ resource sqrPodFailed 'Microsoft.Insights/scheduledQueryRules@2018-04-16' = { description: 'Alert on pod Failed phase.' enabled: 'true' source: { - query: '//https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-alerts \r\n let endDateTime = now(); let startDateTime = ago(1h); let trendBinSize = 1m; let clusterName = "${clusterName}"; KubePodInventory | where TimeGenerated < endDateTime | where TimeGenerated >= startDateTime | where ClusterName == clusterName | distinct ClusterName, TimeGenerated | summarize ClusterSnapshotCount = count() by bin(TimeGenerated, trendBinSize), ClusterName | join hint.strategy=broadcast ( KubePodInventory | where TimeGenerated < endDateTime | where TimeGenerated >= startDateTime | distinct ClusterName, Computer, PodUid, TimeGenerated, PodStatus | summarize TotalCount = count(), PendingCount = sumif(1, PodStatus =~ "Pending"), RunningCount = sumif(1, PodStatus =~ "Running"), SucceededCount = sumif(1, PodStatus =~ "Succeeded"), FailedCount = sumif(1, PodStatus =~ "Failed") by ClusterName, bin(TimeGenerated, trendBinSize) ) on ClusterName, TimeGenerated | extend UnknownCount = TotalCount - PendingCount - RunningCount - SucceededCount - FailedCount | project TimeGenerated, TotalCount = todouble(TotalCount) / ClusterSnapshotCount, PendingCount = todouble(PendingCount) / ClusterSnapshotCount, RunningCount = todouble(RunningCount) / ClusterSnapshotCount, SucceededCount = todouble(SucceededCount) / ClusterSnapshotCount, FailedCount = todouble(FailedCount) / ClusterSnapshotCount, UnknownCount = todouble(UnknownCount) / ClusterSnapshotCount| summarize AggregatedValue = avg(FailedCount) by bin(TimeGenerated, trendBinSize)' + query: '//https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-alerts \r\n let endDateTime = now(); let startDateTime = ago(1h); let trendBinSize = 1m; let clusterName = "${clusterName}"; KubePodInventory | where TimeGenerated < endDateTime | where TimeGenerated >= startDateTime | where ClusterName == clusterName | distinct ClusterName, TimeGenerated | summarize ClusterSnapshotCount = count() by bin(TimeGenerated, trendBinSize), ClusterName | join hint.strategy=broadcast ( KubePodInventory | where TimeGenerated < endDateTime | where TimeGenerated >= startDateTime | distinct ClusterName, Computer, PodUid, TimeGenerated, PodStatus | summarize TotalCount = count(), PendingCount = sumif(1, PodStatus =~ "Pending"), RunningCount = sumif(1, PodStatus =~ "Running"), SucceededCount = sumif(1, PodStatus =~ "Succeeded"), FailedCount = sumif(1, PodStatus =~ "Failed") by ClusterName, bin(TimeGenerated, trendBinSize) ) on ClusterName, TimeGenerated | extend UnknownCount = TotalCount - PendingCount - RunningCount - SucceededCount - FailedCount | project TimeGenerated, TotalCount = todouble(TotalCount) / ClusterSnapshotCount, PendingCount = todouble(PendingCount) / ClusterSnapshotCount, RunningCount = todouble(RunningCount) / ClusterSnapshotCount, SucceededCount = todouble(SucceededCount) / ClusterSnapshotCount, FailedCount = todouble(FailedCount) / ClusterSnapshotCount, UnknownCount = todouble(UnknownCount) / ClusterSnapshotCount| summarize AggregatedValue = avg(FailedCount) by bin(TimeGenerated, trendBinSize)' dataSourceId: la.id queryType: 'ResultCount' } diff --git a/contoso-bicycle/README.md b/contoso-bicycle/README.md index 0fb7125d..28a6b19f 100644 --- a/contoso-bicycle/README.md +++ b/contoso-bicycle/README.md @@ -44,7 +44,7 @@ Responsible for the infrastructure deployment and day-to-day operations of the A ## Business requirements -Here are the requirements based on an initial [Well-Architected Framework review](https://docs.microsoft.com/assessments/?id=azure-architecture-review). +Here are the requirements based on an initial [Well-Architected Framework review](https://learn.microsoft.com/assessments/?id=azure-architecture-review). ### Reliability diff --git a/github-workflow/README.md b/github-workflow/README.md index ad5e55a3..6e950bf0 100644 --- a/github-workflow/README.md +++ b/github-workflow/README.md @@ -23,4 +23,4 @@ Review the yaml file to see the types of steps you'd need to perform. Also consi ## See also * [GitHub Actions](https://help.github.com/actions) -* [GitHub Actions with Azure Kubernetes Service](https://docs.microsoft.com/azure/aks/kubernetes-action) \ No newline at end of file +* [GitHub Actions with Azure Kubernetes Service](https://learn.microsoft.com/azure/aks/kubernetes-action) \ No newline at end of file diff --git a/github-workflow/aks-deploy.yaml b/github-workflow/aks-deploy.yaml index 0598a04c..bfbdfd92 100644 --- a/github-workflow/aks-deploy.yaml +++ b/github-workflow/aks-deploy.yaml @@ -38,7 +38,7 @@ env: RESOURCE_GROUP_LOCATION: '' # The location where the resource group is going to be created RESOURCE_GROUP: '' # The name for the AKS cluster resource group AKS_LOCATION: '' # The location where the AKS cluster is going to be deployed - GEO_REDUNDANCY_LOCATION: '' # The location for Azure resources that support native geo-redunancy. Should be different than the location parameter and ideally should be a paired region - https://docs.microsoft.com/en-us/azure/best-practices-availability-paired-regions. This region does not need to support availability zones. + GEO_REDUNDANCY_LOCATION: '' # The location for Azure resources that support native geo-redunancy. Should be different than the location parameter and ideally should be a paired region - https://learn.microsoft.com/en-us/azure/best-practices-availability-paired-regions. This region does not need to support availability zones. TARGET_VNET_RESOURCE_ID: '' # The regional network spoke VNet Resource ID that the cluster will be joined to K8S_RBAC_AAD_PROFILE_TENANTID: '' # The tenant to integrate AKS-managed Azure AD K8S_RBAC_AAD_PROFILE_ADMIN_GROUP_OBJECTID: '' # The Azure AD group object ID that has admin access to the AKS cluster diff --git a/networking/README.md b/networking/README.md index 47e237ff..2296d97e 100644 --- a/networking/README.md +++ b/networking/README.md @@ -18,4 +18,4 @@ See the [AKS Baseline Network Topology](./topology.md) for specifics on how this ## See also -* [Hub-spoke network topology in Azure](https://docs.microsoft.com/azure/architecture/reference-architectures/hybrid-networking/hub-spoke) +* [Hub-spoke network topology in Azure](https://learn.microsoft.com/azure/architecture/reference-architectures/hybrid-networking/hub-spoke) diff --git a/networking/hub-default.bicep b/networking/hub-default.bicep index f9a0840a..af49f0db 100644 --- a/networking/hub-default.bicep +++ b/networking/hub-default.bicep @@ -466,7 +466,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = { ] dependsOn: [ // This helps prevent multiple PUT updates happening to the firewall causing a CONFLICT race condition - // Ref: https://docs.microsoft.com/azure/firewall-manager/quick-firewall-policy + // Ref: https://learn.microsoft.com/azure/firewall-manager/quick-firewall-policy fwPolicy::defaultApplicationRuleCollectionGroup fwPolicy::defaultNetworkRuleCollectionGroup ] diff --git a/networking/hub-regionA.bicep b/networking/hub-regionA.bicep index cefeb5bc..184e384c 100644 --- a/networking/hub-regionA.bicep +++ b/networking/hub-regionA.bicep @@ -680,7 +680,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = { ] dependsOn: [ // This helps prevent multiple PUT updates happening to the firewall causing a CONFLICT race condition - // Ref: https://docs.microsoft.com/azure/firewall-manager/quick-firewall-policy + // Ref: https://learn.microsoft.com/azure/firewall-manager/quick-firewall-policy fwPolicy::defaultApplicationRuleCollectionGroup fwPolicy::defaultNetworkRuleCollectionGroup ipgNodepoolSubnet diff --git a/networking/spoke-BU0001A0008.bicep b/networking/spoke-BU0001A0008.bicep index 32c6fdf7..1448781c 100644 --- a/networking/spoke-BU0001A0008.bicep +++ b/networking/spoke-BU0001A0008.bicep @@ -144,7 +144,7 @@ resource nsgAppGwSubnet 'Microsoft.Network/networkSecurityGroups@2021-05-01' = { { name: 'AllowControlPlaneInbound' properties: { - description: 'Allow Azure Control Plane in. (https://docs.microsoft.com/azure/application-gateway/configuration-infrastructure#network-security-groups)' + description: 'Allow Azure Control Plane in. (https://learn.microsoft.com/azure/application-gateway/configuration-infrastructure#network-security-groups)' protocol: '*' sourcePortRange: '*' sourceAddressPrefix: 'GatewayManager' @@ -158,7 +158,7 @@ resource nsgAppGwSubnet 'Microsoft.Network/networkSecurityGroups@2021-05-01' = { { name: 'AllowHealthProbesInbound' properties: { - description: 'Allow Azure Health Probes in. (https://docs.microsoft.com/azure/application-gateway/configuration-infrastructure#network-security-groups)' + description: 'Allow Azure Health Probes in. (https://learn.microsoft.com/azure/application-gateway/configuration-infrastructure#network-security-groups)' protocol: '*' sourcePortRange: '*' sourceAddressPrefix: 'AzureLoadBalancer' diff --git a/networking/topology.md b/networking/topology.md index 0b15dec0..45c50c3e 100644 --- a/networking/topology.md +++ b/networking/topology.md @@ -46,27 +46,27 @@ In the future, this VNet might hold more subnets like [ACI Provider instance] su 2. [AKS Internal Load Balancer subnet]: Multi-tenant, multiple SSL termination rules, single PPE supporting dev/QA/UAT, etc could lead to needing more ingress controllers, but for baseline, we should start with one. 3. [Private Endpoints] subnet: Private Links are created for Azure Container Registry and Azure Key Vault, so these Azure services can be accessed using Private Endpoints within the spoke virtual network. There are multiple [Private Link deployment options]; in this implementation they are deployed to a dedicated subnet within the spoke virtual network. -[271]: https://docs.microsoft.com/azure/vpn-gateway/vpn-gateway-about-vpn-gateway-settings#gwsub -[251]: https://docs.microsoft.com/azure/application-gateway/configuration-overview#size-of-the-subnet -[59]: https://docs.microsoft.com/azure/firewall/firewall-faq#does-the-firewall-subnet-size-need-to-change-as-the-service-scales -[272]: https://docs.microsoft.com/azure/bastion/bastion-create-host-portal#createhost -[30]: https://docs.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools +[271]: https://learn.microsoft.com/azure/vpn-gateway/vpn-gateway-about-vpn-gateway-settings#gwsub +[251]: https://learn.microsoft.com/azure/application-gateway/configuration-overview#size-of-the-subnet +[59]: https://learn.microsoft.com/azure/firewall/firewall-faq#does-the-firewall-subnet-size-need-to-change-as-the-service-scales +[272]: https://learn.microsoft.com/azure/bastion/bastion-create-host-portal#createhost +[30]: https://learn.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools [% Max Surge]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#max-surge [% Max Unavailable]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#max-unavailable [Add Ips/Pods]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment -[Azure Subnet not assignable Ips factor]: https://docs.microsoft.com/azure/virtual-network/virtual-network-ip-addresses-overview-arm#allocation-method-1 -[Private Endpoints]: https://docs.microsoft.com/azure/private-link/private-endpoint-overview#private-endpoint-properties -[Minimum Subnet size]: https://docs.microsoft.com/azure/aks/configure-azure-cni#plan-ip-addressing-for-your-cluster -[Subnet Mask bits]: https://docs.microsoft.com/azure/virtual-network/virtual-networks-faq#how-small-and-how-large-can-vnets-and-subnets-be -[Azure Hub-Spoke topology]: https://docs.microsoft.com/azure/architecture/reference-architectures/hybrid-networking/hub-spoke -[Azure Firewall subnet]: https://docs.microsoft.com/azure/firewall/firewall-faq#does-the-firewall-subnet-size-need-to-change-as-the-service-scales -[Gateway subnet]: https://docs.microsoft.com/azure/vpn-gateway/vpn-gateway-about-vpn-gateway-settings#gwsub -[Azure Application Gateway subnet]: https://docs.microsoft.com/azure/application-gateway/configuration-infrastructure#virtual-network-and-dedicated-subnet -[Private Link Endpoint subnet]: https://docs.microsoft.com/azure/architecture/guide/networking/private-link-hub-spoke-network#networking -[Private Link deployment options]: https://docs.microsoft.com/azure/architecture/guide/networking/private-link-hub-spoke-network#decision-tree-for-private-link-deployment -[Azure Bastion subnet]: https://docs.microsoft.com/azure/bastion/bastion-create-host-portal#createhost -[AKS System Nodepool]: https://docs.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools -[AKS User Nodepool]: https://docs.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools -[AKS Internal Load Balancer subnet]: https://docs.microsoft.com/azure/aks/internal-lb#specify-a-different-subnet -[ACI Provider Instance]: https://docs.microsoft.com/azure/container-instances/container-instances-vnet -[AKS Nodepools subnets]: https://docs.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools +[Azure Subnet not assignable Ips factor]: https://learn.microsoft.com/azure/virtual-network/virtual-network-ip-addresses-overview-arm#allocation-method-1 +[Private Endpoints]: https://learn.microsoft.com/azure/private-link/private-endpoint-overview#private-endpoint-properties +[Minimum Subnet size]: https://learn.microsoft.com/azure/aks/configure-azure-cni#plan-ip-addressing-for-your-cluster +[Subnet Mask bits]: https://learn.microsoft.com/azure/virtual-network/virtual-networks-faq#how-small-and-how-large-can-vnets-and-subnets-be +[Azure Hub-Spoke topology]: https://learn.microsoft.com/azure/architecture/reference-architectures/hybrid-networking/hub-spoke +[Azure Firewall subnet]: https://learn.microsoft.com/azure/firewall/firewall-faq#does-the-firewall-subnet-size-need-to-change-as-the-service-scales +[Gateway subnet]: https://learn.microsoft.com/azure/vpn-gateway/vpn-gateway-about-vpn-gateway-settings#gwsub +[Azure Application Gateway subnet]: https://learn.microsoft.com/azure/application-gateway/configuration-infrastructure#virtual-network-and-dedicated-subnet +[Private Link Endpoint subnet]: https://learn.microsoft.com/azure/architecture/guide/networking/private-link-hub-spoke-network#networking +[Private Link deployment options]: https://learn.microsoft.com/azure/architecture/guide/networking/private-link-hub-spoke-network#decision-tree-for-private-link-deployment +[Azure Bastion subnet]: https://learn.microsoft.com/azure/bastion/bastion-create-host-portal#createhost +[AKS System Nodepool]: https://learn.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools +[AKS User Nodepool]: https://learn.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools +[AKS Internal Load Balancer subnet]: https://learn.microsoft.com/azure/aks/internal-lb#specify-a-different-subnet +[ACI Provider Instance]: https://learn.microsoft.com/azure/container-instances/container-instances-vnet +[AKS Nodepools subnets]: https://learn.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools diff --git a/workload/aspnetapp.yaml b/workload/aspnetapp.yaml index ac413627..ad25a40f 100644 --- a/workload/aspnetapp.yaml +++ b/workload/aspnetapp.yaml @@ -99,7 +99,7 @@ metadata: namespace: a0008 annotations: kubernetes.io/ingress.allow-http: "false" - # defines controller implementing this ingress resource: https://docs.microsoft.com/en-us/azure/dev-spaces/how-to/ingress-https-traefik + # defines controller implementing this ingress resource: https://learn.microsoft.com/en-us/azure/dev-spaces/how-to/ingress-https-traefik # ingress.class annotation is being deprecated in Kubernetes 1.18: https://kubernetes.io/docs/concepts/services-networking/ingress/#deprecated-annotation # For backwards compatibility, when this annotation is set, precedence is given over the new field ingressClassName under spec. kubernetes.io/ingress.class: traefik-internal