From bf6ab96a8e0fd8edc59efcda04b0233031239b66 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Fri, 16 Feb 2024 18:41:14 -0500 Subject: [PATCH 01/15] Update container-service-general-considerations.md --- ...ontainer-service-general-considerations.md | 48 ++++++++++--------- 1 file changed, 25 insertions(+), 23 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 9a2a9f38b8..85fc67f3cb 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -96,10 +96,10 @@ Most containerized applications run in Linux containers, which are supported by ## Networking considerations -It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on whether you want to prioritize simplicity or configurability: +It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on preference: -- [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability might be able to take advantage of workload/dedicated profiles rather than resorting to AKS to meet their networking requirements. -- [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. These implementations, however, are customer managed, so they increase operational overhead. +- [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability might be able to take advantage of workload/dedicated profiles before considering AKS to maximize their networking options. +- [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Customers can install and operate any add-on or software from the Kubernetes ecosystem. - [Web App for Containers](https://azure.microsoft.com/products/app-service/containers/?activetab=pivot:deploytab) is feature of App Service, so the networking concepts, especially private networking integration, are very specific to App Service. This service will be familiar to workload teams that already use App Service. Teams that don't have experience with App Service and that want a more familiar Azure virtual network integration are encouraged to consider Container Apps. Keep in mind that networking is a foundational infrastructure layer. It's often difficult to make changes in design without re-deploying the workload, which can lead to downtime. Therefore, if your workload has specific networking requirements, review this section carefully before you narrow down your Azure container service selection. @@ -123,7 +123,7 @@ The following sections provide information about various networking constraints. - Multiple colocated workloads. - Private and/or public ingress. -- An access-controlled flow of east-west traffic in a cluster (for Container Apps and AKS) or within a virtual network (for all Azure container services). +- An access-controlled flow of east-west traffic in a cluster (for Container Apps and AKS) or within a virtual network (for all Azure container services). ### Subnet planning @@ -135,11 +135,11 @@ Ensuring that you have a subnet that's large enough to include instances of your *This describes a best practice, not a technical limitation. -For Container Apps, subnet integration applies only to a single Container Apps environment. Each Container Apps environment is constrained to a single ingress IP, public or private. +For Container Apps, subnet integration applies only to a single Container Apps environment. Each Container Apps environment is constrained to a single ingress IP, public or private. Each Container Apps environment is meant only for a single workload in which dependent applications are colocated. Therefore, you need to introduce additional Azure networking appliances for ingress load balancing if you need both public and private ingress. Examples include Azure Application Gateway and Azure Front Door. Also, if you have multiple workloads that need to be segregated, additional Container Apps environments are required, so an additional subnet must be allocated for each environment. -AKS provides granular east-west network flow control within the cluster in the form of Kubernetes network policy. This flow control enables you to host multiple workloads within the same subnet, which you can't do in Container Apps. As a consequence, AKS has a steeper adoption curve but more configurability options. +AKS provides granular east-west network flow control within the cluster in the form of Kubernetes network policy. This flow control enables you to segregate multiple workloads with different security boundaries within the same cluster. For Web App for Containers, there are no constraints on how many apps you can integrate with a single subnet, as long as the subnet is large enough. There are no best practices for access control between web apps in the same virtual network. Each web app independently manages access control for east-west or north-south traffic from the virtual network or internet, respectively. @@ -203,9 +203,9 @@ Both Container Apps and Web App for Containers support TLS 1.2 for their built-i ### Load balancing -With Container Apps and Web App for Containers, Azure manages the Layer 4 and Layer 7 load balancers. +With Container Apps and Web App for Containers, Azure fully abstracts away the Layer 4 and Layer 7 load balancers. -AKS, however, uses a shared responsibility model in which Azure manages the underlying Azure infrastructure that the workload team configures by using the Kubernetes API. For Layer 7 load balancing in AKS, you can choose an Azure-managed option, like [Application Gateway for Containers](/azure/application-gateway/for-containers/overview), or deploy and self-manage an ingress controller of your choice. +AKS uses a shared responsibility model in which Azure manages the underlying Azure infrastructure that the workload team configures by interfacing with the Kubernetes API. For Layer 7 load balancing in AKS, you can choose an Azure-managed options, for example the [AKS managed application routing add-on](/azure/aks/app-routing) or the [Application Gateway for Containers](/azure/application-gateway/for-containers/overview), or deploy and self-manage an ingress controller of your choice. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| @@ -224,7 +224,7 @@ Web Apps for Containers provides public ingress (north-south communication) FQDN Container Apps also provides public ingress FQDNs. However, Container Apps goes further by allowing the app FQDN to be exposed and [restricting traffic only within the environment](/azure/container-apps/networking). This functionality makes it easier to manage east-west communication and enable components like Dapr. -AKS requires the deployment of Kubernetes services as defined by the Kubernetes API. These services are used to expose applications to the network in an addressable way. Because Kubernetes deployments aren't initially discoverable from within or outside of the cluster, additional configuration is required. +Kubernetes deployments are not initially discoverable within or from outside the cluster. You must create Kubernetes services as defined by the Kubernetes API, which then expose applications to the network in an addressable way. > [!important] > Only Container Apps and AKS provide service discovery through internal DNS schemes within their respective environments. This functionality can simplify DNS configurations across dev/test and production environments. For example, you can create these environments with arbitrary service names that have to be unique only within the environment or cluster, so they can be the same across dev/test and production. With Web App for Containers, service names must be unique across different environments to avoid conflicts with Azure DNS. @@ -239,7 +239,7 @@ Both Container Apps and Web App for Containers provide out-of-the-box solutions | **Managed TLS for Azure FQDNs** | Out of the box | N/A | Out of the box | | **Managed TLS for custom domains** | [In preview](/azure/container-apps/custom-domains-managed-certificates) | BYO | Out of the box or BYO | -AKS requires workload teams to set up their own ingress controllers, configure custom domain names on the cluster and DNS, and manage their own certificates. Teams considering AKS can use the CNCF [cert-manager]( https://www.cncf.io/projects/cert-manager/) to manage TLS certificates, and nginx or Traefik for ingress. +AKS requires workload teams to configure their own ingress controllers, configure custom domain names on the cluster and DNS, and manage their own certificates. Although AKS can use the CNCF [cert-manager]( https://www.cncf.io/projects/cert-manager/) to manage TLS certificates, and nginx or Traefik for ingress. ### Mutual TLS @@ -249,7 +249,7 @@ Web App for Containers has built-in mTLS support for incoming client connections Container Apps also has built-in support for mTLS. It forwards the client certificate to the application in the HTTP header [X-Forwarded-Client-Cert](/azure/container-apps/client-certificate-authorization). You can also easily enable [automatic mTLS for internal communication between apps](/azure/container-apps/networking#mtls) in a single environment. -AKS doesn't provide out-of-the-box mTLS. Workload teams should consider installing a service mesh to implement mTLS. +AKS takes a cloud native approach to implementing mTLS through the [Istio-based service mesh as a managed add-on](/azure/aks/istio-about), which includes mTLS capabilities for incoming client connections and intra cluster communication between services. ### Service-specific networking concepts @@ -269,7 +269,7 @@ Failure to address security risks can lead to unauthorized access, data breaches Most Azure services, including Container Apps, AKS, and Web App for Containers, integrate with key security offerings, including Key Vault and managed identities. -AKS provides more configurability, at the expense of the additional overhead of running a Kubernetes cluster. Although AKS provides unique features like network policies, part of the reason it offers more security features is because its attack surface is larger. For example, AKS surfaces Kubernetes components, including the control plane and virtual machine nodes, which in turn require additional security protection. +AKS provides more configurability, at the expense of the additional overhead of running a Kubernetes cluster. Although AKS provides unique features like network policies, part of the reason it offers more security features is because its attack surface is larger. For example, AKS surfaces the Kubernetes control plane and virtual machine nodes, which in turn require additional security protection. For a detailed comparison, carefully review the following considerations to ensure that your workload security requirements can be met. @@ -296,7 +296,7 @@ Customers who use Container Apps don't have access to the Kubernetes API. Micros If you want to restrict network access to the Kubernetes control plane, you need to use AKS, which provides two options. The first option is to use [private AKS clusters](/azure/aks/private-clusters#create-a-private-aks-cluster), which use Azure Private Link between the API server's private network and the AKS cluster's private network. The second option is [API Server VNet integration (preview)](/azure/aks/api-server-vnet-integration), where the API server is integrated into a delegated subnet. See the documentation to learn more. -There are consequences to implementing network-restricted access to the Kubernetes API. Most notably, management can be performed only from within the private network, which means you need to deploy self-hosted agents for Azure DevOps or GitHub Actions. To learn about other limitations, see the feature-specific documentation. +There are consequences to implementing network-restricted access to the Kubernetes API. Most notably, management can be performed only from within the private network. Typically this means you need to deploy self-hosted agents for Azure DevOps or GitHub Actions. To learn about other limitations, see the product-specific documentation. | | Container Apps | AKS| |---|--|--| @@ -328,11 +328,11 @@ In all scenarios, you can regulate networking communication within the wider vir #### Built-in IP restrictions for ingress -Container Apps and Web App for Containers don't require virtual network integration, so they provide built-in source IP restrictions for ingress traffic. If you use AKS, you need to implement a bring-your-own solution to achieve similar traffic filtering. +Container Apps and Web App for Containers don't require virtual network integration, so they provide built-in source IP restrictions for ingress traffic. If you use AKS, you can leverage the [Kubernetes Service api-resource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core) to achieve traffic filtering. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| -| **Built-in ingress IP restrictions** | ✅ | ❌ | ✅ | +| **Built-in app ingress IP restrictions** | ✅ | ❌ | ✅ | ## Application-level security @@ -358,7 +358,7 @@ For more information, see: It's a best practice to use managed identities to access Azure resources without using secrets. For example, you can pull images from Azure Container Registry without using a user name and password. -Azure container services provide managed identity support, configurable out of the box for Container Apps and Web App for Containers. AKS provides integrated managed identity support for the Kubernetes control plane, Container Registry image management, and cluster add-ons. Managed identity for AKS applications is provided through [Workload ID](/azure/aks/workload-identity-overview). Workload ID is more complex than the implementations for providing managed identities to application code in the other Azure services described in this article. +Azure container services provide managed identity support for the purpose of authenticating to Azure resources. Managed Identity is configurable out of the box for Container Apps and Web App for Containers. AKS provides integrated managed identity support for the Kubernetes control plane, Container Registry image management, and cluster add-ons. Managed identity for AKS applications authenticating to Azure resources is provided through the [Workload ID](/azure/aks/workload-identity-overview) feature. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| @@ -374,7 +374,7 @@ For more information, see: ### Threat protection and vulnerability assessments with Defender for Containers -Threat protection against vulnerabilities is also important. It's a best practice to use [Defender for Containers](/azure/defender-for-cloud/defender-for-containers-introduction). Currently, runtime support is available only for AKS. Vulnerability assessments are supported in Azure container registries, so they can be used by any Azure container service, not just the ones described in this article. +Threat protection against vulnerabilities is also important. It's a best practice to use [Defender for Containers](/azure/defender-for-cloud/defender-for-containers-introduction). Vulnerability assessments are supported in Azure container registries, so they can be used by any Azure container service, not just the ones described in this article. Defender for Containers runtime protection, however, is available only for AKS. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| @@ -386,7 +386,7 @@ Note that container image vulnerability assessments aren't real-time scans. The ## Security baselines -In general, most Azure container services integrate Azure security offerings. AKS is more configurable, in part because its attack surface is larger. Overall, keep in mind that a security feature set is just a small part of implementing cloud security. For more information about implementing security for container services, see the following service-specific security baselines: +In general, most Azure container services integrate Azure security offerings. AKS has more extensive guidelines, in part because its attack surface is larger. Overall, keep in mind that a security feature set is just a small part of implementing cloud security. For more information about implementing security for container services, see the following service-specific security baselines: - [Azure security baseline for Container Apps](/security/benchmark/azure/baselines/azure-container-apps-security-baseline) - [Azure security baseline for Azure Kubernetes Service](/security/benchmark/azure/baselines/azure-kubernetes-service-aks-security-baseline) @@ -404,13 +404,13 @@ For more complete security guidance about AKS, see [Well-Architected Framework r To successfully run a workload in production, teams need to implement operational excellence practices, including centralized logging, monitoring, scalability, regular updates and patching, and image management. -In general, AKS provides the most configurability and therefore the most diversity and complexity for operational tasks. Azure, however, assumes many operational responsibilities, such as updates, for PaaS offerings like Container Apps and Web App for Containers. +In general, AKS provides the most configurability and therefore the most diversity and complexity for operational tasks. Azure assumes many operational responsibilities, such as updates, for offerings like Container Apps and Web App for Containers. ### Updates and patches You need to ensure that your application's underlying OS is updated and regularly patched. Keep in mind, however, that with every update there's a risk of failure. This section and the next one describe the main considerations for the three container services with regard to the shared responsibility between the customer and the platform. -AKS is a hybrid IaaS and PaaS solution, so the workload team, not Microsoft, is responsible for upgrading clusters' control and application planes. Azure provides the updated images for the node OS and control plane components, but customers are responsible for triggering or scheduling the updates. See the AKS day-2 operations guide to learn about [patching and upgrading AKS clusters](/azure/architecture/operator-guides/aks/aks-upgrade-practices). +As a managed Kubernetes service, AKS will provide the updated images for the node OS and control plane components. But customers are responsible for applying updates. You can manually trigger updates or leverage the [cluster auto-upgrade channels](/azure/aks/auto-upgrade-cluster) feature to ensure clusters are up to date. See the AKS day-2 operations guide to learn about [patching and upgrading AKS clusters](/azure/architecture/operator-guides/aks/aks-upgrade-practices). Container Apps and Web App for Containers are PaaS solutions. Azure is responsible for managing updates and patches, so customers can avoid the complexity of AKS upgrade management. @@ -503,7 +503,7 @@ For a detailed description of each of the resource logs in the table, select the Here's a short summary of the logging capabilities of the container services: -- **Container Apps** abstracts all of its internal Kubernetes logs into two simple categories. One, called *Console* logs, contains the workload container logs. A second *System* category contains all platform-related logs. +- **Container Apps** abstracts all of its internal Kubernetes logs into two categories. One, called *Console* logs, contains the workload container logs. A second *System* category contains all platform-related logs. - **AKS** provides all Kubernetes-related logs and granular control over what can be logged. It also retains full compatibility with Kubernetes client tools for log streaming, like kubectl. - **Web App for Containers** provides many categories of resource logs because its platform (App Service) isn't exclusively for container workloads. For container-specific operations that manage its internal Docker platform, it provides the AppServicePlatformLogs log category. Another important category is AppServiceEnvironmentPlatformLogs, which logs events like scaling and configuration changes. @@ -583,7 +583,7 @@ Health checks are easiest to [implement in Web App for Containers](/azure/app-se - It sends the health check at one-minute intervals. You can't change the interval. - The minimum threshold that you can set for an unhealthy instance to be removed from the internal load balancing mechanism is two minutes. The unhealthy instance gets traffic for at least two minutes after it fails a health check. The default value for this setting is 10 minutes. -Container Apps and AKS, on the other hand, are much more flexible and offer similar options (because Container Apps is based on an internal Kubernetes platform). In terms of specific differences, AKS provides the following options for performing health checks, which Container Apps lacks: +Container Apps and AKS, on the other hand, are much more flexible and offer similar options. In terms of some specific differences, AKS provides the following non-exhaustive options for performing health checks, which are not available in Container Apps: - [gRPC support](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) - [Named ports](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#use-a-named-port) @@ -632,7 +632,9 @@ This article focuses on the main differences among the container services featur Well-architected solutions set the foundations for successful workloads. Although architectures can be adjusted as a workload grows and teams progress on their cloud journeys, some decisions, especially around networking, are difficult to reverse without significant downtime or re-deployment. -In general, when you compare Azure container services, a theme emerges: AKS provides the most configurability, at the cost of additional operational overhead. Workload teams that, for example, have less operations experience or prefer to focus on application features, might prefer a PaaS service. We recommend that they consider Container Apps. +In general, when you compare Azure container services, a theme emerges: AKS surfaces the most underlying infrastructure, thus offering the greatest configurability and extensibility. The amount of operational overhead and complexity is highly variable for AKS workloads. Some teams can greatly reduce the operational overhead by using Microsoft managed add-ons and extensions, as well as auto-upgrade features. Other customers may prefer full control of the cluster in order to leverage full extensibility of Kubernetes and the CNCF ecosystem. For example, although Microsoft offers Flux as a managed GitOps extension, many teams choose instead to setup and operate ArgoCD on their own. + +Workload teams, for example, may have less operations experience, prefer to focus on application features, or they may not have a need for open-source software and fully portable systems. In this case, workload teams might prefer a PaaS offering. We recommend that they first consider Container Apps. Although Container Apps and Web App for Containers are both PaaS offerings that provide similar levels of Microsoft-managed infrastructure, a key difference is that Container Apps is closer to Kubernetes and provides additional cloud-native capabilities for service discovery, event-driven autoscaling, [Dapr](https://dapr.io/) integration, and more. However, teams that don't need these capabilities and are familiar with App Service networking and deployment models might prefer Web App for Containers. From e7a3b65ab9f4960fb88c648a2018f478499d0ff5 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:46:55 -0500 Subject: [PATCH 02/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 85fc67f3cb..6bbf234c57 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -96,7 +96,7 @@ Most containerized applications run in Linux containers, which are supported by ## Networking considerations -It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on preference: +It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on preference, such as extensibility or operational paradigms. - [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability might be able to take advantage of workload/dedicated profiles before considering AKS to maximize their networking options. - [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Customers can install and operate any add-on or software from the Kubernetes ecosystem. From 294e9172891c19da46d25ec571cf037689fa79c6 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:47:16 -0500 Subject: [PATCH 03/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 6bbf234c57..225f0ed820 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -98,7 +98,7 @@ Most containerized applications run in Linux containers, which are supported by It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on preference, such as extensibility or operational paradigms. -- [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability might be able to take advantage of workload/dedicated profiles before considering AKS to maximize their networking options. +- [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability can use workload/dedicated profiles before considering alternatives to maximize their networking options. - [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Customers can install and operate any add-on or software from the Kubernetes ecosystem. - [Web App for Containers](https://azure.microsoft.com/products/app-service/containers/?activetab=pivot:deploytab) is feature of App Service, so the networking concepts, especially private networking integration, are very specific to App Service. This service will be familiar to workload teams that already use App Service. Teams that don't have experience with App Service and that want a more familiar Azure virtual network integration are encouraged to consider Container Apps. From 31d972b85d92c36b358fd845a4de758919ca6246 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:48:06 -0500 Subject: [PATCH 04/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 225f0ed820..2eca97e116 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -99,7 +99,7 @@ Most containerized applications run in Linux containers, which are supported by It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on preference, such as extensibility or operational paradigms. - [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability can use workload/dedicated profiles before considering alternatives to maximize their networking options. -- [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Customers can install and operate any add-on or software from the Kubernetes ecosystem. +- [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Workload teams can take advantage of network add-ons provided by Microsoft or can opt to install, manage, and operate network capabilities from the broad Kubernetes ecosystem. - [Web App for Containers](https://azure.microsoft.com/products/app-service/containers/?activetab=pivot:deploytab) is feature of App Service, so the networking concepts, especially private networking integration, are very specific to App Service. This service will be familiar to workload teams that already use App Service. Teams that don't have experience with App Service and that want a more familiar Azure virtual network integration are encouraged to consider Container Apps. Keep in mind that networking is a foundational infrastructure layer. It's often difficult to make changes in design without re-deploying the workload, which can lead to downtime. Therefore, if your workload has specific networking requirements, review this section carefully before you narrow down your Azure container service selection. From 5df8c8100e0d763f024a9d1a7a533dd3df64d81c Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:48:15 -0500 Subject: [PATCH 05/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 2eca97e116..dd5a19395c 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -139,7 +139,7 @@ For Container Apps, subnet integration applies only to a single Container Apps e Each Container Apps environment is meant only for a single workload in which dependent applications are colocated. Therefore, you need to introduce additional Azure networking appliances for ingress load balancing if you need both public and private ingress. Examples include Azure Application Gateway and Azure Front Door. Also, if you have multiple workloads that need to be segregated, additional Container Apps environments are required, so an additional subnet must be allocated for each environment. -AKS provides granular east-west network flow control within the cluster in the form of Kubernetes network policy. This flow control enables you to segregate multiple workloads with different security boundaries within the same cluster. +AKS provides granular east-west network flow control within the cluster in the form of Kubernetes network policy. This flow control enables you to segment multiple workloads with different network security boundaries within the same cluster. For Web App for Containers, there are no constraints on how many apps you can integrate with a single subnet, as long as the subnet is large enough. There are no best practices for access control between web apps in the same virtual network. Each web app independently manages access control for east-west or north-south traffic from the virtual network or internet, respectively. From 11a150bbf6777bc71fa74284e0f1c79b6ddf742c Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:48:26 -0500 Subject: [PATCH 06/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index dd5a19395c..321a685958 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -239,7 +239,7 @@ Both Container Apps and Web App for Containers provide out-of-the-box solutions | **Managed TLS for Azure FQDNs** | Out of the box | N/A | Out of the box | | **Managed TLS for custom domains** | [In preview](/azure/container-apps/custom-domains-managed-certificates) | BYO | Out of the box or BYO | -AKS requires workload teams to configure their own ingress controllers, configure custom domain names on the cluster and DNS, and manage their own certificates. Although AKS can use the CNCF [cert-manager]( https://www.cncf.io/projects/cert-manager/) to manage TLS certificates, and nginx or Traefik for ingress. +AKS requires workload teams to configure their own ingress controllers, configure custom domain names on the cluster and DNS, and manage their own certificates. Although AKS can use the CNCF [cert-manager]( https://www.cncf.io/projects/cert-manager/) to manage TLS certificates, and Nginx or Traefik for ingress. ### Mutual TLS From b51b7d68b003b3acf28ff2be573db1ae3d653a82 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:49:07 -0500 Subject: [PATCH 07/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 321a685958..a841e1b2bd 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -249,7 +249,7 @@ Web App for Containers has built-in mTLS support for incoming client connections Container Apps also has built-in support for mTLS. It forwards the client certificate to the application in the HTTP header [X-Forwarded-Client-Cert](/azure/container-apps/client-certificate-authorization). You can also easily enable [automatic mTLS for internal communication between apps](/azure/container-apps/networking#mtls) in a single environment. -AKS takes a cloud native approach to implementing mTLS through the [Istio-based service mesh as a managed add-on](/azure/aks/istio-about), which includes mTLS capabilities for incoming client connections and intra cluster communication between services. +mTLS in AKS can be implemented through the [Istio-based service mesh as a managed add-on](/azure/aks/istio-about), which includes mTLS capabilities for incoming client connections and intra cluster communication between services. Workload teams could also choose to install and manage another service mesh offering from the Kubernetes ecosystem. These options make mTLS implementation in Kubernetes the most flexible. ### Service-specific networking concepts From b8141b454209104c12bfc9489969caabf2d02171 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:49:54 -0500 Subject: [PATCH 08/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index a841e1b2bd..678104e8a5 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -358,7 +358,7 @@ For more information, see: It's a best practice to use managed identities to access Azure resources without using secrets. For example, you can pull images from Azure Container Registry without using a user name and password. -Azure container services provide managed identity support for the purpose of authenticating to Azure resources. Managed Identity is configurable out of the box for Container Apps and Web App for Containers. AKS provides integrated managed identity support for the Kubernetes control plane, Container Registry image management, and cluster add-ons. Managed identity for AKS applications authenticating to Azure resources is provided through the [Workload ID](/azure/aks/workload-identity-overview) feature. +Azure container services provide managed identity support for the purpose of authenticating to Azure resources. For Container Apps and Web App for Containers, managed identity can be used for both control plane access to Azure resources, such as Azure Container Registry, and workload code access to Microsoft identity platform protected resources. AKS provides integrated managed identity support for the Kubernetes control plane to perform Azure resource control (such as managing the associated virtual machine scale sets), accessing container registry images, and cluster add-ons. Managed identity for workload code is configured by workload teams through the [Microsoft Entra Workload ID](/azure/aks/workload-identity-overview) feature. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| From a0120a719ffb8f4a30fa71ffa42ab640bdd72f50 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:50:31 -0500 Subject: [PATCH 09/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 678104e8a5..9808e26924 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -374,7 +374,9 @@ For more information, see: ### Threat protection and vulnerability assessments with Defender for Containers -Threat protection against vulnerabilities is also important. It's a best practice to use [Defender for Containers](/azure/defender-for-cloud/defender-for-containers-introduction). Vulnerability assessments are supported in Azure container registries, so they can be used by any Azure container service, not just the ones described in this article. Defender for Containers runtime protection, however, is available only for AKS. +Threat protection against vulnerabilities is also important. It's a best practice to use [Defender for Containers](/azure/defender-for-cloud/defender-for-containers-introduction). Vulnerability assessments are supported in Azure container registries, so they can be used by any Azure container service, not just the ones described in this article. Defender for Containers runtime protection, however, is available only for AKS. + +As AKS exposes the native Kubernetes API, it also supports security tooling from the Kubernetes ecosystem. Container Apps and Web App for Containers only supports built-in features and Microsoft Defender offerings. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| From 339a9c5b38f18ee4c2f09d2f0c7d9193770be1b9 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 10:51:02 -0500 Subject: [PATCH 10/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 9808e26924..c204baa32f 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -412,7 +412,7 @@ In general, AKS provides the most configurability and therefore the most diversi You need to ensure that your application's underlying OS is updated and regularly patched. Keep in mind, however, that with every update there's a risk of failure. This section and the next one describe the main considerations for the three container services with regard to the shared responsibility between the customer and the platform. -As a managed Kubernetes service, AKS will provide the updated images for the node OS and control plane components. But customers are responsible for applying updates. You can manually trigger updates or leverage the [cluster auto-upgrade channels](/azure/aks/auto-upgrade-cluster) feature to ensure clusters are up to date. See the AKS day-2 operations guide to learn about [patching and upgrading AKS clusters](/azure/architecture/operator-guides/aks/aks-upgrade-practices). +As a managed Kubernetes service, AKS will provide the updated images for the node OS and control plane components. But cluster operators are responsible for applying updates to their clusters. You can manually trigger updates or use the [cluster auto-upgrade channels](/azure/aks/auto-upgrade-cluster) feature to ensure clusters are up to date. See the AKS day-2 operations guide to learn about [patching and upgrading AKS clusters](/azure/architecture/operator-guides/aks/aks-upgrade-practices). Container Apps and Web App for Containers are PaaS solutions. Azure is responsible for managing updates and patches, so customers can avoid the complexity of AKS upgrade management. From 75968d0e0f6d77d481e4b5360d36a9d35e74f0f9 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 11:04:33 -0500 Subject: [PATCH 11/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index c204baa32f..f1e9bddbba 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -634,7 +634,7 @@ This article focuses on the main differences among the container services featur Well-architected solutions set the foundations for successful workloads. Although architectures can be adjusted as a workload grows and teams progress on their cloud journeys, some decisions, especially around networking, are difficult to reverse without significant downtime or re-deployment. -In general, when you compare Azure container services, a theme emerges: AKS surfaces the most underlying infrastructure, thus offering the greatest configurability and extensibility. The amount of operational overhead and complexity is highly variable for AKS workloads. Some teams can greatly reduce the operational overhead by using Microsoft managed add-ons and extensions, as well as auto-upgrade features. Other customers may prefer full control of the cluster in order to leverage full extensibility of Kubernetes and the CNCF ecosystem. For example, although Microsoft offers Flux as a managed GitOps extension, many teams choose instead to setup and operate ArgoCD on their own. +In general, when you compare Azure container services, a theme emerges: AKS surfaces the most underlying infrastructure, thus offering the greatest configurability and extensibility. The amount of operational overhead and complexity is highly variable for AKS workloads. Some teams can greatly reduce the operational overhead by using Microsoft managed add-ons and extensions, as well as auto-upgrade features. Other customers may prefer full control of the cluster in order to leverage full extensibility of Kubernetes and the CNCF ecosystem. For example, although Microsoft offers Flux as a managed GitOps extension, many teams choose instead to setup and operate ArgoCD on their own. Workload teams, for example, may have less operations experience, prefer to focus on application features, or they may not have a need for open-source software and fully portable systems. In this case, workload teams might prefer a PaaS offering. We recommend that they first consider Container Apps. From 63bfe4def129a05ff9b2fbaf212f4af40a4ae193 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 11:05:11 -0500 Subject: [PATCH 12/15] Update docs/guide/container-service-general-considerations.md Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index f1e9bddbba..dd79112640 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -636,7 +636,7 @@ Well-architected solutions set the foundations for successful workloads. Althoug In general, when you compare Azure container services, a theme emerges: AKS surfaces the most underlying infrastructure, thus offering the greatest configurability and extensibility. The amount of operational overhead and complexity is highly variable for AKS workloads. Some teams can greatly reduce the operational overhead by using Microsoft managed add-ons and extensions, as well as auto-upgrade features. Other customers may prefer full control of the cluster in order to leverage full extensibility of Kubernetes and the CNCF ecosystem. For example, although Microsoft offers Flux as a managed GitOps extension, many teams choose instead to setup and operate ArgoCD on their own. -Workload teams, for example, may have less operations experience, prefer to focus on application features, or they may not have a need for open-source software and fully portable systems. In this case, workload teams might prefer a PaaS offering. We recommend that they first consider Container Apps. +Workload teams, for example, that have less operations experience, prefer to focus on application features, or do not have a need for open-source software and fully portable systems might benefit from starting with a pure PaaS offering. We recommend that they first consider Container Apps. Although Container Apps and Web App for Containers are both PaaS offerings that provide similar levels of Microsoft-managed infrastructure, a key difference is that Container Apps is closer to Kubernetes and provides additional cloud-native capabilities for service discovery, event-driven autoscaling, [Dapr](https://dapr.io/) integration, and more. However, teams that don't need these capabilities and are familiar with App Service networking and deployment models might prefer Web App for Containers. From 91243a22b34498a4a753e591d4c1791042598cf5 Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 11:10:20 -0500 Subject: [PATCH 13/15] Update container-service-general-considerations.md Correcting technical issue with UDR/NAT compatibility for ACA. Ref: https://github.com/MicrosoftDocs/architecture-center/issues/4371 --- docs/guide/container-service-general-considerations.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index dd79112640..f375ef6dd9 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -160,14 +160,14 @@ It's important to note that web apps that are integrated into an App Service Env ### User-defined routes and NAT gateway support -If a workload requires user-defined routes (UDRs) and NAT gateway capabilities for granular networking control, Container Apps requires isolated infrastructure via [workload profiles](/azure/container-apps/workload-profiles-overview), which are available only in the Container Apps dedicated / workload profiles plan. +If a workload requires user-defined routes (UDRs) and NAT gateway capabilities for granular networking control, Container Apps requires the use of [workload profiles](/azure/container-apps/workload-profiles-overview). UDR and NAT gateway compatibility is not available in the consumption-only plan for ACA. AKS and Web App for Containers implement these two networking features through standard virtual network functionality or virtual network integration, respectively. To elaborate, AKS node pools and Web App for Containers in an App Service Environment are already direct virtual network resources. Web App for Containers that aren't in an App Service Environment support UDRs and NAT gateway via [virtual network integration](/azure/app-service/overview-vnet-integration). With virtual network integration, the resource technically doesn't reside directly in the virtual network, but all of its outbound access flows through the virtual network, and the network's associated rules affect traffic as expected. | | Container Apps| AKS| Web App for Containers| |---|---|--|--| -| **UDR support** | Consumption plan: ❌
Dedicated plan: ✅ | ✅ | ✅ | -| **NAT gateway support** | Consumption plan: ❌
Dedicated plan: ✅ | ✅ | ✅| +| **UDR support** | Consumption-only plan: ❌
Workload profile plan: ✅ | ✅ | ✅ | +| **NAT gateway support** | Consumption-only plan: ❌
Workload profile plan: ✅ | ✅ | ✅| ### Private networking integration From 5c25f9a36b6138595e9f9fbd9887dd09c4960c5c Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 11:13:33 -0500 Subject: [PATCH 14/15] Update container-service-general-considerations.md --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index f375ef6dd9..e7a3f6d717 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -332,7 +332,7 @@ Container Apps and Web App for Containers don't require virtual network integrat | | Container Apps| AKS| Web App for Containers| |---|--|--|--| -| **Built-in app ingress IP restrictions** | ✅ | ❌ | ✅ | +| **Built-in ingress IP restrictions** | ✅ | ❌ | ✅ | ## Application-level security From 1a4ee4c4d2d9c2a3ebd36a561821880df4d6316f Mon Sep 17 00:00:00 2001 From: Marcos M Martinez Date: Wed, 21 Feb 2024 11:30:26 -0500 Subject: [PATCH 15/15] Update container-service-general-considerations.md changing built-in ip restrictions section. --- docs/guide/container-service-general-considerations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index e7a3f6d717..294173aae8 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -328,7 +328,7 @@ In all scenarios, you can regulate networking communication within the wider vir #### Built-in IP restrictions for ingress -Container Apps and Web App for Containers don't require virtual network integration, so they provide built-in source IP restrictions for ingress traffic. If you use AKS, you can leverage the [Kubernetes Service api-resource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core) to achieve traffic filtering. +Container Apps and Web App for Containers provide built-in source IP restrictions for ingress traffic on individual applications. If you use AKS, you can leverage the [Kubernetes Service api-resource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core) to achieve traffic filtering on a per-app basis. | | Container Apps| AKS| Web App for Containers| |---|--|--|--|