From 812a2c6635eaca976afde23cec944897ce135aaf Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Tue, 13 Feb 2024 19:47:12 +0100 Subject: [PATCH 01/13] fix: errors and non-neutral phrasing --- ...ontainer-service-general-considerations.md | 45 ++++++++++--------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 294173aae8..a4fa5e8120 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -96,11 +96,13 @@ Most containerized applications run in Linux containers, which are supported by ## Networking considerations -It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on preference, such as extensibility or operational paradigms. +It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on preference: - [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability can use workload/dedicated profiles before considering alternatives to maximize their networking options. -- [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Workload teams can take advantage of network add-ons provided by Microsoft or can opt to install, manage, and operate network capabilities from the broad Kubernetes ecosystem. -- [Web App for Containers](https://azure.microsoft.com/products/app-service/containers/?activetab=pivot:deploytab) is feature of App Service, so the networking concepts, especially private networking integration, are very specific to App Service. This service will be familiar to workload teams that already use App Service. Teams that don't have experience with App Service and that want a more familiar Azure virtual network integration are encouraged to consider Container Apps. +- +- [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Workload teams can take advantage of various Azure managed [networking add-ons](https://learn.microsoft.com/azure/aks/integrations), as well as install and operate any add-ons from the broader Kubernetes ecosystem. + +- [Web App for Containers](https://azure.microsoft.com/products/app-service/containers/?activetab=pivot:deploytab) is feature of App Service. Thus the networking concepts, especially private networking integration, are very specific to App Service. This service will be familiar to workload teams that already use App Service. Teams that don't have experience with App Service and that want a more familiar Azure virtual network integration are encouraged to consider Container Apps. Keep in mind that networking is a foundational infrastructure layer. It's often difficult to make changes in design without re-deploying the workload, which can lead to downtime. Therefore, if your workload has specific networking requirements, review this section carefully before you narrow down your Azure container service selection. @@ -205,7 +207,7 @@ Both Container Apps and Web App for Containers support TLS 1.2 for their built-i With Container Apps and Web App for Containers, Azure fully abstracts away the Layer 4 and Layer 7 load balancers. -AKS uses a shared responsibility model in which Azure manages the underlying Azure infrastructure that the workload team configures by interfacing with the Kubernetes API. For Layer 7 load balancing in AKS, you can choose an Azure-managed options, for example the [AKS managed application routing add-on](/azure/aks/app-routing) or the [Application Gateway for Containers](/azure/application-gateway/for-containers/overview), or deploy and self-manage an ingress controller of your choice. +In contrast AKS uses a shared responsibility model in which Azure manages the underlying Azure infrastructure that the workload team configures by interfacing with the Kubernetes API. For Layer 7 load balancing in AKS, you can choose an Azure-managed options, for example the [AKS managed application routing add-on](/azure/aks/app-routing) or the [Application Gateway for Containers](/azure/application-gateway/for-containers/overview), or deploy and self-manage an ingress controller of your choice. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| @@ -239,7 +241,7 @@ Both Container Apps and Web App for Containers provide out-of-the-box solutions | **Managed TLS for Azure FQDNs** | Out of the box | N/A | Out of the box | | **Managed TLS for custom domains** | [In preview](/azure/container-apps/custom-domains-managed-certificates) | BYO | Out of the box or BYO | -AKS requires workload teams to configure their own ingress controllers, configure custom domain names on the cluster and DNS, and manage their own certificates. Although AKS can use the CNCF [cert-manager]( https://www.cncf.io/projects/cert-manager/) to manage TLS certificates, and Nginx or Traefik for ingress. +AKS users are responsible for managing DNS, cluster configurations and TLS certificates for their custom domains. Although AKS does not offer managed TLS, customers can leverage software from the Kubernetes ecosystem, for example the popular [cert-manager]( https://www.cncf.io/projects/cert-manager/) to manage TLS certificates. ### Mutual TLS @@ -249,7 +251,7 @@ Web App for Containers has built-in mTLS support for incoming client connections Container Apps also has built-in support for mTLS. It forwards the client certificate to the application in the HTTP header [X-Forwarded-Client-Cert](/azure/container-apps/client-certificate-authorization). You can also easily enable [automatic mTLS for internal communication between apps](/azure/container-apps/networking#mtls) in a single environment. -mTLS in AKS can be implemented through the [Istio-based service mesh as a managed add-on](/azure/aks/istio-about), which includes mTLS capabilities for incoming client connections and intra cluster communication between services. Workload teams could also choose to install and manage another service mesh offering from the Kubernetes ecosystem. These options make mTLS implementation in Kubernetes the most flexible. +Mutual TLS in AKS can be implemented through the [Istio-based service mesh as a managed add-on](/azure/aks/istio-about), which includes mTLS capabilities for incoming client connections and intra cluster communication between services. Workload teams could also choose to install and manage another service mesh offering from the Kubernetes ecosystem. These options make mTLS implementation in Kubernetes the most flexible. ### Service-specific networking concepts @@ -269,7 +271,7 @@ Failure to address security risks can lead to unauthorized access, data breaches Most Azure services, including Container Apps, AKS, and Web App for Containers, integrate with key security offerings, including Key Vault and managed identities. -AKS provides more configurability, at the expense of the additional overhead of running a Kubernetes cluster. Although AKS provides unique features like network policies, part of the reason it offers more security features is because its attack surface is larger. For example, AKS surfaces the Kubernetes control plane and virtual machine nodes, which in turn require additional security protection. +Of the services in this guide, AKS offers the most configurability and extensibility in part by surfacing underlying components, which often can be secured via configuration options. For example, customers can disable local accounts to the Kubernetes API server or turn on automatic updates to underlying nodes via configuration options. For a detailed comparison, carefully review the following considerations to ensure that your workload security requirements can be met. @@ -328,11 +330,16 @@ In all scenarios, you can regulate networking communication within the wider vir #### Built-in IP restrictions for ingress -Container Apps and Web App for Containers provide built-in source IP restrictions for ingress traffic on individual applications. If you use AKS, you can leverage the [Kubernetes Service api-resource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core) to achieve traffic filtering on a per-app basis. +While some customers restrict traffic on the network level via NSGs described above, others may choose to restrict traffic on the application level. -| | Container Apps| AKS| Web App for Containers| +Container Apps and Web App for Containers provide built-in source IP restrictions for ingress traffic on individual applications. + +For AKS workloads, implementation depends on the chosen ingress controller. When using the Azure managed [application routing add-on](/azure/aks/app-routing) IP restrictions can be [configured in the ingress object with nginx-specific annotations](/aks/app-routing-nginx-configuration#configuration-per-ingress-resource-through-annotations). + + +| | Container Apps| AKS | Web App for Containers| |---|--|--|--| -| **Built-in ingress IP restrictions** | ✅ | ❌ | ✅ | +| **Built-in ingress IP restrictions** | ✅ | Shared or self-managed | ✅ | ## Application-level security @@ -356,9 +363,9 @@ For more information, see: ### Managed identity support -It's a best practice to use managed identities to access Azure resources without using secrets. For example, you can pull images from Azure Container Registry without using a user name and password. +It's a best practice to use managed identities to access Azure resources without using secrets. For example, you can pull images from Azure Container Registry without using a username and password. -Azure container services provide managed identity support for the purpose of authenticating to Azure resources. For Container Apps and Web App for Containers, managed identity can be used for both control plane access to Azure resources, such as Azure Container Registry, and workload code access to Microsoft identity platform protected resources. AKS provides integrated managed identity support for the Kubernetes control plane to perform Azure resource control (such as managing the associated virtual machine scale sets), accessing container registry images, and cluster add-ons. Managed identity for workload code is configured by workload teams through the [Microsoft Entra Workload ID](/azure/aks/workload-identity-overview) feature. +Azure container services provide managed identity support, configurable out of the box for Container Apps and Web App for Containers. AKS provides integrated managed identity support for the Kubernetes control plane, Container Registry image management, and cluster add-ons. Managed identity for AKS applications is provided through the [Microsoft Entra Workload ID](/azure/aks/workload-identity-overview) feature. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| @@ -376,8 +383,6 @@ For more information, see: Threat protection against vulnerabilities is also important. It's a best practice to use [Defender for Containers](/azure/defender-for-cloud/defender-for-containers-introduction). Vulnerability assessments are supported in Azure container registries, so they can be used by any Azure container service, not just the ones described in this article. Defender for Containers runtime protection, however, is available only for AKS. -As AKS exposes the native Kubernetes API, it also supports security tooling from the Kubernetes ecosystem. Container Apps and Web App for Containers only supports built-in features and Microsoft Defender offerings. - | | Container Apps| AKS| Web App for Containers| |---|--|--|--| | **Runtime threat protection** | ❌ | ✅ | ❌ | @@ -388,7 +393,7 @@ Note that container image vulnerability assessments aren't real-time scans. The ## Security baselines -In general, most Azure container services integrate Azure security offerings. AKS has more extensive guidelines, in part because its attack surface is larger. Overall, keep in mind that a security feature set is just a small part of implementing cloud security. For more information about implementing security for container services, see the following service-specific security baselines: +In general, most Azure container services integrate Azure security offerings. Overall, keep in mind that a security feature set is just a small part of implementing cloud security. For more information about implementing security for container services, see the following service-specific security baselines: - [Azure security baseline for Container Apps](/security/benchmark/azure/baselines/azure-container-apps-security-baseline) - [Azure security baseline for Azure Kubernetes Service](/security/benchmark/azure/baselines/azure-kubernetes-service-aks-security-baseline) @@ -406,13 +411,11 @@ For more complete security guidance about AKS, see [Well-Architected Framework r To successfully run a workload in production, teams need to implement operational excellence practices, including centralized logging, monitoring, scalability, regular updates and patching, and image management. -In general, AKS provides the most configurability and therefore the most diversity and complexity for operational tasks. Azure assumes many operational responsibilities, such as updates, for offerings like Container Apps and Web App for Containers. - ### Updates and patches -You need to ensure that your application's underlying OS is updated and regularly patched. Keep in mind, however, that with every update there's a risk of failure. This section and the next one describe the main considerations for the three container services with regard to the shared responsibility between the customer and the platform. +It is important that an application's underlying OS is updated and regularly patched. Keep in mind, however, that with every update there's a risk of failure. This section and the next one describe the main considerations for the three container services with regard to the shared responsibility between the customer and the platform. -As a managed Kubernetes service, AKS will provide the updated images for the node OS and control plane components. But cluster operators are responsible for applying updates to their clusters. You can manually trigger updates or use the [cluster auto-upgrade channels](/azure/aks/auto-upgrade-cluster) feature to ensure clusters are up to date. See the AKS day-2 operations guide to learn about [patching and upgrading AKS clusters](/azure/architecture/operator-guides/aks/aks-upgrade-practices). +As a managed Kubernetes service, AKS will provide the updated images for the node OS and control plane components. But customers are responsible for applying updates. You can manually triggger updates or leverage the [cluster auto-upgrade channels](/azure/aks/auto-upgrade-cluster) feature to ensure your clusters are up to date. See the AKS day-2 operations guide to learn about [patching and upgrading AKS clusters](/azure/architecture/operator-guides/aks/aks-upgrade-practices). Container Apps and Web App for Containers are PaaS solutions. Azure is responsible for managing updates and patches, so customers can avoid the complexity of AKS upgrade management. @@ -585,7 +588,7 @@ Health checks are easiest to [implement in Web App for Containers](/azure/app-se - It sends the health check at one-minute intervals. You can't change the interval. - The minimum threshold that you can set for an unhealthy instance to be removed from the internal load balancing mechanism is two minutes. The unhealthy instance gets traffic for at least two minutes after it fails a health check. The default value for this setting is 10 minutes. -Container Apps and AKS, on the other hand, are much more flexible and offer similar options. In terms of some specific differences, AKS provides the following non-exhaustive options for performing health checks, which are not available in Container Apps: +Container Apps and AKS, on the other hand, are much more flexible and offer similar options. In terms of specific differences, AKS provides the following options for performing health checks, which are not available in Container Apps: - [gRPC support](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) - [Named ports](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#use-a-named-port) @@ -636,7 +639,7 @@ Well-architected solutions set the foundations for successful workloads. Althoug In general, when you compare Azure container services, a theme emerges: AKS surfaces the most underlying infrastructure, thus offering the greatest configurability and extensibility. The amount of operational overhead and complexity is highly variable for AKS workloads. Some teams can greatly reduce the operational overhead by using Microsoft managed add-ons and extensions, as well as auto-upgrade features. Other customers may prefer full control of the cluster in order to leverage full extensibility of Kubernetes and the CNCF ecosystem. For example, although Microsoft offers Flux as a managed GitOps extension, many teams choose instead to setup and operate ArgoCD on their own. -Workload teams, for example, that have less operations experience, prefer to focus on application features, or do not have a need for open-source software and fully portable systems might benefit from starting with a pure PaaS offering. We recommend that they first consider Container Apps. +Workload teams that, for example, do not require CNCF applications, have less operations experience or prefer to focus on application features might prefer a PaaS offering. We recommend that they first consider Container Apps. Although Container Apps and Web App for Containers are both PaaS offerings that provide similar levels of Microsoft-managed infrastructure, a key difference is that Container Apps is closer to Kubernetes and provides additional cloud-native capabilities for service discovery, event-driven autoscaling, [Dapr](https://dapr.io/) integration, and more. However, teams that don't need these capabilities and are familiar with App Service networking and deployment models might prefer Web App for Containers. From 1e5beb905dc5612df40c847d13a9edea6b8681b6 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Mon, 4 Mar 2024 16:46:41 +0100 Subject: [PATCH 02/13] fix links per build service --- docs/guide/container-service-general-considerations.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index a4fa5e8120..b8cf4371cb 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -100,7 +100,7 @@ It's important to understand networking design early in your planning processes - [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability can use workload/dedicated profiles before considering alternatives to maximize their networking options. - -- [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Workload teams can take advantage of various Azure managed [networking add-ons](https://learn.microsoft.com/azure/aks/integrations), as well as install and operate any add-ons from the broader Kubernetes ecosystem. +- [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Workload teams can take advantage of various Azure managed [networking add-ons](/azure/aks/integrations), as well as install and operate any add-ons from the broader Kubernetes ecosystem. - [Web App for Containers](https://azure.microsoft.com/products/app-service/containers/?activetab=pivot:deploytab) is feature of App Service. Thus the networking concepts, especially private networking integration, are very specific to App Service. This service will be familiar to workload teams that already use App Service. Teams that don't have experience with App Service and that want a more familiar Azure virtual network integration are encouraged to consider Container Apps. @@ -334,7 +334,7 @@ While some customers restrict traffic on the network level via NSGs described ab Container Apps and Web App for Containers provide built-in source IP restrictions for ingress traffic on individual applications. -For AKS workloads, implementation depends on the chosen ingress controller. When using the Azure managed [application routing add-on](/azure/aks/app-routing) IP restrictions can be [configured in the ingress object with nginx-specific annotations](/aks/app-routing-nginx-configuration#configuration-per-ingress-resource-through-annotations). +For AKS workloads, implementation depends on the chosen ingress controller. When using the Azure managed [application routing add-on](/azure/aks/app-routing) IP restrictions can be [configured in the ingress object with nginx-specific annotations](/aks/app-routing-nginx-configuration). | | Container Apps| AKS | Web App for Containers| From 27530d7f39e82f339b5b90d837b6d73f31eaf05f Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Mon, 4 Mar 2024 17:40:48 +0100 Subject: [PATCH 03/13] containers-guide(deployments): simplify. note application architecture reqs --- docs/guide/container-service-general-considerations.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index b8cf4371cb..ea957bca0c 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -607,11 +607,9 @@ The ability to deploy and replace applications without causing any downtime for | | Container Apps| AKS| Web App for Containers| |---|--|--|--| -| **Zero-downtime strategy** | Rolling update | Rolling update and all other Kubernetes strategies | Slots | +| **Zero-downtime strategy** | [Rolling update](/azure/container-apps/revisions#zero-downtime-deployment) | [Rolling update, plus all other Kubernetes strategies](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy) | [Deployment slots](/azure/app-service/deploy-staging-slots) | -- **Web App for Containers**: Web App for Containers implements [slots](/azure/app-service/deploy-staging-slots), which are placeholders in which you can deploy new versions of your containers and test them before sending them to the production slot. Each slot has its own host name, configuration, and binaries. However, this feature needs to be implemented and isn't configured out-of-the-box. -- **AKS**: The default deployment strategy in Kubernetes is called [rolling update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/). The process starts a new instance of your container in parallel with the existing one. The old one is removed only after the new one starts, so there's no downtime. You can also choose [other deployment strategies that are available in Kubernetes](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy). -- **Container Apps**: By default, any container that's deployed to Container Apps has [zero downtime](/azure/container-apps/revisions#zero-downtime-deployment). The implementation works like a Kubernetes rolling update. You also have the option to validate and test containers before you send them to production by using [direct revision access](/azure/container-apps/revisions#direct-revision-access). +Please note that the application architectures must also support zero-downtime deployment. See [Azure Well-Architected Framework](/azure/well-architected/) for guidance. ### Resource limits From cbb02f8c93e9b353891c8835513124d99de05c8d Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Mon, 4 Mar 2024 17:42:58 +0100 Subject: [PATCH 04/13] containers-guide(ip-restrictions): remove for now per Jorge's feedback. Rework later --- .../container-service-general-considerations.md | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index ea957bca0c..3e5cd0c536 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -328,19 +328,6 @@ In all scenarios, you can regulate networking communication within the wider vir |---|--|--|--| | **Network security groups** | Consumption plan: ✅
Dedicated plan: ✅ | ✅ | ✅ VNet-integrated apps: egress only | -#### Built-in IP restrictions for ingress - -While some customers restrict traffic on the network level via NSGs described above, others may choose to restrict traffic on the application level. - -Container Apps and Web App for Containers provide built-in source IP restrictions for ingress traffic on individual applications. - -For AKS workloads, implementation depends on the chosen ingress controller. When using the Azure managed [application routing add-on](/azure/aks/app-routing) IP restrictions can be [configured in the ingress object with nginx-specific annotations](/aks/app-routing-nginx-configuration). - - -| | Container Apps| AKS | Web App for Containers| -|---|--|--|--| -| **Built-in ingress IP restrictions** | ✅ | Shared or self-managed | ✅ | - ## Application-level security You need to secure workloads not just at the network and infrastructure level, but also at the workload and application level. Azure container solutions integrate with Azure security offerings to help standardize security implementation and controls for your applications. From bfb71e6621884ffd9e376afcbc763e05b979abb2 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Tue, 5 Mar 2024 14:07:37 +0100 Subject: [PATCH 05/13] containers-guide(service-discovery): from customer perspective, it's about OOTB URLs --- ...ontainer-service-general-considerations.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 3e5cd0c536..a5bcb2f7cb 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -38,7 +38,6 @@ The considerations in this article are divided into four categories: - Protocol coverage - Load balancing - Service discovery -- Custom domains and managed TLS - Mutual TLS - Networking concepts for specific Azure services @@ -218,30 +217,31 @@ In contrast AKS uses a shared responsibility model in which Azure manages the un In cloud architectures, runtimes can be removed and re-created at any time to rebalance resources, so instance IP addresses regularly change. These architectures use fully qualified domain names (FQDNs) for reliable and consistent communication. -| | Container Apps| AKS| Web App for Containers| -|---|--|--|--| -| **Service discovery** | Azure-managed FQDN | Kubernetes configurable | Azure-managed FQDN | +#### Azure Managed FQDNs + +To help developers quickly test their deployed applications, customers can leverage Azure managed domains that are accessible both over http and https. -Web Apps for Containers provides public ingress (north-south communication) FQDNs out of the box. No additional DNS configuration is required. However, there's no built-in mechanism to facilitate or restrict traffic between other apps (east-west communication). +Both [Azure Container Apps](/azure/container-apps/connect-apps) and [Web App for Containers](/azure/app-service/tutorial-secure-domain-certificate) provide default FQDNs including variations for container app [revisions](/azure/container-apps/revisions) and web app [slots](/azure/app-service/deploy-staging-slots). -Container Apps also provides public ingress FQDNs. However, Container Apps goes further by allowing the app FQDN to be exposed and [restricting traffic only within the environment](/azure/container-apps/networking). This functionality makes it easier to manage east-west communication and enable components like Dapr. +| | Container Apps | AKS | Web App for Containers | +|---|--|--|--| +| **Default FQDNs** | `...azurecontainerapps.io` | via [AGC](/azure/application-gateway/for-containers/overview)* | `.azurewebsites.net` | -Kubernetes deployments are not initially discoverable within or from outside the cluster. You must create Kubernetes services as defined by the Kubernetes API, which then expose applications to the network in an addressable way. +For AKS users the quickest way to expose an AKS application for external access is via a Kubernetes service of type `LoadBalancer`, which makes application reachable via an IP address. To connect securely to an application via https with Azure managed FQDN and TLS, with an Azure managed solution, AKS customers can use [Application Gateway for Containers (AGC)](/azure/application-gateway/for-containers/overview). -> [!important] -> Only Container Apps and AKS provide service discovery through internal DNS schemes within their respective environments. This functionality can simplify DNS configurations across dev/test and production environments. For example, you can create these environments with arbitrary service names that have to be unique only within the environment or cluster, so they can be the same across dev/test and production. With Web App for Containers, service names must be unique across different environments to avoid conflicts with Azure DNS. +While default FQDNs are helpful for quick tests, most customers use custom domains and custom certificates, especially in production. ### Custom domains and managed TLS -Both Container Apps and Web App for Containers provide out-of-the-box solutions for custom domains and certificate management. +Both Container Apps and Web App for Containers provide out-of-the-box solutions for custom domains and certificate management, including the issuing of certificates. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| | **Configure custom domains** | Out of the box | BYO | Out of the box | | **Managed TLS for Azure FQDNs** | Out of the box | N/A | Out of the box | -| **Managed TLS for custom domains** | [In preview](/azure/container-apps/custom-domains-managed-certificates) | BYO | Out of the box or BYO | +| **Managed TLS for custom domains** | [In preview](/azure/container-apps/custom-domains-managed-certificates) | BYO | [Supported](/azure/app-service/configure-ssl-certificate) or BYO | -AKS users are responsible for managing DNS, cluster configurations and TLS certificates for their custom domains. Although AKS does not offer managed TLS, customers can leverage software from the Kubernetes ecosystem, for example the popular [cert-manager]( https://www.cncf.io/projects/cert-manager/) to manage TLS certificates. +AKS users are responsible for managing and configuring custom domains, DNS and ingress, for which there are many established patterns depending on the chosen ingress solution. For TLS certificate management, e.g. issuance and rotation, it is common to leverage [cert-manager]( https://www.cncf.io/projects/cert-manager/) from the Kubernetes ecosystem or leverage [Azure Key Vault certificates](/azure/key-vault/certificates/certificate-scenarios). ### Mutual TLS From 8f08d20670bbbcaede6aef3fbc7b7aa5197c8d05 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Tue, 5 Mar 2024 14:42:07 +0100 Subject: [PATCH 06/13] containers-guide(managed-identity): simplified, full support. customer responsible for details. --- docs/guide/container-service-general-considerations.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index a5bcb2f7cb..ed8d5891aa 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -350,15 +350,11 @@ For more information, see: ### Managed identity support -It's a best practice to use managed identities to access Azure resources without using secrets. For example, you can pull images from Azure Container Registry without using a username and password. - -Azure container services provide managed identity support, configurable out of the box for Container Apps and Web App for Containers. AKS provides integrated managed identity support for the Kubernetes control plane, Container Registry image management, and cluster add-ons. Managed identity for AKS applications is provided through the [Microsoft Entra Workload ID](/azure/aks/workload-identity-overview) feature. +Applications with assigned managed identities can access Azure resources without passwords. All container services mentioned in this guide support managed identities. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| -| **Infrastructure managed identity support** | N/A | ✅ | N/A | -| **Container-pull managed identity support** | ✅ | ✅ | ✅ | -| **Application managed identity support** | ✅ | ✅ | ✅ | +| **Managed identity support** | ✅ | ✅ | ✅ | For more information, see: From 0ec7594ff72df9e6feba03cb022c7ad9983bc92f Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Tue, 5 Mar 2024 14:55:29 +0100 Subject: [PATCH 07/13] containers-guide(security): remove comparison intro, references to k8s api --- ...ontainer-service-general-considerations.md | 27 ------------------- 1 file changed, 27 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index ed8d5891aa..28d7704d28 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -267,33 +267,6 @@ The preceding sections focus on network design. Continue to the next section to Failure to address security risks can lead to unauthorized access, data breaches, or leaks of sensitive information. Containers offer an encapsulated environment for your application. The hosting systems and underlying network overlays, however, require additional guardrails. Your choice of Azure container service needs to support your specific requirements for securing each application individually and provide proper security measures to prevent unauthorized access and mitigate the risk of attacks. -### Security comparison overview - -Most Azure services, including Container Apps, AKS, and Web App for Containers, integrate with key security offerings, including Key Vault and managed identities. - -Of the services in this guide, AKS offers the most configurability and extensibility in part by surfacing underlying components, which often can be secured via configuration options. For example, customers can disable local accounts to the Kubernetes API server or turn on automatic updates to underlying nodes via configuration options. - -For a detailed comparison, carefully review the following considerations to ensure that your workload security requirements can be met. - -### Kubernetes control plane security - -AKS offers the most flexibility of the three options considered in this article, providing full access to the Kubernetes API so that you can customize container orchestration. This access to the Kubernetes API, however, also represents a significant attack surface, and you need to secure it. - -> [!Important] -> Note that this section isn't relevant for Web App for Containers, which uses the Azure Resource Manager API as its control plane. - -#### Identity-based security - -Customers are responsible for securing identity-based access to the API. Out of the box, Kubernetes provides its own authentication and authorization management system, which also needs to be secured with access controls. - -To take advantage of a single plane of glass for identity and access management on Azure, it's a best practice to [disable Kubernetes-specific local accounts](/azure/aks/manage-local-accounts-managed-azure-ad) and instead [implement AKS-managed Microsoft Entra integration](/azure/aks/enable-authentication-microsoft-entra-id) together with [Azure RBAC for Kubernetes](/azure/aks/manage-azure-rbac). If you implement this best practice, administrators don't need to perform identity and access management on multiple platforms. - -| | Container Apps | AKS| -|---|--|--| -| **Kubernetes API access controls** | No access | Full access | - -Customers who use Container Apps don't have access to the Kubernetes API. Microsoft provides security for this API. - #### Network-based security If you want to restrict network access to the Kubernetes control plane, you need to use AKS, which provides two options. The first option is to use [private AKS clusters](/azure/aks/private-clusters#create-a-private-aks-cluster), which use Azure Private Link between the API server's private network and the AKS cluster's private network. The second option is [API Server VNet integration (preview)](/azure/aks/api-server-vnet-integration), where the API server is integrated into a delegated subnet. See the documentation to learn more. From bff6231fb3e67b627bc7755902f6e0751a039590 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Tue, 5 Mar 2024 15:30:09 +0100 Subject: [PATCH 08/13] containers-guide(ip-restrictions): bring back, reword as PaaS specific --- .../container-service-general-considerations.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 28d7704d28..89e5f56741 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -301,6 +301,18 @@ In all scenarios, you can regulate networking communication within the wider vir |---|--|--|--| | **Network security groups** | Consumption plan: ✅
Dedicated plan: ✅ | ✅ | ✅ VNet-integrated apps: egress only | +#### Built-in IP restrictions for ingress + +Typically network traffic restrictions are applied via Layer 4 rules described above. However, in PaaS scenarios of applications *without* virtual network integration, it can be useful to restrict traffic on the application layer. + +Container Apps and Web App for Containers provide built-in source IP restrictions for ingress traffic on individual applications. + +| | Container Apps| AKS | Web App for Containers| +|---|--|--|--| +| **Built-in ingress IP restrictions** | ✅ | Shared or self-managed | ✅ | + +For AKS workloads, implementation depends on the chosen ingress controller. This is configurable on the managed [application routing add-on](/azure/aks/app-routing). Keep in mind, however that this scenario is less relevant for AKS and it is recommended to control network traffic via network security groups on the virtual network. + ## Application-level security You need to secure workloads not just at the network and infrastructure level, but also at the workload and application level. Azure container solutions integrate with Azure security offerings to help standardize security implementation and controls for your applications. From 7229ad383a3006546be5b040841c735c94faaf25 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Wed, 6 Mar 2024 16:07:42 +0100 Subject: [PATCH 09/13] containers-guide: apply chad's suggestions from code review Co-authored-by: Chad Kittel --- docs/guide/container-service-general-considerations.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 89e5f56741..20e80d8916 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -98,10 +98,9 @@ Most containerized applications run in Linux containers, which are supported by It's important to understand networking design early in your planning processes due to security and compliance constraints and imposed guidelines. In general, the major differences among the Azure services covered in this guide depend on preference: - [Container Apps](https://azure.microsoft.com/products/container-apps) is a PaaS offering that provides many Azure-managed networking features, like service discovery and internal managed domains. Workload teams that need a bit more configurability can use workload/dedicated profiles before considering alternatives to maximize their networking options. -- - [AKS](https://azure.microsoft.com/products/kubernetes-service/) is the most configurable of the three services and provides the most control over network flow. For example, it provides custom ingress controllers and the control of intra-cluster traffic via Kubernetes network policies. Workload teams can take advantage of various Azure managed [networking add-ons](/azure/aks/integrations), as well as install and operate any add-ons from the broader Kubernetes ecosystem. -- [Web App for Containers](https://azure.microsoft.com/products/app-service/containers/?activetab=pivot:deploytab) is feature of App Service. Thus the networking concepts, especially private networking integration, are very specific to App Service. This service will be familiar to workload teams that already use App Service. Teams that don't have experience with App Service and that want a more familiar Azure virtual network integration are encouraged to consider Container Apps. +- [Web App for Containers](https://azure.microsoft.com/products/app-service/containers/) is feature of App Service. Thus, the networking concepts, especially private networking integration, are very specific to App Service. This service will be familiar to workload teams that already use App Service. Teams that don't have experience with App Service and that want a more familiar Azure virtual network integration are encouraged to consider Container Apps. Keep in mind that networking is a foundational infrastructure layer. It's often difficult to make changes in design without re-deploying the workload, which can lead to downtime. Therefore, if your workload has specific networking requirements, review this section carefully before you narrow down your Azure container service selection. @@ -241,7 +240,7 @@ Both Container Apps and Web App for Containers provide out-of-the-box solutions | **Managed TLS for Azure FQDNs** | Out of the box | N/A | Out of the box | | **Managed TLS for custom domains** | [In preview](/azure/container-apps/custom-domains-managed-certificates) | BYO | [Supported](/azure/app-service/configure-ssl-certificate) or BYO | -AKS users are responsible for managing and configuring custom domains, DNS and ingress, for which there are many established patterns depending on the chosen ingress solution. For TLS certificate management, e.g. issuance and rotation, it is common to leverage [cert-manager]( https://www.cncf.io/projects/cert-manager/) from the Kubernetes ecosystem or leverage [Azure Key Vault certificates](/azure/key-vault/certificates/certificate-scenarios). +AKS users are responsible for managing and configuring custom domains, DNS and ingress, for which there are many established patterns depending on the chosen ingress solution. For TLS certificate management, e.g. issuance and rotation, it is common to leverage [cert-manager]( https://www.cncf.io/projects/cert-manager/) from the Kubernetes ecosystem or use [Azure Key Vault certificates](/azure/key-vault/certificates/certificate-scenarios). ### Mutual TLS @@ -383,7 +382,7 @@ To successfully run a workload in production, teams need to implement operationa It is important that an application's underlying OS is updated and regularly patched. Keep in mind, however, that with every update there's a risk of failure. This section and the next one describe the main considerations for the three container services with regard to the shared responsibility between the customer and the platform. -As a managed Kubernetes service, AKS will provide the updated images for the node OS and control plane components. But customers are responsible for applying updates. You can manually triggger updates or leverage the [cluster auto-upgrade channels](/azure/aks/auto-upgrade-cluster) feature to ensure your clusters are up to date. See the AKS day-2 operations guide to learn about [patching and upgrading AKS clusters](/azure/architecture/operator-guides/aks/aks-upgrade-practices). +As a managed Kubernetes service, AKS will provide the updated images for the node OS and control plane components. But workload teams are responsible for applying updates to their clusters. You can manually trigger updates or leverage the [cluster auto-upgrade channels](/azure/aks/auto-upgrade-cluster) feature to ensure your clusters are up to date. See the AKS day-2 operations guide to learn about [patching and upgrading AKS clusters](/azure/architecture/operator-guides/aks/aks-upgrade-practices). Container Apps and Web App for Containers are PaaS solutions. Azure is responsible for managing updates and patches, so customers can avoid the complexity of AKS upgrade management. From 6c3ed5cc7907dbcdb3e2320ef7f674112d41c450 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Wed, 13 Mar 2024 17:53:19 +0100 Subject: [PATCH 10/13] Revert "containers-guide(service-discovery): from customer perspective, it's about OOTB URLs" because it was too much for a patch/fix This reverts commit bfb71e6621884ffd9e376afcbc763e05b979abb2. --- ...ontainer-service-general-considerations.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 20e80d8916..15f4e746d7 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -38,6 +38,7 @@ The considerations in this article are divided into four categories: - Protocol coverage - Load balancing - Service discovery +- Custom domains and managed TLS - Mutual TLS - Networking concepts for specific Azure services @@ -216,31 +217,30 @@ In contrast AKS uses a shared responsibility model in which Azure manages the un In cloud architectures, runtimes can be removed and re-created at any time to rebalance resources, so instance IP addresses regularly change. These architectures use fully qualified domain names (FQDNs) for reliable and consistent communication. -#### Azure Managed FQDNs - -To help developers quickly test their deployed applications, customers can leverage Azure managed domains that are accessible both over http and https. +| | Container Apps| AKS| Web App for Containers| +|---|--|--|--| +| **Service discovery** | Azure-managed FQDN | Kubernetes configurable | Azure-managed FQDN | -Both [Azure Container Apps](/azure/container-apps/connect-apps) and [Web App for Containers](/azure/app-service/tutorial-secure-domain-certificate) provide default FQDNs including variations for container app [revisions](/azure/container-apps/revisions) and web app [slots](/azure/app-service/deploy-staging-slots). +Web Apps for Containers provides public ingress (north-south communication) FQDNs out of the box. No additional DNS configuration is required. However, there's no built-in mechanism to facilitate or restrict traffic between other apps (east-west communication). -| | Container Apps | AKS | Web App for Containers | -|---|--|--|--| -| **Default FQDNs** | `...azurecontainerapps.io` | via [AGC](/azure/application-gateway/for-containers/overview)* | `.azurewebsites.net` | +Container Apps also provides public ingress FQDNs. However, Container Apps goes further by allowing the app FQDN to be exposed and [restricting traffic only within the environment](/azure/container-apps/networking). This functionality makes it easier to manage east-west communication and enable components like Dapr. -For AKS users the quickest way to expose an AKS application for external access is via a Kubernetes service of type `LoadBalancer`, which makes application reachable via an IP address. To connect securely to an application via https with Azure managed FQDN and TLS, with an Azure managed solution, AKS customers can use [Application Gateway for Containers (AGC)](/azure/application-gateway/for-containers/overview). +Kubernetes deployments are not initially discoverable within or from outside the cluster. You must create Kubernetes services as defined by the Kubernetes API, which then expose applications to the network in an addressable way. -While default FQDNs are helpful for quick tests, most customers use custom domains and custom certificates, especially in production. +> [!important] +> Only Container Apps and AKS provide service discovery through internal DNS schemes within their respective environments. This functionality can simplify DNS configurations across dev/test and production environments. For example, you can create these environments with arbitrary service names that have to be unique only within the environment or cluster, so they can be the same across dev/test and production. With Web App for Containers, service names must be unique across different environments to avoid conflicts with Azure DNS. ### Custom domains and managed TLS -Both Container Apps and Web App for Containers provide out-of-the-box solutions for custom domains and certificate management, including the issuing of certificates. +Both Container Apps and Web App for Containers provide out-of-the-box solutions for custom domains and certificate management. | | Container Apps| AKS| Web App for Containers| |---|--|--|--| | **Configure custom domains** | Out of the box | BYO | Out of the box | | **Managed TLS for Azure FQDNs** | Out of the box | N/A | Out of the box | -| **Managed TLS for custom domains** | [In preview](/azure/container-apps/custom-domains-managed-certificates) | BYO | [Supported](/azure/app-service/configure-ssl-certificate) or BYO | +| **Managed TLS for custom domains** | [In preview](/azure/container-apps/custom-domains-managed-certificates) | BYO | Out of the box or BYO | -AKS users are responsible for managing and configuring custom domains, DNS and ingress, for which there are many established patterns depending on the chosen ingress solution. For TLS certificate management, e.g. issuance and rotation, it is common to leverage [cert-manager]( https://www.cncf.io/projects/cert-manager/) from the Kubernetes ecosystem or use [Azure Key Vault certificates](/azure/key-vault/certificates/certificate-scenarios). +AKS users are responsible for managing DNS, cluster configurations and TLS certificates for their custom domains. Although AKS does not offer managed TLS, customers can leverage software from the Kubernetes ecosystem, for example the popular [cert-manager]( https://www.cncf.io/projects/cert-manager/) to manage TLS certificates. ### Mutual TLS From b8b61f0c1b6966974e00261b3ec264edfe5838d7 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Wed, 13 Mar 2024 18:01:03 +0100 Subject: [PATCH 11/13] Revert "containers-guide(security): remove comparison intro, references to k8s api" preparing to make changes per Chad's suggestions in PR This reverts commit 0ec7594ff72df9e6feba03cb022c7ad9983bc92f. --- ...ontainer-service-general-considerations.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 15f4e746d7..ec661daed1 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -266,6 +266,33 @@ The preceding sections focus on network design. Continue to the next section to Failure to address security risks can lead to unauthorized access, data breaches, or leaks of sensitive information. Containers offer an encapsulated environment for your application. The hosting systems and underlying network overlays, however, require additional guardrails. Your choice of Azure container service needs to support your specific requirements for securing each application individually and provide proper security measures to prevent unauthorized access and mitigate the risk of attacks. +### Security comparison overview + +Most Azure services, including Container Apps, AKS, and Web App for Containers, integrate with key security offerings, including Key Vault and managed identities. + +Of the services in this guide, AKS offers the most configurability and extensibility in part by surfacing underlying components, which often can be secured via configuration options. For example, customers can disable local accounts to the Kubernetes API server or turn on automatic updates to underlying nodes via configuration options. + +For a detailed comparison, carefully review the following considerations to ensure that your workload security requirements can be met. + +### Kubernetes control plane security + +AKS offers the most flexibility of the three options considered in this article, providing full access to the Kubernetes API so that you can customize container orchestration. This access to the Kubernetes API, however, also represents a significant attack surface, and you need to secure it. + +> [!Important] +> Note that this section isn't relevant for Web App for Containers, which uses the Azure Resource Manager API as its control plane. + +#### Identity-based security + +Customers are responsible for securing identity-based access to the API. Out of the box, Kubernetes provides its own authentication and authorization management system, which also needs to be secured with access controls. + +To take advantage of a single plane of glass for identity and access management on Azure, it's a best practice to [disable Kubernetes-specific local accounts](/azure/aks/manage-local-accounts-managed-azure-ad) and instead [implement AKS-managed Microsoft Entra integration](/azure/aks/enable-authentication-microsoft-entra-id) together with [Azure RBAC for Kubernetes](/azure/aks/manage-azure-rbac). If you implement this best practice, administrators don't need to perform identity and access management on multiple platforms. + +| | Container Apps | AKS| +|---|--|--| +| **Kubernetes API access controls** | No access | Full access | + +Customers who use Container Apps don't have access to the Kubernetes API. Microsoft provides security for this API. + #### Network-based security If you want to restrict network access to the Kubernetes control plane, you need to use AKS, which provides two options. The first option is to use [private AKS clusters](/azure/aks/private-clusters#create-a-private-aks-cluster), which use Azure Private Link between the API server's private network and the AKS cluster's private network. The second option is [API Server VNet integration (preview)](/azure/aks/api-server-vnet-integration), where the API server is integrated into a delegated subnet. See the documentation to learn more. From 4f4a8c610369680bc3e90f6552c5e2a37e0bca28 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Thu, 14 Mar 2024 15:20:58 +0100 Subject: [PATCH 12/13] containers-guide(ip-restrictions): apply feedback re: consumption, 'built-in' wording --- docs/guide/container-service-general-considerations.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index ec661daed1..4f2761959c 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -327,7 +327,7 @@ In all scenarios, you can regulate networking communication within the wider vir |---|--|--|--| | **Network security groups** | Consumption plan: ✅
Dedicated plan: ✅ | ✅ | ✅ VNet-integrated apps: egress only | -#### Built-in IP restrictions for ingress +#### IP restrictions for ingress Typically network traffic restrictions are applied via Layer 4 rules described above. However, in PaaS scenarios of applications *without* virtual network integration, it can be useful to restrict traffic on the application layer. @@ -335,9 +335,10 @@ Container Apps and Web App for Containers provide built-in source IP restriction | | Container Apps| AKS | Web App for Containers| |---|--|--|--| -| **Built-in ingress IP restrictions** | ✅ | Shared or self-managed | ✅ | +| **Ingress IP restrictions on application layer** | Out of the box | Self-managed or via managed add-on | Out of the box | +| **Resource consumption** | - | Consumes cluster resources | - | -For AKS workloads, implementation depends on the chosen ingress controller. This is configurable on the managed [application routing add-on](/azure/aks/app-routing). Keep in mind, however that this scenario is less relevant for AKS and it is recommended to control network traffic via network security groups on the virtual network. +For AKS workloads, implementation depends on the chosen ingress controller. Both self-managed and the Azure managed [application routing add-on](/azure/aks/app-routing) consume cluster resources. ## Application-level security From c39b5a9a7abc4fa2975fa03bf64d87dd676870e3 Mon Sep 17 00:00:00 2001 From: Julie Ng Date: Thu, 14 Mar 2024 15:23:09 +0100 Subject: [PATCH 13/13] containers-guide(k8s-security-tooling): apply feedback from chad --- docs/guide/container-service-general-considerations.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/guide/container-service-general-considerations.md b/docs/guide/container-service-general-considerations.md index 4f2761959c..4525a3b44a 100644 --- a/docs/guide/container-service-general-considerations.md +++ b/docs/guide/container-service-general-considerations.md @@ -378,6 +378,8 @@ For more information, see: Threat protection against vulnerabilities is also important. It's a best practice to use [Defender for Containers](/azure/defender-for-cloud/defender-for-containers-introduction). Vulnerability assessments are supported in Azure container registries, so they can be used by any Azure container service, not just the ones described in this article. Defender for Containers runtime protection, however, is available only for AKS. +As AKS exposes the native Kubernetes API, cluster security also can be evaluated with Kubernetes specific security tooling from the Kubernetes ecosystem. + | | Container Apps| AKS| Web App for Containers| |---|--|--|--| | **Runtime threat protection** | ❌ | ✅ | ❌ |