Skip to content

Commit

Permalink
networking changes
Browse files Browse the repository at this point in the history
  • Loading branch information
@craigshoemaker
craigshoemaker committed Mar 31, 2023
1 parent c9c5059 commit 08ff85c
Show file tree
Hide file tree
Showing 6 changed files with 523 additions and 38 deletions.
6 changes: 6 additions & 0 deletions articles/container-apps/TOC.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,12 @@
href: custom-domains-certificates.md
- name: Set up environment custom DNS suffix
href: environment-custom-dns-suffix.md
- name: Network security
items:
- name: Configure WAF Application Gateway
href: waf-app-gateway.md
- name: Enable Azure Firewall
href: user-defined-routes.md
- name: Connect to a cloud service using Service Connector
items:
- name: .NET app with Blob Storage
Expand Down
14 changes: 8 additions & 6 deletions articles/container-apps/firewall-integration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,23 @@
title: Securing a custom VNET in Azure Container Apps
description: Firewall settings to secure a custom VNET in Azure Container Apps
services: container-apps
author: JennyLawrance
author: CaryChai
ms.service: container-apps
ms.custom: event-tier1-build-2022
ms.topic: reference
ms.date: 07/15/2022
ms.author: jennylaw
ms.date: 03/29/2023
ms.author: cachai
---

# Securing a custom VNET in Azure Container Apps
# Securing a custom VNET in Azure Container Apps with Network Security Groups

Network Security Groups (NSGs) needed to configure virtual networks closely resemble the settings required by Kubernetes.

You can lock down a network via NSGs with more restrictive rules than the default NSG rules to control all inbound and outbound traffic for the Container App Environment.
You can lock down a network via NSGs with more restrictive rules than the default NSG rules to control all inbound and outbound traffic for the Container Apps environment at the subscription level.

Using custom user-defined routes (UDRs) or ExpressRoutes, other than with UDRs of selected destinations that you own, are not yet supported for Container App Environments with VNETs. Therefore, securing outbound traffic with a firewall is not yet supported.
In the Consumption + Dedicated plan structure, user-defined routes (UDRs) and securing outbound traffic with a firewall are supported. Learn more in the [networking architecture document](./networking.md#user-defined-routes-udr---preview).

In the Consumption plan, custom user-defined routes (UDRs) and ExpressRoutes are not supported.

## NSG allow rules

Expand Down
107 changes: 76 additions & 31 deletions articles/container-apps/networking.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,47 +2,56 @@
title: Networking architecture in Azure Container Apps
description: Learn how to configure virtual networks in Azure Container Apps.
services: container-apps
author: craigshoemaker
author: cachai
ms.service: container-apps
ms.topic: conceptual
ms.date: 03/30/2023
ms.author: cshoe
ms.date: 03/29/2023
ms.author: cachai
---

# Networking architecture in Azure Container Apps

Azure Container Apps run in the context of an [environment](environment.md), supported by a virtual network (VNET). When you create an environment, you can provide a custom VNET, otherwise a VNET is automatically generated for you. Generated VNETs are inaccessible to you as they're created in Microsoft's tenant. To take full control over your VNET, provide an existing VNET to Container Apps as you create your environment.
Azure Container Apps run in the context of an [environment](environment.md), which is supported by a virtual network (VNet). By default, your Container App Environment is created with a VNet that is automatically generated for you. Generated VNets are inaccessible to you as they're created in Microsoft's tenant. This VNet is publicly accessible over the internet, can only reach internet accessible endpoints, and supports a limited subset of networking capabilities such as ingress IP restrictions and container app level ingress controls.

The following articles feature step-by-step instructions for creating Container Apps environments with different accessibility levels.
It is recommended that you use the Custom VNet configuration to provide your own VNet if you need additional Azure Networking features such as integration with Application Gateway, Network Security Groups, or communicating with resources behind private endpoints in your virtual network. The features available will depend on your plan structure selection.

## Plan Structure Selection

There are two plan structures in Container Apps. One that supports only the Consumption plan (GA), and the other Consumption + Dedicated plan structure (public preview) supports both the Dedicated and Consumption plans. The two plan structures share many of the same networking characteristics. However, there are some key differences.

| Plan Type | Description |
|-----------|-------------|
| [Consumption + Dedicated (preview)](filler) | Supports UDR and egress through NAT Gateway. The minimum required subnet size is /27. |
| [Consumption](filler) | Doesn't support user defined routes (UDRs) and egress through NAT Gateway. The minimum required subnet size is /23. |

## Accessibility Levels

In Container Apps, you can configure whether your Container Apps will allow public ingress or only ingress from within your VNet at the environment level.

| Accessibility level | Description |
|--|--|
|---------------------|-------------|
| [External](vnet-custom.md) | Container Apps environments deployed as external resources are available for public requests. External environments are deployed with a virtual IP on an external, public facing IP address. |
| [Internal](vnet-custom-internal.md) | When set to internal, the environment has no public endpoint. Internal environments are deployed with a virtual IP (VIP) mapped to an internal IP address. The internal endpoint is an Azure internal load balancer (ILB) and IP addresses are issued from the custom VNET's list of private IP addresses. |
| [Internal](vnet-custom-internal.md) | When set to internal, the environment has no public endpoint. Internal environments are deployed with a virtual IP (VIP) mapped to an internal IP address. The internal endpoint is an Azure internal load balancer (ILB) and IP addresses are issued from the custom VNet's list of private IP addresses. |

## Custom VNET configuration
## Custom VNet configuration

As you create a custom VNET, keep in mind the following situations:
As you create a custom VNet, keep in mind the following situations:

- If you want your container app to restrict all outside access, create an [internal Container Apps environment](vnet-custom-internal.md).

- When you provide your own VNET, extra [managed resources](networking.md#managed-resources) are created which incur billing.

- When you provide your own VNET, you need to provide a subnet that is dedicated to the Container App environment you deploy. This subnet isn't available to other services.
- When you provide your own VNet, you need to provide a subnet that is dedicated to the Container App environment you deploy. This subnet can't be used by other services.

- Network addresses are assigned from a subnet range you define as the environment is created.

- You can define the subnet range used by the Container Apps environment.
- Once the environment is created, the subnet range is immutable.
- Each [revision](revisions.md) is assigned an IP address in the subnet.
- You can restrict inbound requests to the environment exclusively to the VNET by deploying the environment as [internal](vnet-custom-internal.md).
- You can restrict inbound requests to the environment exclusively to the VNet by deploying the environment as [internal](vnet-custom-internal.md).

As you begin to design the network around your container app, refer to [Plan virtual networks](../virtual-network/virtual-network-vnet-plan-design-arm.md) for important concerns surrounding running virtual networks on Azure.

:::image type="content" source="media/networking/azure-container-apps-virtual-network.png" alt-text="Diagram of how Azure Container Apps environments use an existing V NET, or you can provide your own.":::

> [!NOTE]
> Moving VNETs among different resource groups or subscriptions is not supported if the VNET is in use by a Container Apps environment.
> Moving VNets among different resource groups or subscriptions is not supported if the VNet is in use by a Container Apps environment.
<!--
https://learn.microsoft.com/azure/azure-functions/functions-networking-options
Expand Down Expand Up @@ -89,43 +98,80 @@ The following ports are exposed for inbound connections.
|--|--|
| HTTP/HTTPS | 80, 443 |

Container Apps reserves 60 IPs in your VNET, and the amount may grow as your container environment scales.

IP addresses are broken down into the following types:

| Type | Description |
|--|--|
| Public inbound IP address | Used for app traffic in an external deployment, and management traffic in both internal and external deployments. |
| Outbound public IP | Used as the "from" IP for outbound connections that leave the virtual network. These connections aren't routed down a VPN. Using a NAT gateway or other proxy for outbound traffic from a Container App environment isn't supported. Outbound IPs aren't guaranteed and may change over time. |
| Outbound public IP | Used as the "from" IP for outbound connections that leave the virtual network. These connections aren't routed down a VPN. Outbound IPs aren't guaranteed and may change over time. Using a NAT gateway or other proxy for outbound traffic from a Container App environment is only supported on the Consumption + Dedicated plan structure. |
| Internal load balancer IP address | This address only exists in an internal deployment. |
| App-assigned IP-based TLS/SSL addresses | These addresses are only possible with an external deployment, and when IP-based TLS/SSL binding is configured. |

## Subnet Address Range Restrictions

Subnet address ranges can't overlap with the following reserved ranges:
## Subnet

- 169.254.0.0/16
- 172.30.0.0/16
- 172.31.0.0/16
- 192.0.2.0/24
Virtual network integration depends on a dedicated subnet. How IP addresses are allocated in a subnet and what subnet sizes are supported depends on which plan you are using in Azure Container Apps. Selecting an appropriately sized subnet for the scale of your Container Apps is important as subnet sizes can't be modified post creation in Azure.

## Subnet
- Consumption plan:
- /23 is the minimum subnet size required for virtual network integration.
- Container Apps reserves a minimum of 60 IPs for infrastructure in your VNet, and the amount may increase up to 256 addresses as your container environment scales.
- For every replica your container apps are scaled out to, an additional IP address is allocated.

- Consumption and Dedicated Workload Profiles:
- /27 is the minimum subnet size required for virtual network integration.
- The subnet you are integrating your container app with must be delegated to `Microsoft.App/environments`.
- 11 IP addresses are automatically reserved for integration with the subnet. When running on workload profiles, the number of IP addresses required for infrastructure integration doesn't vary based on the scale of your container apps.
- Additional IP addresses are allocated depending on your Container App's workload profile:
- When using Consumption workload profiles for your container app, IP address assignment behaves the same as when running on the Consumption plan without workload profiles. For every replica your container apps are scaled out to, an additional IP address is allocated.
- When using the Dedicated workload profile for your container app, each node will have 1 IP address assigned.

As a Container Apps environment is created, you provide resource IDs for a single subnet.

If you're using the CLI, the parameter to define the subnet resource ID is `infrastructure-subnet-resource-id`. The subnet hosts infrastructure components and user app containers.

### Subnet Address Range Restrictions

Subnet address ranges can't overlap with the following ranges reserved by AKS:
- 169.254.0.0/16
- 172.30.0.0/16
- 172.31.0.0/16
- 192.0.2.0/24

In addition to the above, Container Apps on Consumption and Dedicated workload profiles reserve the following addresses:
- 100.100.0.0/17
- 100.100.128.0/19
- 100.100.160.0/19
- 100.100.192.0/19

If you're using the Azure CLI and the [platformReservedCidr](vnet-custom-internal.md#networking-parameters) range is defined, both subnets must not overlap with the IP range defined in `platformReservedCidr`.


## Routes

There's no forced tunneling in Container Apps routes.
User Defined Routes (UDR) and controlled egress through NAT Gateway are supported in the Consumption + Dedicated plan structure which is in preview. In the Consumption plan structure, these features are not supported.

### User defined routes (UDR) - preview

You can use UDR on the Consumption + Dedicated plan structure to restrict outbound traffic from your container app through Azure Firewall or other network appliances. Configuring UDR is done outside of the Container Apps environment scope.

:::image type="content" source="media/networking/udr-architecture.png" alt-text="Diagram of how UDR is implemented for Container Apps.":::

Important notes for configuring UDR with Azure Firewall:
- You need to allow the `MicrosoftContainerRegistry` and its dependency `AzureFrontDoor.FirstParty` service tags to your Azure Firewall. Alternatively, you can add the following FQDNs: *mcr.microsoft.com* and **.data.mcr.microsoft.com*.
- If you are using Azure Container Registry (ACR), you will also need to add your *ACR address* and either the `AzureContainerRegistry` service tag or the **.blob.core.windows.net* FQDN in the Azure Firewall.
- External environments are not supported.

Azure creates a default route table for your virtual networks upon create. By implementing a user-defined route table, you can control how traffic is routed within your virtual network. For example, you can create a UDR that routes all traffic to the firewall. For a guide on how to setup UDR with Container Apps to restrict outbound traffic with Azure Firewall, visit the [how to for Container Apps and Azure Firewall](./user-defined-routes.md).

### NAT gateway integration - preview

You can use NAT Gateway to simplify outbound connectivity for your outbound internet traffic in your virtual network on the Consumption + Dedicated plan structure. NAT Gateway is used to provide a static public IP address, so when you configure NAT Gateway on your Container Apps subnet, all outbound traffic from your container app will be routed through the NAT Gateway's static public IP address.

## DNS

- **Custom DNS**: If your VNET uses a custom DNS server instead of the default Azure-provided DNS server, configure your DNS server to forward unresolved DNS queries to `168.63.129.16`. [Azure recursive resolvers](../virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances.md#name-resolution-that-uses-your-own-dns-server) uses this IP address to resolve requests. If you don't use the Azure recursive resolvers, the Container Apps environment can't function.
- **Custom DNS**: If your VNet uses a custom DNS server instead of the default Azure-provided DNS server, configure your DNS server to forward unresolved DNS queries to `168.63.129.16`. [Azure recursive resolvers](../virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances.md#name-resolution-that-uses-your-own-dns-server) uses this IP address to resolve requests. If you don't use the Azure recursive resolvers, the Container Apps environment can't function.

- **VNET-scope ingress**: If you plan to use VNET-scope [ingress](ingress-overview.md) in an internal Container Apps environment, configure your domains in one of the following ways:
- **VNet-scope ingress**: If you plan to use VNet-scope [ingress](ingress-overview.md) in an internal Container Apps environment, configure your domains in one of the following ways:

1. **Non-custom domains**: If you don't plan to use custom domains, create a private DNS zone that resolves the Container Apps environment's default domain to the static IP address of the Container Apps environment. You can use [Azure Private DNS](../dns/private-dns-overview.md) or your own DNS server. If you use Azure Private DNS, create a Private DNS Zone named as the Container App Environment’s default domain (`<UNIQUE_IDENTIFIER>.<REGION_NAME>.azurecontainerapps.io`), with an `A` record. The A record contains the name `*<DNS Suffix>` and the static IP address of the Container Apps environment.

Expand All @@ -135,13 +181,12 @@ The static IP address of the Container Apps environment can be found in the Azur

## Managed resources

When you deploy an internal or an external environment into your own network, a new resource group prefixed with `MC_` is created in the Azure subscription where your environment is hosted. This resource group contains infrastructure components managed by the Azure Container Apps platform, and shouldn't be modified. The resource group contains Public IP addresses used specifically for outbound connectivity from your environment and a load balancer. In addition to the [Azure Container Apps billing](./billing.md), you're billed for:
When you deploy an internal or an external environment into your own network, a new resource group prefixed with `MC_` is created in the Azure subscription where your environment is hosted. This resource group contains infrastructure components managed by the Azure Container Apps platform, and shouldn't be modified. The resource group contains Public IP addresses used specifically for outbound connectivity from your environment and a load balancer. The resource group name can be configured during container app environment creation. In addition to the [Azure Container Apps billing](./billing.md), you're billed for:

- Two standard static [public IPs](https://azure.microsoft.com/pricing/details/ip-addresses/), one for ingress and one for egress. If you need more IPs for egress due to SNAT issues, [open a support ticket to request an override](https://azure.microsoft.com/support/create-ticket/).

- Two standard [Load Balancers](https://azure.microsoft.com/pricing/details/load-balancer/) if using an internal environment, or one standard [Load Balancer](https://azure.microsoft.com/pricing/details/load-balancer/) if using an external environment. Each load balancer has fewer than six rules. The cost of data processed (GB) includes both ingress and egress for management operations.


## Next steps

- [Deploy with an external environment](vnet-custom.md)
Expand Down
2 changes: 1 addition & 1 deletion articles/container-apps/plans.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Azure Container Apps features two different plan types.
| Plan type | Description | In Preview |
|--|--|--|
| [Consumption](#consumption-plan) | Serverless environment with support for scale-to-zero and pay only for resources your apps use. | No |
| [Consumption with Dedicated workload profiles](#consumption-dedicated) | Fully managed environment with support for scale-to-zero and pay only for resources your apps use. Optionally, run apps with customized hardware and increased cost predictability using Dedicated workload profiles. | Yes |
| [Consumption and Dedicated plan structures](#consumption-dedicated) | Fully managed environment with support for scale-to-zero and pay only for resources your apps use. Optionally, run apps with customized hardware and increased cost predictability using Dedicated workload profiles. | Yes |

## Consumption plan

Expand Down

0 comments on commit 08ff85c

Please sign in to comment.