diff --git a/docs/architecture.md b/docs/architecture.md index edf2e4ff8a..4d8c90ed0f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -94,39 +94,6 @@ The flow to provision a Workspace is as follows (the flow is the same for all ki 1. The status of a Porter bundle execution is received. 1. The status of a Porter bundle execution is updated in the Configuration Store. -## Network Architecture +## Network architecture -The network topology is based on [hub-spoke](https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/hybrid-networking/hub-spoke). The TRE Management VNET ([Azure Virtual Network](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-overview)) is the central hub and each Workspace are spokes. - -> **Note:** TRE Management is referred to as Core in scripts and code. - -![Network Architecture](./assets/network-architecture.png) - -Azure TRE VNETs are segregated allowing limited traffic between the TRE Management VNET and Workspace VNETs. The rules are managed in the `nsg-ws` Network Security Group (NSG): - -- Inbound traffic from TRE Management VNET to Workspace allowed for [Azure Bastion](https://docs.microsoft.com/en-us/azure/bastion/bastion-overview) (22, 3389) - All other inbound traffic from Core to Workspace denied. -- Outbound traffic to `SharedSubnet` from Workspace allowed. -- Outbound traffic to Internet allowed on HTTPS port 443 (next hop Azure Firewall). -- All other outbound traffic denied. - -> In Azure traffic between subnets are allowed except explicitly denied. - -Each of these rules can be managed per Workspace. - -Each Workspace has a default route routing all egress traffic through the Azure Firewall, to ensure only explicitly allowed destinations on the Internet to be accessed. It is planned that all other subnet will use the same pattern (Issue [#421](https://github.com/microsoft/AzureTRE/issues/421)) - -The Azure Firewall rules are: - -- No default inbound rules – block all. -- No default outbound rules – block all. - -Inbound traffic from the Internet is only allowed through the Application Gateway, which forwards HTTPS (port 443) call to the TRE API in the `WebAppSubnet`. - -| Subnet | Description | -| -------| ----------- | -| `AzureBastionSubnet` | A dedicated subnet for Azure Bastion hosts. | -| `AppGwSubnet` | Subnet for Azure Application Gateway controlling ingress traffic. | -| `AzureFirewallSubnet` | Subnet for Azure Firewall controlling egress traffic. | -| `ResourceProcessorSubnet` | Subnet for VMSS used by the Composition Service to host Docker containers to execute Porter bundles that deploys Workspaces. | -| `WebAppSubnet` | Subnet for TRE API. | -| `SharedSubnet` | Shared Services subnet for all things shared by TRE Management and Workspaces. Future Shared Services are Firewall Shared Service, Source Mirror Shared Service and Package Mirror Shared Service. | +See [networking](./networking.md). diff --git a/docs/index.md b/docs/index.md index 23292f8ae7..64242a75d3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,6 +4,7 @@ * [Concepts](./concepts.md) * [User roles](./user-roles.md) * [Architecture](./architecture.md) + * [Networking](./networking.md) * [Logical data model](./logical-data-model.md) * Getting started * [Dev environment](./dev-environment.md) diff --git a/docs/networking.md b/docs/networking.md new file mode 100644 index 0000000000..1ec0780582 --- /dev/null +++ b/docs/networking.md @@ -0,0 +1,52 @@ +# Networking + +The Trusted Research Environment (TRE) network topology is based on [hub-spoke](https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/hybrid-networking/hub-spoke). The TRE Management VNET ([Azure Virtual Network](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-overview)) is the central hub and each workspace is a spoke. + +> Note: TRE Management is referred to as **core** in scripts and code. + +![Network architecture](./assets/network-architecture.png) + +Azure TRE VNETs are segregated allowing limited traffic between the TRE Management VNET and Workspace VNETs. The security rules are managed by `nsg-ws` network security group. See [workspace network security groups (NSG)](#workspaces) further down. + +Each workspace has a default route routing all egress traffic through the Azure Firewall, to ensure only explicitly allowed destinations on the Internet to be accessed. It is planned that all other subnet will use the same pattern (Issue [#421](https://github.com/microsoft/AzureTRE/issues/421)) + +The Azure Firewall rules are: + +- No default inbound rules – block all. +- No default outbound rules – block all. + +Inbound traffic from the Internet is only allowed through the Application Gateway, which forwards HTTPS (port 443) call to the TRE API in the `WebAppSubnet`. + +| Subnet | Description | +| -------| ----------- | +| `AzureBastionSubnet` | A dedicated subnet for Azure Bastion hosts. | +| `AppGwSubnet` | Subnet for Azure Application Gateway controlling ingress traffic. | +| `AzureFirewallSubnet` | Subnet for Azure Firewall controlling egress traffic. | +| `ResourceProcessorSubnet` | Subnet for VMSS used by the Composition Service to host Docker containers to execute Porter bundles that deploys Workspaces. | +| `WebAppSubnet` | Subnet for TRE API. | +| `SharedSubnet` | Shared Services subnet for all things shared by TRE Management and Workspaces. Future Shared Services are Firewall Shared Service, Source Mirror Shared Service and Package Mirror Shared Service. | + +## Network security groups + +### TRE Management/core + +Network security groups (NSG) and their security rules for TRE core resources are defined in [`/templates/core/terraform/network/network_security_groups.tf`](../templates/core/terraform/network/network_security_groups.tf). + +| Network security group | Associated subnet(s) | +| ---------------------- | -------------------- | +| `nsg-bastion-subnet` | `AzureBastionSubnet` | +| `nsg-app-gw` | `AppGwSubnet` | +| `nsg-default-rules` | `ResourceProcessorSubnet`, `SharedSubnet`, `WebAppSubnet` | + +### Workspaces + +Azure TRE VNETs are segregated allowing limited traffic between the TRE Management VNET and Workspace VNETs. The rules to manage and limit the traffic between the TRE Management VNET and Workspace VNETs are defined by the `nsg-ws` network security group: + +- Inbound traffic from TRE Management VNET to workspace allowed for [Azure Bastion](https://docs.microsoft.com/en-us/azure/bastion/bastion-overview) (22, 3389) - All other inbound traffic from Core to workspace denied. +- Outbound traffic to `SharedSubnet` from Workspace allowed. +- Outbound traffic to Internet allowed on HTTPS port 443 (next hop Azure Firewall). +- All other outbound traffic denied. + +> In Azure, traffic between subnets are allowed except explicitly denied. + +Each of these rules can be managed per workspace. diff --git a/templates/core/terraform/main.tf b/templates/core/terraform/main.tf index 8f90b6ae16..e6762c5369 100644 --- a/templates/core/terraform/main.tf +++ b/templates/core/terraform/main.tf @@ -63,8 +63,8 @@ module "storage" { tre_id = var.tre_id location = var.location resource_group_name = azurerm_resource_group.core.name - shared_subnet = module.network.shared - core_vnet = module.network.core + shared_subnet = module.network.shared_subnet_id + core_vnet = module.network.core_vnet_id } module "appgateway" { @@ -72,8 +72,8 @@ module "appgateway" { tre_id = var.tre_id location = var.location resource_group_name = azurerm_resource_group.core.name - app_gw_subnet = module.network.app_gw - shared_subnet = module.network.shared + app_gw_subnet = module.network.app_gw_subnet_id + shared_subnet = module.network.shared_subnet_id api_fqdn = module.api-webapp.api_fqdn keyvault_id = module.keyvault.keyvault_id static_web_dns_zone_id = module.network.static_web_dns_zone_id @@ -85,10 +85,10 @@ module "api-webapp" { tre_id = var.tre_id location = var.location resource_group_name = azurerm_resource_group.core.name - web_app_subnet = module.network.web_app - shared_subnet = module.network.shared - app_gw_subnet = module.network.app_gw - core_vnet = module.network.core + web_app_subnet = module.network.web_app_subnet_id + shared_subnet = module.network.shared_subnet_id + app_gw_subnet = module.network.app_gw_subnet_id + core_vnet = module.network.core_vnet_id app_insights_connection_string = azurerm_application_insights.core.connection_string app_insights_instrumentation_key = azurerm_application_insights.core.instrumentation_key log_analytics_workspace_id = azurerm_log_analytics_workspace.core.id @@ -125,7 +125,7 @@ module "resource_processor_vmss_porter" { resource_group_name = azurerm_resource_group.core.name acr_id = data.azurerm_container_registry.mgmt_acr.id app_insights_connection_string = azurerm_application_insights.core.connection_string - resource_processor_subnet_id = module.network.resource_processor + resource_processor_subnet_id = module.network.resource_processor_subnet_id docker_registry_server = var.docker_registry_server resource_processor_vmss_porter_image_repository = var.resource_processor_vmss_porter_image_repository resource_processor_vmss_porter_image_tag = var.resource_processor_vmss_porter_image_tag @@ -148,8 +148,8 @@ module "servicebus" { tre_id = var.tre_id location = var.location resource_group_name = azurerm_resource_group.core.name - core_vnet = module.network.core - resource_processor_subnet_id = module.network.resource_processor + core_vnet = module.network.core_vnet_id + resource_processor_subnet_id = module.network.resource_processor_subnet_id } module "keyvault" { @@ -157,8 +157,8 @@ module "keyvault" { tre_id = var.tre_id location = var.location resource_group_name = azurerm_resource_group.core.name - shared_subnet = module.network.shared - core_vnet = module.network.core + shared_subnet = module.network.shared_subnet_id + core_vnet = module.network.core_vnet_id tenant_id = data.azurerm_client_config.current.tenant_id managed_identity_tenant_id = module.identity.managed_identity.tenant_id managed_identity_object_id = module.identity.managed_identity.principal_id @@ -186,8 +186,8 @@ module "routetable" { tre_id = var.tre_id location = var.location resource_group_name = azurerm_resource_group.core.name - shared_subnet_id = module.network.shared - resource_processor_subnet_id = module.network.resource_processor + shared_subnet_id = module.network.shared_subnet_id + resource_processor_subnet_id = module.network.resource_processor_subnet_id firewall_private_ip_address = module.firewall.firewall_private_ip_address } @@ -196,8 +196,8 @@ module "state-store" { tre_id = var.tre_id location = var.location resource_group_name = azurerm_resource_group.core.name - shared_subnet = module.network.shared - core_vnet = module.network.core + shared_subnet = module.network.shared_subnet_id + core_vnet = module.network.core_vnet_id } module "bastion" { @@ -205,7 +205,7 @@ module "bastion" { tre_id = var.tre_id location = var.location resource_group_name = azurerm_resource_group.core.name - bastion_subnet = module.network.bastion + bastion_subnet = module.network.bastion_subnet_id } module "gitea" { diff --git a/templates/core/terraform/network/network.tf b/templates/core/terraform/network/network.tf index 4151720edd..75e74e4283 100644 --- a/templates/core/terraform/network/network.tf +++ b/templates/core/terraform/network/network.tf @@ -48,41 +48,6 @@ resource "azurerm_subnet" "web_app" { } } -resource "azurerm_subnet" "aci" { - name = "AciSubnet" - virtual_network_name = azurerm_virtual_network.core.name - resource_group_name = var.resource_group_name - address_prefixes = [local.aci_subnet_address_prefix] - enforce_private_link_endpoint_network_policies = true - enforce_private_link_service_network_policies = true - - delegation { - name = "acidelegationservice" - - service_delegation { - name = "Microsoft.ContainerInstance/containerGroups" - actions = ["Microsoft.Network/virtualNetworks/subnets/action"] - } - } -} - -resource "azurerm_network_profile" "aciprofile" { - name = "aciprofile" - location = var.location - resource_group_name = var.resource_group_name - - container_network_interface { - name = "acr-frontal-nic" - - ip_configuration { - name = "acrfrontal" - subnet_id = azurerm_subnet.aci.id - } - } - - lifecycle { ignore_changes = [tags] } -} - resource "azurerm_subnet" "shared" { name = "SharedSubnet" virtual_network_name = azurerm_virtual_network.core.name diff --git a/templates/core/terraform/network/network_security_groups.tf b/templates/core/terraform/network/network_security_groups.tf new file mode 100644 index 0000000000..2473e1f6f3 --- /dev/null +++ b/templates/core/terraform/network/network_security_groups.tf @@ -0,0 +1,168 @@ +# Network security group for Azure Bastion subnet +# See https://docs.microsoft.com/azure/bastion/bastion-nsg +resource "azurerm_network_security_group" "bastion" { + name = "nsg-bastion-subnet" + location = var.location + resource_group_name = var.resource_group_name + + security_rule { + name = "AllowInboundInternet" + priority = 4000 + direction = "Inbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "443" + source_address_prefix = "Internet" + destination_address_prefix = "*" + } + + security_rule { + name = "AllowInboundGatewayManager" + priority = 4001 + direction = "Inbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "443" + source_address_prefix = "GatewayManager" + destination_address_prefix = "*" + } + + security_rule { + name = "AllowInboundAzureLoadBalancer" + priority = 4002 + direction = "Inbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "443" + source_address_prefix = "AzureLoadBalancer" + destination_address_prefix = "*" + } + + security_rule { + name = "AllowInboundHostCommunication" + priority = 4003 + direction = "Inbound" + access = "Allow" + protocol = "*" + source_port_range = "*" + destination_port_ranges = ["5701", "8080"] + source_address_prefix = "VirtualNetwork" + destination_address_prefix = "VirtualNetwork" + } + + security_rule { + name = "AllowOutboundSshRdp" + priority = 4020 + direction = "Outbound" + access = "Allow" + protocol = "*" + source_port_range = "*" + destination_port_ranges = ["22", "3389"] + source_address_prefix = "*" + destination_address_prefix = "VirtualNetwork" + } + + security_rule { + name = "AllowOutboundAzureCloud" + priority = 4021 + direction = "Outbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "443" + source_address_prefix = "*" + destination_address_prefix = "AzureCloud" + } + + security_rule { + name = "AllowOutboundHostCommunication" + priority = 4022 + direction = "Outbound" + access = "Allow" + protocol = "*" + source_port_range = "*" + destination_port_ranges = ["5701", "8080"] + source_address_prefix = "VirtualNetwork" + destination_address_prefix = "VirtualNetwork" + } + + security_rule { + name = "AllowOutboundGetSessionInformation" + priority = 4023 + direction = "Outbound" + access = "Allow" + protocol = "*" + source_port_range = "*" + destination_port_range = "80" + source_address_prefix = "*" + destination_address_prefix = "Internet" + } +} + +resource "azurerm_subnet_network_security_group_association" "bastion" { + subnet_id = azurerm_subnet.bastion.id + network_security_group_id = azurerm_network_security_group.bastion.id +} + +# Network security group for Application Gateway +# See https://docs.microsoft.com/azure/application-gateway/configuration-infrastructure#network-security-groups +resource "azurerm_network_security_group" "app_gw" { + name = "nsg-app-gw" + location = var.location + resource_group_name = var.resource_group_name + + security_rule { + name = "AllowInboundGatewayManager" + priority = 3800 + direction = "Inbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "65200-65535" + source_address_prefix = "GatewayManager" + destination_address_prefix = "*" + } + + security_rule { + name = "AllowInboundInternet" + priority = 3801 + direction = "Inbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "443" + source_address_prefix = "Internet" + destination_address_prefix = "*" + } +} + +resource "azurerm_subnet_network_security_group_association" "app_gw" { + subnet_id = azurerm_subnet.app_gw.id + network_security_group_id = azurerm_network_security_group.app_gw.id +} + +# Network security group with only default security rules +# See https://docs.microsoft.com/azure/virtual-network/network-security-groups-overview#default-security-rules +resource "azurerm_network_security_group" "default_rules" { + name = "nsg-default-rules" + location = var.location + resource_group_name = var.resource_group_name +} + +resource "azurerm_subnet_network_security_group_association" "shared" { + subnet_id = azurerm_subnet.shared.id + network_security_group_id = azurerm_network_security_group.default_rules.id +} + +resource "azurerm_subnet_network_security_group_association" "web_app" { + subnet_id = azurerm_subnet.web_app.id + network_security_group_id = azurerm_network_security_group.default_rules.id +} + +resource "azurerm_subnet_network_security_group_association" "resource_processor" { + subnet_id = azurerm_subnet.resource_processor.id + network_security_group_id = azurerm_network_security_group.default_rules.id +} diff --git a/templates/core/terraform/network/output.tf b/templates/core/terraform/network/output.tf index 4668830427..edd64a5f34 100644 --- a/templates/core/terraform/network/output.tf +++ b/templates/core/terraform/network/output.tf @@ -1,32 +1,28 @@ -output "core" { +output "core_vnet_id" { value = azurerm_virtual_network.core.id } -output "bastion" { +output "bastion_subnet_id" { value = azurerm_subnet.bastion.id } -output "azure_firewall" { +output "azure_firewall_subnet_id" { value = azurerm_subnet.azure_firewall.id } -output "app_gw" { +output "app_gw_subnet_id" { value = azurerm_subnet.app_gw.id } -output "web_app" { +output "web_app_subnet_id" { value = azurerm_subnet.web_app.id } -output "shared" { +output "shared_subnet_id" { value = azurerm_subnet.shared.id } -output "aci" { - value = azurerm_subnet.aci.id -} - -output "resource_processor" { +output "resource_processor_subnet_id" { value = azurerm_subnet.resource_processor.id } @@ -36,4 +32,4 @@ output "azurewebsites_dns_zone_id" { output "static_web_dns_zone_id" { value = azurerm_private_dns_zone.static_web.id -} \ No newline at end of file +}