Introduction note: This guide aims at providing a better understanding of the Azure routing mechanisms and how they translate from On-Prem networking. The focus will be on private routing in Hub & Spoke topologies. For clarity, network security and resiliency best practices as well as internet breakout considerations have been left out of this guide.
3.1. NVA default routing configuration & packet walk
3.1.1. NVA Effective routes & NVA routing table alignment
3.2. On-Prem connectivity via an NVA
3.2.1. NVA routing table & reachability between the NVA and the branches
3.2.2. Azure VM Effective routes and NVA routing table misalignment
3.2.3. Solution: Align the data-plane (VM Effective routes) to the control-plane (NVA routing table)
An Azure VM running a non-native 3rd Party image with networking capabilities is referred to as an NVA (Network Virtual Appliance). An NVA could be a firewall, a router, a proxy, an SDWAN hub, an IPSec concentrator etc.
To illustrate the NVA use case, a Cisco CSR is deployed in a new subnet of the hub VNET ("NVAsubnet": 10.0.10.0/24), and terminates connectivity to On-Prem branches (could be SDWAN or IPSec).
This scenario represents an alternative to the On-Prem connectivity via the Azure native Virtual Network GW discussed in Episode #1 and Episode #2.
Let's start without any branches connected to the Concentrator NVA and observe its default configuration and behaviour.
When freshly deployed, the CSR comes with a default configuration, inherited from the VM on which it is hosted.
Interface GigabitEthernet1 of the router has retrieved the IP address of the NVA’s NIC:
Extract of the routing table (OS-level):
As expected from a router, it contains the directly connected route of its GigabitEthernet1 interface (10.0.10.0/24).
And since by default all resources in a VNET can communicate outbound to the internet, the 0.0.0.0/0 route of the NIC Effective routes is reflected in the Concentrator routing table, pointing towards the default GW of the subnet (10.0.10.1 *).
* In Azure, 5 IPs are reserved per subnet: 1 network + 1 broadcast + 3 Azure reserved IPs (1 subnet default gateway + 2 DNS).
168.63.129.16 and 169.254.169.254 are also part of the default configuration of any device in Azure and are used for the communication to Azure management platform resources. These IPs are not relevant in the context of this article.
Because the router’s OS route processing differs from standard VNET routing, there isn’t an exact 1:1 match between the entries of the NVA routing table (containing the local subnet + a default route) and the (underlying) VM’s NIC Effective routes (VNET ranges + default route).
The data plane reachability between the different VNETs is validated with pings:
Let’s use the example of the ping from the Concentrator NVA (10.0.10.4) to Spoke1VM (10.1.1.4 in the Spoke1 peered VNET).
- Packets are first processed by the router OS. Because in the CSR routing table there isn’t any match for the Spoke1VM subnet (10.1.1.0/24) or for the Spoke1 VNET range (10.1.0.0/16), packets will be caught by the default route pointing towards 10.0.10.1 (the VNET default GW).
- As 10.0.10.1 belongs to the 10.0.10.0/24 subnet for which an outbound interface is known in the routing table (GigabitEthernet1), the usual recursive routing will result in the packets being forwarded out the GigabitEthernet1 interface.
In traditional networking, packets would be processed through an SFP (inserted in a physical port labelled “GE1”) and according to the L3 routing decision made by the router.
- ➡️ In Azure, after the routing at NVA OS level has determined to which (logical) interface to forward traffic to (here GigabitEthernet1), packets reach the underlying associated physical NIC, where they are routed based on the NIC Effective routes and the packets destination IP, and are finally sent to the wire accordingly (here the destination matches the 10.1.0.0/16 entry know via VNET peering).
Understanding this additional step and most importantly the need of alignment between a VM’s Effective routes and the routing table sitting on top of the VM has been a key learning for me.
We are now back to our initial setup, with On-Prem branches connected to the Concentrator NVA.
In real-life scenarios the On-Prem prefixes would likely be learnt dynamically by the Concentrator, via BGP over IPSec tunnels or via an SDWAN overlay for example, and the Concentrator would in return advertise the Azure IP ranges to the branches.
Whether IPSec or SDWAN, the solution run between the Concentrator and its branches is not relevant to understand the Azure routing principles. Loopback addresses have been configured on the CSR for branch emulation.
Connectivity between the Concentrator NVA and the On-Prem branches is confirmed by the Concentrator routing table and successful pings:
The static 10/8 supernet route covering the Azure environment and pointing to the subnet default gateway (10.0.10.1) is configured on the Concentrator NVA to be further advertised to the branches.
Although existing in the Concentrator NVA routing table, the branch prefixes are NOT reflected on the underlying NVA’s NIC Effective routes nor known or reachable by any other VM in the VNET or peered VNETs, resulting in failed connectivity to the On-Prem branches. Step 3 of the above packet walk cannot be completed.
To enable end-to-end connectivity, the NICs of the VMs must also know about the branch prefixes, having the information at the NVA OS level is not enough.
This is achieved by 1) creating an Azure Route table (or updating an existing one), 2) configuring a static entry (a UDR) in this Route table for the branch prefixes (the 192.168.0.0/16 supernet is enough) with the NVA as Next-Hop (10.0.10.4), and 3) associating the Route table with any subnet requiring connectivity to these branches. Steps 2 and 3 are represented below:
The Concentrator NVA doesn't need the On-Prem prefixes in its Effective routes.
➡️ Effective routes are only checked for packets leaving the VMs to identify the Next-Hop to forward traffic to. It is IP forwarding that will make an NVA process traffic reaching its NIC.
All the subnets that should know about the On-Prem prefixes must be explicitly associated to this Route table, containing the /16 static route to the Concentrator NVA. For this scenario we have associated all the subnets of Spoke1 and Spoke2 VNETs and the Hub Test subnet to a Route table named "SpokeRT":
The impact on the VMs Effective routes of associating these Route tables to the Spoke subnets is represented on the updated diagram below:
➡️ The branch prefixes are not propagated by the Concentrator NVA in its VNET and peered VNETs like a Virtual Network Gateway would do. Connectivity is achieved via UDRs pointing to the Concentrator and configured on every target subnet.
➡️ The Gateway Transit and Gateway route propagation settings only apply to native Azure gateways (Expressroute or VPN Virtual Network Gateways, or Azure Route Servers as we will see in Episode #5) and as a result are not relevant in the context of the current use case (UDRs + non-native Concentrator NVA).