Skip to content

update and refine the fully converged design after team discussion #249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
liunick-msft wants to merge 1 commit into
base: main
Choose a base branch
from
Open

update and refine the fully converged design after team discussion #249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...or/Networking/Troubleshoot-Network-Test-StorageConnections-ConnectivityCheck.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ In converged deployments, the Storage Connections validator will create a tempor

4. If any ping fails, check the following:

- That the VLANs are correctly configured on the TOR switches. In a converged deployment, both storage VLANs should be configured on the interface.
- That the VLANs are correctly configured on the TOR switches. In a converged deployment, each storage VLAN should be configured on its respective ToR switch (Storage VLAN A on ToR-A, Storage VLAN B on ToR-B).
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What: This bullet mixes terminology: it refers to “TOR switches” but then says “respective ToR switch”.
Why: Inconsistent terminology within the same sentence is confusing, especially for readers unfamiliar with the acronym.
How: Use one form consistently in this document (either “TOR” everywhere or “ToR” everywhere).

Copilot uses AI. Check for mistakes.
- That physical NICs are connected to the correct ports on the TOR switches.
- That no VLANs are configured on the physical NICs.
- That no firewall rules or other configuration are blocking APIPA traffic.
Expand Down
31 changes: 12 additions & 19 deletions TSG/Networking/Top-Of-Rack-Switch/Overview-Azure-Local-Deployment-Pattern.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ A high-performance design utilizing dedicated NICs for management/compute and st
![Switched with 2 ToRs](images/AzureLocalPhysicalNetworkDiagram_Switched.png)

**Fully Converged Deployment**
A balanced design where all traffic types (management, compute, storage) share the same physical NICs through VLAN segmentation. This pattern minimizes hardware footprint while maintaining high scalability. **Both storage VLANs must be configured on both ToR switches** because SET (Switch Embedded Teaming) may route either storage VLAN through either physical NIC.
A balanced design where all traffic types (management, compute, storage) share the same physical NICs through VLAN segmentation. This pattern minimizes hardware footprint while maintaining high scalability. The **recommended** configuration uses **one storage VLAN per ToR switch**: Storage VLAN A on ToR-A (mapped to one physical NIC) and Storage VLAN B on ToR-B (mapped to the other physical NIC). In failure scenarios (NIC or ToR), SMB/RDMA traffic automatically fails over to the remaining path.

![Fully-Converged with 2 ToRs](images/AzureLocalPhysicalNetworkDiagram_FullyConverged.png)

Expand All @@ -83,10 +83,10 @@ A balanced design where all traffic types (management, compute, storage) share t
|---------------------|------------------------|-------------------------------|-------------------|
| **Switchless** | 2 NICs to switches (M+C traffic) + (N−1) direct inter-node NICs (S traffic) | Trunk ports with M, C VLANs only; no storage VLANs on ToRs | Edge deployments, remote sites, cost-sensitive environments |
| **Switched** | 4 NICs per host: 2 for M+C traffic, 2 dedicated for storage | M and C VLANs on both ToRs; S1 VLAN on ToR1 only, S2 VLAN on ToR2 only (dedicated storage NICs) | Enterprise deployments requiring dedicated storage performance and traffic isolation |
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What: The table uses ToR1/ToR2 terminology for Switched deployments but ToR-A/ToR-B for Fully Converged.
Why: Inconsistent ToR naming in the same comparison table makes it harder to map guidance to the physical switches.
How: Standardize on a single convention (e.g., ToR-A/ToR-B everywhere, or ToR1/ToR2 everywhere).

Suggested change
| **Switched** | 4 NICs per host: 2 for M+C traffic, 2 dedicated for storage | M and C VLANs on both ToRs; S1 VLAN on ToR1 only, S2 VLAN on ToR2 only (dedicated storage NICs) | Enterprise deployments requiring dedicated storage performance and traffic isolation |
| **Switched** | 4 NICs per host: 2 for M+C traffic, 2 dedicated for storage | M and C VLANs on both ToRs; S1 VLAN on ToR-A only, S2 VLAN on ToR-B only (dedicated storage NICs) | Enterprise deployments requiring dedicated storage performance and traffic isolation |

Copilot uses AI. Check for mistakes.
| **Fully Converged** | 2 NICs per host carrying all traffic types (M+C+S) via VLAN segmentation | Both storage VLANs (S1, S2) on both ToRs (required for SET) | General-purpose deployments balancing performance, simplicity, and hardware efficiency |
| **Fully Converged** | 2 NICs per host carrying all traffic types (M+C+S) via VLAN segmentation | S1 VLAN on ToR-A only, S2 VLAN on ToR-B only (recommended) | General-purpose deployments balancing performance, simplicity, and hardware efficiency |

> [!NOTE]
> **Storage VLAN Configuration**: Storage VLANs can be configured as either **Layer 3 (L3) networks with IP subnets** or **Layer 2 (L2) networks without IP subnets**. **Layer 2 configuration is recommended** because it simplifies VLAN tagging, allowing Azure Local hosts to use any IP addresses without hardcoding subnet configurations on the switch or requiring predefined IP ranges. Since Azure Local nodes handle storage traffic tagging, ensure these VLANs are configured as **tagged VLANs on trunk ports** across all ToR switches.
> **Storage VLAN Configuration**: Storage VLANs can be configured as either **Layer 3 (L3) networks with IP subnets** or **Layer 2 (L2) networks without IP subnets**. **Layer 2 configuration is recommended** because it simplifies VLAN tagging, allowing Azure Local hosts to use any IP addresses without hardcoding subnet configurations on the switch or requiring predefined IP ranges. Since Azure Local nodes handle storage traffic tagging, ensure these VLANs are configured as **tagged VLANs on trunk ports** on their respective ToR switches.
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What: This note now says storage VLANs should be configured only on their respective ToR switches, but later in the same document there is still guidance stating storage VLANs should be tagged on trunk ports “across all ToR switches”.
Why: Conflicting instructions in one document can lead to incorrect switch configs.
How: Update the later note to match this new recommended pattern (or clarify when/why the ‘across all ToRs’ option applies).

Suggested change
> **Storage VLAN Configuration**: Storage VLANs can be configured as either **Layer 3 (L3) networks with IP subnets** or **Layer 2 (L2) networks without IP subnets**. **Layer 2 configuration is recommended** because it simplifies VLAN tagging, allowing Azure Local hosts to use any IP addresses without hardcoding subnet configurations on the switch or requiring predefined IP ranges. Since Azure Local nodes handle storage traffic tagging, ensure these VLANs are configured as **tagged VLANs on trunk ports** on their respective ToR switches.
> **Storage VLAN Configuration**: Storage VLANs can be configured as either **Layer 3 (L3) networks with IP subnets** or **Layer 2 (L2) networks without IP subnets**. **Layer 2 configuration is recommended** because it simplifies VLAN tagging, allowing Azure Local hosts to use any IP addresses without hardcoding subnet configurations on the switch or requiring predefined IP ranges. For the recommended deployment patterns in this document, storage VLANs must be configured as **tagged VLANs on trunk ports only on their respective ToR switches**, and **must not be tagged across all ToR switches** unless you are intentionally implementing a non-recommended, legacy, or special-case design that explicitly requires global storage VLAN reachability.

Copilot uses AI. Check for mistakes.


---
Expand Down Expand Up @@ -131,27 +131,20 @@ This tool is designed to automate the generation of Azure Local switch configura
### Q: How should Storage VLANs be configured across ToR switches?

**A:**
Storage VLAN configuration depends on the **deployment pattern**:
The recommended baseline design uses **one storage VLAN per ToR switch** for both Switched and Fully Converged deployments:

| Deployment Pattern | ToR VLAN Configuration | Why |
|-------------------|------------------------|-----|
| **Switched** | S1 on ToR1 only, S2 on ToR2 only | Dedicated storage NICs connect to specific ToRs |
| **Fully Converged** | Both S1 & S2 on both ToRs | SET may route either storage VLAN through either physical NIC |
| **Switched** | S1 on ToR-A only, S2 on ToR-B only | Dedicated storage NICs connect to specific ToRs |
| **Fully Converged** | S1 on ToR-A only, S2 on ToR-B only | Each storage VLAN is mapped to one physical NIC; failover occurs automatically |

**Switched Deployment (One Storage VLAN per ToR):**
- Each host has **dedicated storage NICs** (4 NICs total)
- Storage NIC1 connects to ToR1 → only needs VLAN 711
- Storage NIC2 connects to ToR2 → only needs VLAN 712
- This reduces MC-LAG utilization and optimizes RDMA performance
**Storage VLAN Configuration:**
- Storage VLAN A is configured only on ToR-A and mapped to one physical NIC
- Storage VLAN B is configured only on ToR-B and mapped to the other physical NIC
- In failure scenarios (NIC or ToR failure), SMB/RDMA traffic automatically fails over to the remaining path with reduced bandwidth but no functional impact

**Fully Converged Deployment (Both Storage VLANs on Both ToRs):**
- Each host has only **2 NICs** shared for all traffic
- SET (Switch Embedded Teaming) handles vNIC-to-pNIC mapping
- SET may route either storage VLAN through either physical NIC
- **Both ToRs must carry both storage VLANs** to support SET's flexibility

> [!IMPORTANT]
> In Fully Converged deployments, configuring only one storage VLAN per ToR will cause connectivity issues when SET routes a storage vNIC to a physical NIC connected to a ToR that doesn't have that VLAN configured.
> [!NOTE]
> Configuring both storage VLANs on both ToR switches is also supported but optional. Testing has confirmed there is no meaningful resiliency or failover benefit from this configuration, and it increases complexity without improving availability.


### Q: Are **DCB (Data Center Bridging)** features like **PFC** and **ETS** required for RDMA in Azure Local deployments?
Expand Down
50 changes: 23 additions & 27 deletions TSG/Networking/Top-Of-Rack-Switch/Reference-TOR-Fully-Converged-Storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ This document provides a comprehensive reference for implementing a fully conver
- [Quality of Service (QoS)](#quality-of-service-qos)
- [BGP Routing](#bgp-routing)
- [Frequently Asked Questions](#frequently-asked-questions)
- [Q: Why must both Storage VLANs be on both ToR switches in Fully Converged?](#q-why-must-both-storage-vlans-be-on-both-tor-switches-in-fully-converged)
- [Q: How should Storage VLANs be configured in Fully Converged deployments?](#q-how-should-storage-vlans-be-configured-in-fully-converged-deployments)
- [Additional Resources](#additional-resources)
- [Official Documentation](#official-documentation)
- [Technical Deep Dives](#technical-deep-dives)
Expand All @@ -44,7 +44,7 @@ Azure Local's fully converged network design provides a unified approach to hand

The fully converged physical network architecture integrates **management**, **compute**, and **storage** traffic over the same physical Ethernet interfaces. This design minimizes hardware footprint while maximizing scalability and deployment simplicity.

**Key Design Principle**: In Fully Converged deployments, **both storage VLANs must be configured on both ToR switches**. This is because each host has only 2 NICs (shared for all traffic), and SET (Switch Embedded Teaming) may route either storage VLAN through either physical NIC based on its load balancing algorithm.
**Key Design Principle**: In Fully Converged deployments, the **recommended** baseline design uses **one storage VLAN per ToR switch**: Storage VLAN A is configured only on ToR-A and mapped to one physical NIC, while Storage VLAN B is configured only on ToR-B and mapped to the other physical NIC. In failure scenarios (NIC or ToR), SMB/RDMA traffic automatically fails over to the remaining path with reduced bandwidth but no functional impact. Configuring both storage VLANs on both ToR switches is also supported but optional.

## Architecture Components

Expand Down Expand Up @@ -132,14 +132,14 @@ The fully converged design uses VLAN segmentation to isolate different traffic t
| Storage 1 | SMB storage over RDMA (first path) | 711 | Tagged VLAN, L2 only (no SVI) |
| Storage 2 | SMB storage over RDMA (second path) | 712 | Tagged VLAN, L2 only (no SVI) |

> [!IMPORTANT]
> **Storage VLAN Design Pattern for Fully Converged**: In Fully Converged deployments, **both storage VLANs (711 and 712) must be configured on both ToR switches**. This is because:
> [!NOTE]
> **Storage VLAN Design Pattern for Fully Converged**: The **recommended** baseline design uses **one storage VLAN per ToR switch**:
>
> - Each host has only **2 NICs** connecting to both ToRs (no dedicated storage NICs)
> - **SET (Switch Embedded Teaming)** handles vNIC-to-pNIC mapping at the host level
> - SET may route either storage VLAN through either physical NIC based on its load balancing algorithm
> - Storage VLAN 711 is configured only on ToR-A and mapped to one physical NIC
> - Storage VLAN 712 is configured only on ToR-B and mapped to the other physical NIC
> - In failure scenarios (NIC or ToR), SMB/RDMA traffic automatically fails over to the remaining path
>
> This differs from **Switched** deployments where dedicated storage NICs connect to specific ToRs, allowing one storage VLAN per ToR.
> Configuring both storage VLANs on both ToR switches is also supported but optional. Testing has confirmed no meaningful resiliency benefit from this configuration.

### Top-of-Rack Switch Configuration

Expand Down Expand Up @@ -168,7 +168,7 @@ This section provides configuration guidance using **Cisco Nexus 93180YC-FX3 (NX
- **VLAN 712 (Storage 2)**: Layer 2 only VLAN (no SVI), tagged on trunk ports for RDMA traffic

> [!NOTE]
> In Fully Converged deployments, **both storage VLANs must be configured on both ToR switches** because SET handles vNIC-to-pNIC mapping at the host level and may route either storage VLAN through either physical NIC.
> In Fully Converged deployments, the recommended design uses **one storage VLAN per ToR switch**: Storage VLAN 711 on ToR-A only, Storage VLAN 712 on ToR-B only. This simplifies configuration while automatic failover handles NIC or ToR failures.

> [!IMPORTANT]
> Storage VLANs 711 and 712 should **NOT** be permitted on the ToR-to-ToR peer-link (vPC peer-link, MLAG inter-switch trunk, or any L2 interconnect between ToR switches). Storage traffic must flow directly from host to ToR to destination host to maintain optimal RDMA performance. Allowing storage VLANs on peer links can cause performance degradation.
Expand Down Expand Up @@ -213,7 +213,7 @@ interface Ethernet1/1-3
switchport
switchport mode trunk
switchport trunk native vlan 7
switchport trunk allowed vlan 7,201,711,712
switchport trunk allowed vlan 7,201,711
Comment on lines 213 to +216
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What: The ToR1 sample config still defines Storage VLAN 712 even though the updated recommended pattern says ToR-A/ToR1 should carry only VLAN 711.
Why: Including VLAN 712 in the ToR1 VLAN database contradicts the “one storage VLAN per ToR” guidance and can confuse readers into thinking both storage VLANs must exist on both switches.
How: Remove VLAN 712 from the ToR1 configuration block (and keep the trunk allowed list aligned with the VLANs actually present on that ToR).

Copilot uses AI. Check for mistakes.
priority-flow-control mode on send-tlv
spanning-tree port type edge trunk
mtu 9216
Expand All @@ -227,8 +227,6 @@ vlan 7
name Management_7
vlan 201
name Compute_201
vlan 711
name Storage_711
vlan 712
name Storage_712

Expand All @@ -253,7 +251,7 @@ interface Ethernet1/1-3
switchport
switchport mode trunk
switchport trunk native vlan 7
switchport trunk allowed vlan 7,201,711,712
switchport trunk allowed vlan 7,201,712
priority-flow-control mode on send-tlv
spanning-tree port type edge trunk
mtu 9216
Expand All @@ -262,8 +260,8 @@ interface Ethernet1/1-3
```

> [!NOTE]
> - Both ToR switches have **identical VLAN configurations** (7, 201, 711, 712) in Fully Converged deployments
> - SET at the host level handles vNIC-to-pNIC mapping to optimize storage traffic paths
> - ToR-A has Storage VLAN 711 only, ToR-B has Storage VLAN 712 only (one storage VLAN per ToR)
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What: This note uses ToR-A/ToR-B terminology, but the surrounding sample configuration headings are labeled ToR1/ToR2.
Why: Mixing naming schemes in the same section makes it ambiguous which config block corresponds to “ToR-A” vs “ToR-B”.
How: Use a single naming convention throughout (either rename the config headings to ToR-A/ToR-B, or adjust the note to refer to ToR1/ToR2 consistently).

Suggested change
> - ToR-A has Storage VLAN 711 only, ToR-B has Storage VLAN 712 only (one storage VLAN per ToR)
> - ToR1 has Storage VLAN 711 only, ToR2 has Storage VLAN 712 only (one storage VLAN per ToR)

Copilot uses AI. Check for mistakes.
> - In failure scenarios, SMB/RDMA traffic automatically fails over to the remaining path
> - QoS policies and routing design (e.g., uplinks, BGP/OSPF, default gateway) will be introduced in a separate document


Expand Down Expand Up @@ -326,7 +324,7 @@ Host4 c$ Administrator Administrator 3.1.1 2

> [!NOTE]
> **SMB Multichannel Validation Key Points:**
> - Both storage VLANs (711 and 712) are operational with RDMA enabled
> - Storage VLANs 711 and 712 are operational with RDMA enabled (each mapped to its respective ToR)
> - `RdmaConnectionCount = 2` confirms RDMA is being used for storage traffic
> - `TcpConnectionCount = 0` shows no fallback to regular TCP
> - SMB 3.1.1 dialect is being used for optimal performance
Expand Down Expand Up @@ -405,26 +403,24 @@ For BGP routing configuration and best practices in Azure Local deployments:

## Frequently Asked Questions

### Q: Why must both Storage VLANs be on both ToR switches in Fully Converged?
### Q: How should Storage VLANs be configured in Fully Converged deployments?

**A:**
In Fully Converged deployments, **both storage VLANs (711 and 712) must be configured on both ToR switches**. This is required because:
The recommended baseline design uses **one storage VLAN per ToR switch** for Fully Converged deployments:

1. **Only 2 NICs per host**: Each host connects one NIC to ToR1 and one to ToR2
2. **SET handles traffic routing**: Switch Embedded Teaming maps storage vNICs to physical NICs at the host level
3. **Either VLAN through either NIC**: SET's load balancing may route Storage VLAN 711 or 712 through either physical NIC
- Storage VLAN A (711) is configured only on ToR-A and mapped to one physical NIC
- Storage VLAN B (712) is configured only on ToR-B and mapped to the other physical NIC
- In failure scenarios (NIC or ToR failure), SMB/RDMA traffic automatically fails over to the remaining path with reduced bandwidth but no functional impact

**How it differs from Switched deployment:**
**Storage VLAN Configuration:**

| Deployment Pattern | Storage NICs | ToR VLAN Config | Why |
|-------------------|--------------|-----------------|-----|
| **Fully Converged** | Shared (2 NICs total) | Both VLANs on both ToRs | SET may route either VLAN through either NIC |
| **Switched** | Dedicated (4 NICs total) | One VLAN per ToR | Each storage NIC connects to a specific ToR |

**Key Point:** The "one storage VLAN per ToR" optimization applies to **Switched** deployments where dedicated storage NICs connect to specific ToRs. In Fully Converged, SET's flexibility requires both VLANs on both switches.
| **Fully Converged** | Shared (2 NICs total) | S1 on ToR-A only, S2 on ToR-B only | One storage VLAN per NIC; failover occurs automatically |
| **Switched** | Dedicated (4 NICs total) | S1 on ToR-A only, S2 on ToR-B only | Each storage NIC connects to a specific ToR |

> [!NOTE]
> SET uses vNIC-to-pNIC affinity mapping to optimize traffic paths, but the switches must still be configured to carry both storage VLANs to handle any mapping SET chooses.
> Configuring both storage VLANs on both ToR switches is also supported but optional. Testing has confirmed there is no meaningful resiliency or failover benefit from this configuration, and it increases complexity without improving availability.


## Additional Resources
Expand Down