do-we-need-distributed-switching-on.html

---
date: 2011-07-11T06:44:00.001+02:00
tags:
- switching
- data center
- workshop
title: Do we need distributed switching on Nexus 2000?
url: /2011/07/do-we-need-distributed-switching-on/
---

<p>Yandy sent me an interesting question:</p>
<blockquote class="cite">Is it just me or do you also see the Nexus 2000 series not having any type of distributed forwarding as a major design flaw? Cisco keeps throwing in the “it's a line-card” line, but any dumb modular switch nowadays has distributed forwarding in all its line cards.</blockquote>
<p>I’m at least as annoyed as Yandy is by the lack of distributed switching in the Nexus port (oops, <a href="/2010/08/port-or-fabric-extenders/">fabric</a>) extender product range, but let’s focus on a different question: does it matter?<!--more--></p>
<p>Some background information first. A Nexus 2000 fabric extender (FEX) looks like a linecard in a Nexus 5000 switch with a significant caveat: <a href="/2010/08/port-or-fabric-extenders/">all traffic between devices connected to a FEX must pass through the controlling switch</a>. This design allows the FEX to remain a dumb device and while distributed switching (the ability of a FEX to forward traffic between locally attached devices independently from the controlling switch) was promised a while ago, it seems that the evolution of the FEX architecture is steering away from it (the <a href="/2011/06/vn-tag8021qbh-basics/">802.1Qbh standard</a> has no provision for distributed switching).</p>
<p>Distributed low-latency any-to-any switching is highly desirable in most environments running <strong>distributed computations</strong> (for example, HPC or large-scale map-reduce workloads) where there’s heavy east-west traffic between hosts in the same VLAN. In such an environment, the FEX architecture will definitely increase latency and maybe even cause congestion due to large amount of traffic going through the controlling Nexus 5000. However, most workloads I see don’t fall into that category.</p>
<p><strong>Web server</strong><strong>s</strong> exchange traffic with end-users (browser clients) or back-end application or database servers, which are usually in a different VLAN (so the traffic has to cross a layer-3 device or even a firewall). <strong>Application servers</strong> usually have the same traffic characteristics. </p>
<p><strong>Virtual Desktop I</strong><strong>nfrastructure</strong> (VDI) traffic is similar. Screen, keyboard and mouse (KVM) traffic is sent to the end-user, application traffic is exchanged with the Internet or web/application servers (again, in a different VLAN) and the storage traffic is usually handled separately anyway. Yet again, no east-west traffic.</p>
<p>Mid-range <strong>database servers </strong>(like Microsoft’s SQL server) generate little intra-VLAN traffic unless you deploy a redundant setup, but even then the amount of intra-VLAN traffic is limited by the number of SQL transactions the in-memory caches and the spinning rust can support. High-end distributed database architecture (for example, MySQL cluster) could be a different story.</p>
<p><strong>Multi-tenant environment</strong> is probably a more interesting use case, more so if someone decides L3 inter-VLAN virtual appliances are a good idea (hint: they are NOT). There you could get a lot of east-west traffic due to inter-VLAN traffic being handled by a virtual machine that could reside anywhere in your network.</p>
<p>Summary: while distributed FEX switching could be very useful in some environments, you won’t see much difference in many cases. As always in the real life, you have to make tradeoffs; you could either have a Swiss army knife (which can always solve the problem ... more so if your last name is McGyver) or a durable Phillips screwdriver (which works really well, but might not be a good option if you need to open a bottle of wine). </p>
<p>In our scenario, if you have tens (or hundreds) of servers connected to a distribution-layer switch through access-layer devices, you have to decide whether you prefer lack of distributed switching and the ability to manage only two devices (a redundant pair of Nexus 5000s, which configure and manage all FEX devices connected to them) to managing tens of access-layer devices (redundant ToR/EoR or blade chassis switches). You could also decide to wait for the <a href="/2011/06/speculation-this-is-how-i-would-build/">QFabric magic</a>, but remember that (at least in its first incarnation) it won’t help you solve the blade chassis switch management problem.</p>
<h4>More information</h4><p>If you want to learn more about modern data center architectures, <a href="https://www.ipspace.net/Recordings?code=DC30">buy a recording</a> of my <a href="https://www.ipspace.net/DC30">Data Center 3.0 for Networking Engineers</a> webinar. For more Data Center-related webinars, check the <a href="https://www.ipspace.net/Roadmap/Data_center_webinars">Data Center webinar roadmap</a>. All those webinars are available as part of the <a href="https://www.ipspace.net/Subscription">yearly subscription package</a>.</p>
<br/>