FMADIO Systems have multiple 1G, 10G and 40G management interfaces, depending on the ordered SKU.
Management interfaces are all bridged by default per the following block diagram
Using the above configuration allows
- LXC containers full pass-thru IP address (no NAT)
- Bonded management mode for Redundancy (Hot-Standby)
- Bonded management mode for Throughput ( LAG )
Example ifconfig of the system is as follows
fmadio@fmadio20n40v3-363:~$ ifconfig -a
man0 Link encap:Ethernet HWaddr 18:C0:4D:B4:0E:6C
inet addr:192.168.2.225 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::1ac0:4dff:feb4:e6c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2371 errors:0 dropped:0 overruns:0 frame:0
TX packets:136 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:911945 (890.5 KiB) TX bytes:24130 (23.5 KiB)
man10 Link encap:Ethernet HWaddr 18:C0:4D:17:7A:4E
inet addr:192.168.15.225 Bcast:192.168.15.255 Mask:255.255.255.0
inet6 addr: fe80::1ac0:4dff:fe17:7a4e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9200 Metric:1
RX packets:5227 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:253670 (247.7 KiB) TX bytes:1068 (1.0 KiB)
phy0 Link encap:Ethernet HWaddr 18:C0:4D:B4:0E:6C
inet6 addr: fe80::1ac0:4dff:feb4:e6c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2372 errors:0 dropped:0 overruns:0 frame:0
TX packets:151 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:945205 (923.0 KiB) TX bytes:25260 (24.6 KiB)
phy1 Link encap:Ethernet HWaddr 18:C0:4D:B4:0E:6D
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
phy10 Link encap:Ethernet HWaddr 18:C0:4D:17:7A:4E
inet6 addr: fe80::1ac0:4dff:fe17:7a4e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9200 Metric:1
RX packets:5262 errors:0 dropped:35 overruns:0 frame:0
TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:331398 (323.6 KiB) TX bytes:2066 (2.0 KiB)
phy11 Link encap:Ethernet HWaddr 18:C0:4D:17:7A:4F
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
fmadio@fmadio20n40v3-363:~$
And bridge settings
fmadio@fmadio20n40v3-363:~$ brctl show man0
bridge name bridge id STP enabled interfaces
man0 8000.18c04db40e6c no phy0
fmadio@fmadio20n40v3-363:~$ brctl show man10
bridge name bridge id STP enabled interfaces
man10 8000.18c04d177a4e no phy10
By default MTU size is set to 1500B for maximum compatibility. This can be configure for 9200 Jumbo frame support to maximize download throughput. This is done by setting
["MTU"] = 9200,
For both man10 and phy10 network interfaces in the network configuration script below.
/opt/fmadio/etc/network.lua
This has to be set on both the man10 and phy10 (optionally phy11 if used) interfaces to be fully effective as per below example.
["man10"] =
{
["Mode"] = "static",
["Address"] = "192.168.15.225",
["Netmask"] = "255.255.255.0",
["Gateway"] = "",
["DNS0"] = "",
["DNS1"] = "",
["MTU"] = 9200,
}
,
["phy10"] =
{
["MTU"] = 9200,
}
Requires FW:6508+
LACP or Link Bonding is critical for fail over / redundancy planning. FMADIO Packet Capture devices run on Linux thus we support LCAP/Bonding on the management interfaces.
/opt/fmadio/etc/network.lua
Add a bonded interface "bond0" as follows
["bond0"] =
{
["Mode"] = "bond",
["Address"] = "192.168.1.2",
["Netmask"] = "255.255.255.0",
["Gateway"] = "192.168.1.1",
["DNS0"] = "",
["DNS1"] = "",
["Slave"] = { "phy0", "phy1" }
},
In the above example the "Slave" field contains the list of physical interfaces the bonding runs on. This example is bonding the two 1G RJ45 interfaces on the system. To bond the 10G interfaces on a separate LCAP link (bond1), use the following:
["bond1"] =
{
["Mode"] = "bond",
["Address"] = "192.168.1.2",
["Netmask"] = "255.255.255.0",
["Gateway"] = "192.168.1.1",
["DNS0"] = "",
["DNS1"] = "",
["Slave"] = { "phy10", "phy11" }
},
Requires FW: 6633+
By default 802.3ad bonding mode is used, full list of Linux bonding modes can be seen on kernel.org. Note "BondMode" specifies the Linux bonding mode to be used.
["bond1"] =
{
["Mode"] = "bond",
["BondMode"] = "active-backup",
["Address"] = "192.168.1.2",
["Netmask"] = "255.255.255.0",
["Gateway"] = "192.168.1.1",
["DNS0"] = "",
["DNS1"] = "",
["Slave"] = { "phy10", "phy11" }
},
Line Bonding mode options (details ripped from kernel.org)
Round-robin (balance-rr)
Transmit network packets in sequential order from the first available network interface (NIC) slave through the last. This mode provides load balancing and fault tolerance.
Active-backup (active-backup)
Only one NIC slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The single logical bonded interface's MAC address is externally visible on only one NIC (port) to avoid distortion in the network switch. This mode provides fault tolerance.
XOR (balance-xor)
Transmit network packets based on a hash of the packet's source and destination. The default algorithm only considers MAC addresses (layer2). Newer versions allow selection of additional policies based on IP addresses (layer2+3) and TCP/UDP port numbers (layer3+4). This selects the same NIC slave for each destination MAC address, IP address, or IP address and port combination, respectively. This mode provides load balancing and fault tolerance.
Broadcast (broadcast)
Transmit network packets on all slave network interfaces. This mode provides fault tolerance.
Default mode
IEEE 802.3ad Dynamic link aggregation (802.3ad, LACP)
Creates aggregation groups that share the same speed and duplex settings. Utilizes all slave network interfaces in the active aggregator group according to the 802.3ad specification. This mode is similar to the XOR mode above and supports the same balancing policies. The link is set up dynamically between two LACP-supporting peers.
Adaptive transmit load balancing (balance-tlb)
Linux bonding driver mode that does not require any special network-switch support. The outgoing network packet traffic is distributed according to the current load (computed relative to the speed) on each network interface slave. Incoming traffic is received by one currently designated slave network interface. If this receiving slave fails, another slave takes over the MAC address of the failed receiving slave.
Adaptive load balancing (balance-alb)
includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special network switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the NIC slaves in the single logical bonded interface such that different network-peers use different MAC addresses for their network packet traffic.
NOTE: PTPv2 and LCAP on the 10G Management interfaces are mutually exclusive.