Archive Trac jetstream network

Jetstream Network

Legacy Trac archive page imported from jetstream_network. Source: https://genapp.rocks/wiki/wiki/jetstream_network. Review age, links, and examples before treating as current.

jetstream network notes

G'afternoon Emre,

    I have not done any MPI type cross-node parallel benchmarking.  I have done point-to-point network benchmarks and we are able to pretty much get line rate; though, it took some optimization to push it to the limit.  I have also done parallel I/O to storage benchmarks looking for the saturation point on I/O.  In the Jetstream grant proposal, we only identified node level parallelization as a priority.  
    
    I've appended a detailed description of the network topologies of the two clouds below. In simple terms, the network fabric for both clouds is based on 10 gb/s ethernet; Intel X710 NICs in the host systems, DELL MXL switches in the blade chassis and Dell S6000 top of rack (leaf) and spine switches.   IU and TACC have different topologies. TACC is a pretty simple 10 gb/s fat-tree; nothing fancy.  IU has bonded LAgs at each level resulting in a true two-to-one oversubscription at each level fat-tree topology.  
    
    As for optimizing your network stack
    
MTU=9000

Kernel tunables:
  sysctl -w net.core.netdev_max_backlog=300000
  sysctl -w 'net.ipv4.tcp_wmem=4096 87380 16777216'
  sysctl -w 'net.ipv4.tcp_rmem=4096 87380 16777216'
  sysctl -w net.core.wmem_max=16777216
  sysctl -w net.core.rmem_max=16777216
  sysctl -w net.ipv4.tcp_mtu_probing=1

Open up your firewalls
I run CentOS7 with firewalld, your mileage may vary;  use your local subnet CIDR block
Either of these should work; but, in the interest of time I gave up and just dropped the firewall. If all of your worker nodes have non-routable addresses, you don't need a firewall.

   firewall-cmd --add-rich-rule='rule family="ipv4" source address="www.xxx.yyy.0/26" accept'

or

cat > /etc/firewalld/services/local-network.xml << EOF
<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>local-network</short>
  <description>Enable MPI communications on local network</description>
  <port protocol="udp" port=""/>
  <port protocol="tcp" port=""/>
  <destination ipv4="10.0.0.0/24"/>
</service>
EOF

firewall-cmd --permanent --zone=public --add-service=local-network

If all of your worker nodes have non-routable addresses, you don't need a firewall

You will need to add a security group that allows inbound/outbound traffic on your local network;  here's my not-default default security group.

   # create a security group that will enable inbound ping & ssh
   # as well as communications on the local non-routing network
   nova secgroup-create global-ssh "ssh & icmp enabled"
   nova secgroup-add-rule global-ssh tcp 22 22 0.0.0.0/0
   nova secgroup-add-rule global-ssh icmp -1 -1 0.0.0.0/0
   nova secgroup-add-rule global-ssh tcp 1 65535 10.0.0.0/24
   nova secgroup-add-rule global-ssh udp 1 65535 10.0.0.0/24

There's probably more that I'm forgetting.  I'll ping our guru and see if he has any other suggestions.  Of course, if ya' have questions, comments, suggestions, ya' know where to find us.   - geo


George Wm Turner 
High Performance & Research Computing Architect
<redacted-email>
812.855.5156

Emre,

   Doh!  I forgot to include the topology write up

Cluster Internal Network Topologies

IU
   IU took the approach of bonding their network connections at each
   layer in the Fat Tree hierarchy into Link Aggregations (LAGs) and
   running both management and client traffic over a high capability,
   high availability network.  Dual 10 Gb/s connections from each blade
   are bonded together and connect into the two in-chassis switches for
   a link aggregate of 20 Gb/s. These two chassis swiches are themselves
   cross connected with two 40 Gb/s links, 80 Gb/s link aggregate, to
   form a single virtual switch.  Dual 40 Gb/s uplink connections run
   from each chassis switch to the Top of Rack (ToR) switch resulting in
   a link aggregate of 160 Gb/s. Dual 40 Gb/s uplinks from the ToR
   switches runs to each of the two Spine switches for resulting in a
   link aggregate of 160 Gb/s. And, the two Spine switches are
   interconnected via three 40 Gb/s connections resulting in 120 Gb/s
   inter-Spline link aggregate. The result is a highly fault tolerant,
   high capacity network topology.


TACC
   TACC took the approach of separating network traffic into two
   independent 10 Gb/s based  management network and client VM networks.
   The advantage of this topology allows for OpenStack operations as
   well as cluster management traffic to occur without interferring
   with, nor interference from, the client VM traffic. The two 10 Gb/s
   NICs within each blade uplink to two separate switches installed in
   the chassis. Each switch then uplinks via one 40 Gb/s connection to
   the Top of Rack (ToR) switch which has two separate VLANs to
   accommodate the separate traffic.  These VLANs are then uplinked to
   two separate Spine switches carrying the management and client
   traffic respectively. This results in a highly reliable management
   and operations network while still providing sufficient capacity for
   the client instances.

 - geo