get-started-with-iperf3-LP

content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/_index.md

@@ -0,0 +1,43 @@
+---
+title: Get started with network microbenchmarking and tuning with iperf3
+
+minutes_to_complete: 30
+
+who_is_this_for: Performance Engineers, Linux system administrators or application developers looking to microbenchmark, simulate or tune the networking performance of distributed systems.
+
+learning_objectives: 
+    - Understand how to use the iperf3 tool to microbenchmark different network conditions
+    - Understand how to use the tc tool to simulate different network environments
+    - Understand basic runtime parameters to tune performance for your application
+
+prerequisites:
+    - Foundational understanding on networking principles such as TCP/IP and UDP.
+    - Access to Arm-based cloud instances or access to physical hardware
+
+author: Kieran Hejmadi
+
+### Tags
+skilllevels: Introductory
+subjects: Performance and Optimization
+armips:
+    - Neoverse
+tools_software_languages:
+    - iperf3
+operatingsystems:
+    - Linux
+
+
+
+further_reading:
+    - resource:
+        title: iperf3 user manual 
+        link: https://iperf.fr/iperf-doc.php
+        type: documentation
+
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1                       # _index.md always has weight of 1 to order correctly
+layout: "learningpathall"       # All files under learning paths have this same wrapper
+learning_path_main_page: "yes"  # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
+---

content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/_next-steps.md

@@ -0,0 +1,8 @@
+---
+# ================================================================================
+#       FIXED, DO NOT MODIFY THIS FILE
+# ================================================================================
+weight: 21                  # Set to always be larger than the content in this path to be at the end of the navigation.
+title: "Next Steps"         # Always the same, html page title.
+layout: "learningpathall"   # All files under learning paths have this same wrapper for Hugo processing.
+---

content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/basic-microbenchmarking.md

@@ -0,0 +1,101 @@
+---
+title: Microbenchmark Existing Network Connection
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Microbenchmark TCP Connection
+
+
+First we will microbenchmark the bandwidth between the client and server. First start the `iperf` server on the server node with the following command. 
+
+```bash
+iperf3 -s
+```
+
+```output
+-----------------------------------------------------------
+Server listening on 5201 (test #1)
+-----------------------------------------------------------
+
+
+```
+By default, the server listens on port 5201. Use the `-p` flag to specify another port if it is in use.
+
+{{% notice Tip %}}
+If you already have an `iperf3` server running, you can manually kill the process with the following command. 
+ ```bash
+ sudo kill $(pgrep iperf3)
+ ```
+{{% /notice %}}
+
+
+Next, on the client node, run the following command to run a simple 10-second microbenchmark using the TCP protocol. 
+
+```bash
+iperf3 -c SERVER -V
+```
+
+```output
+...
+[  5] local 10.248.213.97 port 42176 connected to 10.248.213.104 port 5201
+Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
+[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
+[  5]   0.00-1.00   sec   594 MBytes  4.98 Gbits/sec    0   1.48 MBytes       
+[  5]   1.00-2.00   sec   593 MBytes  4.97 Gbits/sec    0   2.07 MBytes       
+[  5]   2.00-3.00   sec   592 MBytes  4.97 Gbits/sec    0   2.07 MBytes       
+[  5]   3.00-4.00   sec   590 MBytes  4.96 Gbits/sec    0   2.07 MBytes       
+[  5]   4.00-5.00   sec   593 MBytes  4.97 Gbits/sec    0   2.18 MBytes       
+[  5]   5.00-6.00   sec   591 MBytes  4.96 Gbits/sec    0   2.18 MBytes       
+[  5]   6.00-7.00   sec   592 MBytes  4.97 Gbits/sec    0   2.18 MBytes       
+[  5]   7.00-8.00   sec   593 MBytes  4.97 Gbits/sec    0   2.18 MBytes       
+[  5]   8.00-9.00   sec   588 MBytes  4.93 Gbits/sec    0   2.18 MBytes       
+[  5]   9.00-10.00  sec   592 MBytes  4.96 Gbits/sec    0   2.18 MBytes       
+- - - - - - - - - - - - - - - - - - - - - - - - -
+Test Complete. Summary Results:
+[ ID] Interval           Transfer     Bitrate         Retr
+[  5]   0.00-10.00  sec  5.78 GBytes  4.96 Gbits/sec    0             sender
+[  5]   0.00-10.00  sec  5.78 GBytes  4.96 Gbits/sec                  receiver
+CPU Utilization: local/sender 5.3% (0.1%u/5.2%s), remote/receiver 26.7% (1.2%u/25.5%s)
+snd_tcp_congestion cubic
+rcv_tcp_congestion cubic
+
+iperf Done.
+```
+
+- The`Cwnd` stands for the control window size and corresponds to the allowed number of TCP transactions inflight before receiving an acknowledgment `ACK` from the server. This adjusts dynamically to not overwhelm the receiver and adjust for variable link connection strengths. 
+
+- The `CPU Utilization` row shows both the usage on the sender and receiver. If you are migrating your workload to a different platform, such as from `x86` to `AArch64`, there may be subtle variations. 
+
+- The `snd_tcp_congestion cubic` abd `rcv_tcp_congestion cubic` variables show the congestion control algorithm used.
+
+- This `bitrate` shows the throughput achieved under this microbenchmark. As we can see from the above, we have saturated the 5 Gbps bandwidth available to our `t4g.xlarge` AWS instance. 
+
+![instance-network-size](./instance-network-size.png)
+
+### Microbenchmark UDP connection
+
+We can also microbenchmark the `UDP` protocol with the `-u` flag. As a reminder, UDP does not guarantee packet delivery with some packets being lost. As such we need to observe the statistics on the server side to see the % of packet lost and the variation in packet arrival time (jitter). The UDP protocol is widely used in applications that need timely packet delivery, such as online gaming on video calls. 
+
+Run the following command from the client to send 2 parallel UDP streams with the `-P 2` option.
+
+```bash
+iperf3 -c SERVER -V -u -P 2
+```
+
+Looking at the server output we can observe 0% of packets where lost for our short test. 
+
+```output
+[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
+[  5]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  0.016 ms  0/147 (0%)  receiver
+[  6]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  0.014 ms  0/147 (0%)  receiver
+[SUM]   0.00-10.00  sec  2.51 MBytes  2.10 Mbits/sec  0.015 ms  0/294 (0%)  receiver
+```
+
+Additionally on the client side, our 2 streams saturated 2 of our 4 cores in the local node. 
+
+```output
+CPU Utilization: local/sender 200.3% (200.3%u/0.0%s), remote/receiver 0.2% (0.0%u/0.2%s)
+```

content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/setup.md

@@ -0,0 +1,75 @@
+---
+title: Setup
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+### Setup
+
+For this demonstration I will be using instances available from AWS within a virtual private cloud (VPC)
+
+Create 2 Arm-based linux instances, 1 to act as the server and the other to act as the client. In this tutorial I will be using two `t4g.xlarge` instance running Ubuntu 22.04 LTS. 
+
+
+### Install dependencies
+
+Run the following command to install the microbenchmark tool, `iperf3`. 
+
+```bash
+sudo apt update
+sudo apt install iperf3 -y
+```
+
+
+### Update Security Rules 
+
+Next, we need to update the default security rules to enable specific inbound and outbound protocols. From the AWS console, navigate to the security tab. Edit the inbound rules to enable `ICMP`, `UDP` and `TCP` traffic to enable communication between the client and server
+
+
+![example_traffic](./example_traffic_rules.png)
+
+{{% notice Note %}}
+For security set the source and port ranges to those that are being used
+{{% /notice %}}
+
+
+### Update local DNS
+
+For readability, we will add the server IP address and an alias to the local DNS cache in `/etc/hosts`. The local IP address of the server and client can be found in the AWS dashboard. 
+
+On the client, add the IP address of the server to the `/etc/hosts` file. Likewise on the server add the IP address of the client to the `/etc/hosts` file. 
+
+![server-ip](./server-ip.png). 
+
+### Confirm server is reachable
+
+Finally, confirm the client can reach the server with the ping command below. As a reference we also ping the localhost. 
+
+```bash
+ping SERVER -c 3 && ping 127.0.0.1 -c 3
+```
+
+The output below shows that both SERVER and localhost (127.0.0.1) are reachable. Naturally, on this system local host response tile is ~10x faster than the server. Your results will vary depending on geographic colocation and other networking factors. 
+
+```output
+PING SERVER (10.248.213.104) 56(84) bytes of data.
+64 bytes from SERVER (10.248.213.104): icmp_seq=1 ttl=64 time=0.217 ms
+64 bytes from SERVER (10.248.213.104): icmp_seq=2 ttl=64 time=0.218 ms
+64 bytes from SERVER (10.248.213.104): icmp_seq=3 ttl=64 time=0.219 ms
+
+--- SERVER ping statistics ---
+3 packets transmitted, 3 received, 0% packet loss, time 2056ms
+rtt min/avg/max/mdev = 0.217/0.218/0.219/0.000 ms
+PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
+64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.022 ms
+64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.032 ms
+64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.029 ms
+
+--- 127.0.0.1 ping statistics ---
+3 packets transmitted, 3 received, 0% packet loss, time 2046ms
+rtt min/avg/max/mdev = 0.022/0.027/0.032/0.004 ms
+
+```
+

content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/simulating-network-conditions.md

@@ -0,0 +1,91 @@
+---
+title: Simulating Different Network Conditions
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Adding a delay to a TCP connection
+
+The linux `tc` utility can be used to manipulate traffic control settings. First, find the name of interface with the following command. 
+
+```bash
+ip addr show
+```
+
+The output below shows the `ens5` network interface device (NIC) is the device we want to manipulate.
+
+```output
+1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+    inet6 ::1/128 scope host noprefixroute 
+       valid_lft forever preferred_lft forever
+2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
+    link/ether 0a:92:1b:a9:63:29 brd ff:ff:ff:ff:ff:ff
+    inet 10.248.213.97/26 metric 100 brd 10.248.213.127 scope global dynamic ens5
+       valid_lft 1984sec preferred_lft 1984sec
+    inet6 fe80::892:1bff:fea9:6329/64 scope link 
+       valid_lft forever preferred_lft forever
+
+```
+
+Run the following command to add an emulated delay of 10ms on `ens5`. 
+
+```bash
+sudo tc qdisc add dev ens5 root netem delay 10ms
+```
+
+Rerunning the basic TCP test (`iperf3 -c SERVER -V`) with a delay we observe the `Cwnd` size has grew larger to compensate for the longer response time. Additionally, the bitrate has dropped from ~4.9 to ~2.3 `Gbit/sec`.
+
+
+```output
+[  5] local 10.248.213.97 port 43170 connected to 10.248.213.104 port 5201
+Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
+[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
+[  5]   0.00-1.00   sec   282 MBytes  2.36 Gbits/sec    0   8.02 MBytes       
+[  5]   1.00-2.00   sec   302 MBytes  2.53 Gbits/sec    0   8.02 MBytes       
+[  5]   2.00-3.00   sec   301 MBytes  2.52 Gbits/sec    0   8.02 MBytes       
+[  5]   3.00-4.00   sec   302 MBytes  2.54 Gbits/sec    0   8.02 MBytes       
+[  5]   4.00-5.00   sec   302 MBytes  2.53 Gbits/sec    0   8.02 MBytes       
+[  5]   5.00-6.00   sec   304 MBytes  2.55 Gbits/sec    0   8.02 MBytes       
+[  5]   6.00-7.00   sec   302 MBytes  2.53 Gbits/sec    0   8.02 MBytes       
+[  5]   7.00-8.00   sec   303 MBytes  2.54 Gbits/sec    0   8.02 MBytes       
+[  5]   8.00-9.00   sec   303 MBytes  2.54 Gbits/sec    0   8.02 MBytes       
+[  5]   9.00-10.00  sec   301 MBytes  2.53 Gbits/sec    0   8.02 MBytes       
+- - - - - - - - - - - - - - - - - - - - - - - - -
+Test Complete. Summary Results:
+[ ID] Interval           Transfer     Bitrate         Retr
+[  5]   0.00-10.00  sec  2.93 GBytes  2.52 Gbits/sec    0             sender
+[  5]   0.00-10.01  sec  2.93 GBytes  2.52 Gbits/sec                  receiver
+CPU Utilization: local/sender 3.4% (0.0%u/3.4%s), remote/receiver 11.0% (0.4%u/10.7%s)
+snd_tcp_congestion cubic
+rcv_tcp_congestion cubic
+
+iperf Done.
+```
+
+### Simulating Packet Loss
+
+To test the resiliency of a distributed application we can add a simulated packet loss of 1%. As opposed to a 10ms delay, this will result in no acknowledgment being received for 1% of packets. Given TCP is a lossless protocol a retry must be sent. 
+
+```bash
+sudo tc qdisc del dev ens5 root
+sudo tc qdisc add dev ens5 root netem loss 1%
+```
+
+Rerunning the basic TCP test we observe an increased number of retries (`Retr`) and a corresponding drop in bitrate. 
+
+```bash
+iperf3 -c SERVER -V
+```
+```output
+Test Complete. Summary Results:
+[ ID] Interval           Transfer     Bitrate         Retr
+[  5]   0.00-10.00  sec  4.41 GBytes  3.78 Gbits/sec  5030             sender
+[  5]   0.00-10.00  sec  4.40 GBytes  3.78 Gbits/sec                  receiver
+```
+
+Please see the `tc` [user documentation](https://man7.org/linux/man-pages/man8/tc.8.html) for the different ways to simulate different perturbation and your systems resiliency to such events.

content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/tuning.md

@@ -0,0 +1,66 @@
+---
+title: Tuning Kernel Parameters
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+### Connecting from Local Machine
+
+Now we can observe ways to mitigate performance degradation due to events such as packet loss. In this example, I will connect to the AWS server node from my local machine to demonstrate a longer response time. Please check the `iperf3` installation guide on the [official documentation](https://iperf.fr/iperf-download.php) if you're not using Ubuntu. As the output below shows we have a larger round trip time in excess of 40ms. 
+
+```output
+5 packets transmitted, 5 packets received, 0.0% packet loss
+round-trip min/avg/max/stddev = 44.896/46.967/49.279/1.444 ms
+```
+
+Running a standard TCP client connection with the `iperf3 -c SERVER -V` command shows an average bitrate of 157 Mbps.
+
+```output
+Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
+...
+Test Complete. Summary Results:
+[ ID] Interval           Transfer     Bitrate
+[  8]   0.00-10.01  sec   187 MBytes   157 Mbits/sec                  sender
+[  8]   0.00-10.03  sec   187 MBytes   156 Mbits/sec                  receiver
+```
+
+
+
+### Modify kernel parameters
+
+On the server, we can configure linux kernel runtime parameters with the `sysctl` command. 
+
+There are a plenthora of dials to tune that relate to performance and security. The following command can be used to list all available dials. The corresponding [kernel documentation](https://docs.kernel.org/networking/ip-sysctl.html#ip-sysctl) can provide a more detailed description of each parameter. 
+
+```bash
+sysctl -a | grep tcp
+```
+
+{{% notice Note %}}
+Depending on your operating system, some parameters may not be available. For example on AWS Ubuntu 22.04 LTS only the `cubic` and `reno` congestion control algorithms are available.
+```bash
+net.ipv4.tcp_available_congestion_control = reno cubic
+```
+{{% /notice %}}
+
+
+We can increase the read and write max buffer sizes of the kernel on the server to enable more data to be held. This is at the tradeoff of increased memory utilisation. run the following commands from the server.
+
+```bash
+sudo sysctl net.core.rmem_max=134217728 # default = 212992
+sudo sysctl net.core.wmem_max=134217728 # default = 212992
+```
+
+Restart the `iperf3` server.  Run the `iperf3 -c SERVER -V` command from the client leads to significantly improved bitrate with no modification on the client side. 
+
+```output
+Test Complete. Summary Results:
+[ ID] Interval           Transfer     Bitrate
+[  8]   0.00-10.00  sec   308 MBytes   258 Mbits/sec                  sender
+[  8]   0.00-10.03  sec   307 MBytes   257 Mbits/sec                  receiver
+
+```
+
+This learning path serves as an introduction to microbenchmarking and performance tuning. Which parameters to adjust depends on your own use case and non-functional performance requirements of your system.