docs: improve the bandwidth manager page

Documentation/gettingstarted/bandwidth-manager.rst

@@ -63,19 +63,20 @@ To enable the bandwidth manager on an existing installation, run
    kubectl -n kube-system rollout restart ds/cilium
 
 The native host networking devices are auto detected as native devices which have
-the default route on the host or have Kubernetes InternalIP or ExternalIP assigned.
-InternalIP is preferred over ExternalIP if both exist. To change and manually specify
-the devices, set their names in the ``devices`` helm option, e.g.
-``devices='{eth0,eth1,eth2}'``. Each listed device has to be named the same
-on all Cilium managed nodes.
+the default route on the host or have Kubernetes ``InternalIP`` or ``ExternalIP`` assigned.
+``InternalIP`` is preferred over ``ExternalIP`` if both exist. To change and manually specify
+the devices, set their names in the ``devices`` helm option (e.g.
+``devices='{eth0,eth1,eth2}'``). Each listed device has to be named the same
+on all Cilium-managed nodes.
 
-Verify that the Cilium Pod has come up correctly:
+Verify that the Cilium Pods have come up correctly:
 
 .. code-block:: shell-session
 
     $ kubectl -n kube-system get pods -l k8s-app=cilium
     NAME                READY     STATUS    RESTARTS   AGE
     cilium-crf7f        1/1       Running   0          10m
+    cilium-db21a        1/1       Running   0          10m
 
 In order to verify whether the bandwidth manager feature has been enabled in Cilium,
 the ``cilium status`` CLI command provides visibility through the ``BandwidthManager``
@@ -84,45 +85,66 @@ is enforced:
 
 .. code-block:: shell-session
 
-    $ kubectl exec -it -n kube-system cilium-xxxxx -- cilium status | grep BandwidthManager
+    $ kubectl -n kube-system exec ds/cilium -- cilium status | grep BandwidthManager
     BandwidthManager:       EDT with BPF   [eth0]
 
-Assuming we have a multi-node cluster, in the next step, we deploy a netperf Pod on
-the node named foobar. The following example deployment yaml limits the egress
-bandwidth of the netperf Pod on the node's physical device:
+To verify that egress bandwidth limits are indeed being enforced, one can deploy two
+``netperf`` Pods in different nodes — one acting as a server and one acting as the client:
 
 .. code-block:: yaml
 
-    apiVersion: apps/v1
-    kind: Deployment
+    ---
+    apiVersion: v1
+    kind: Pod
     metadata:
-      name: netperf
+      annotations:
+        # Limits egress bandwidth to 10Mbit/s.
+        kubernetes.io/egress-bandwidth: "10M"
+      labels:
+        # This pod will act as server.
+        app.kubernetes.io/name: netperf-server
+      name: netperf-server
     spec:
-      selector:
-        matchLabels:
-          run: netperf
-      replicas: 1
-      template:
-        metadata:
-          labels:
-            run: netperf
-          annotations:
-            kubernetes.io/egress-bandwidth: "10M"
-        spec:
-          nodeName: foobar
-          containers:
-          - name: netperf
-            image: cilium/netperf
-            ports:
-            - containerPort: 12865
-
-Once up and running, ``netperf`` client can be invoked from a different node in the
-cluster using the Pod's IP (in this case ``10.217.0.254``) directly. The test streaming
-direction is from the netperf deployment towards the client, hence ``TCP_MAERTS``:
+      containers:
+      - name: netperf
+        image: cilium/netperf
+        ports:
+        - containerPort: 12865
+    ---
+    apiVersion: v1
+    kind: Pod
+    metadata:
+      # This Pod will act as client.
+      name: netperf-client
+    spec:
+      affinity:
+        # Prevents the client from being scheduled to the
+        # same node as the server.
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchExpressions:
+              - key: app.kubernetes.io/name
+                operator: In
+                values:
+                - netperf-server
+            topologyKey: kubernetes.io/hostname
+      containers:
+      - name: netperf
+        args:
+        - sleep
+        - infinity
+        image: cilium/netperf
+
+Once up and running, the ``netperf-client`` Pod can be used to test egress bandwidth enforcement
+on the ``netperf-server`` Pod. As the test streaming direction is from the ``netperf-server`` Pod
+towards the client, we need to check ``TCP_MAERTS``:
 
 .. code-block:: shell-session
 
-  $ netperf -t TCP_MAERTS -H 10.217.0.254
+  $ NETPERF_SERVER_IP=$(kubectl get pod netperf-server -o jsonpath='{.status.podIP}')
+  $ kubectl exec netperf-client -- \
+      netperf -t TCP_MAERTS -H "${NETPERF_SERVER_IP}"
   MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.217.0.254 () port 0 AF_INET
   Recv   Send    Send
   Socket Socket  Message  Elapsed
@@ -131,14 +153,15 @@ direction is from the netperf deployment towards the client, hence ``TCP_MAERTS`
 
    87380  16384  16384    10.00       9.56
 
-As can be seen, egress traffic of the netperf Pod has been limited to 10Mbit per second.
+As can be seen, egress traffic of the ``netperf-server`` Pod has been limited to 10Mbit per second.
 
 In order to introspect current endpoint bandwidth settings from BPF side, the following
-command can be run:
+command can be run (replace ``cilium-xxxxx`` with the name of the Cilium Pod that is co-located with
+the ``netperf-server`` Pod):
 
 .. code-block:: shell-session
 
-    $ kubectl exec -it -n kube-system cilium-xxxxx -- cilium bpf bandwidth list
+    $ kubectl exec -it -n kube-system cilium-xxxxxx -- cilium bpf bandwidth list
     IDENTITY   EGRESS BANDWIDTH (BitsPerSec)
     491        10M