bwm: temporarily disable setting bbr until we have a pacing fix

Consider a socket which has SO_MAX_PACING_RATE of 4Gbit/s and the
socket being part of a Pod. This is currently broken given skb->tstamps
are cleared on BPF redirect as well as netns traversal even though
fq in hostns manages the socket's pacing. Rates would result being
unpredictable:

  root@apoc:~/go/src/github.com/cilium/cilium# netperf -H 10.217.1.19 -t TCP_STREAM -l40 -s2
  MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.217.1.19 () port 0 AF_INET : demo
  Recv   Send    Send
  Socket Socket  Message  Elapsed
  Size   Size    Size     Time     Throughput
  bytes  bytes   bytes    secs.    10^6bits/sec
   87380  16384  16384    40.04     655.52

  root@apoc:~/go/src/github.com/cilium/cilium# netperf -H 10.217.1.19 -t TCP_STREAM -l40 -s2
  MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.217.1.19 () port 0 AF_INET : demo
  Recv   Send    Send
  Socket Socket  Message  Elapsed
  Size   Size    Size     Time     Throughput
  bytes  bytes   bytes    secs.    10^6bits/sec
   87380  16384  16384    40.07    1274.70

  root@apoc:~/go/src/github.com/cilium/cilium# netperf -H 10.217.1.19 -t TCP_STREAM -l40 -s2
  MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.217.1.19 () port 0 AF_INET : demo
  Recv   Send    Send
  Socket Socket  Message  Elapsed
  Size   Size    Size     Time     Throughput
  bytes  bytes   bytes    secs.    10^6bits/sec
   87380  16384  16384    40.07    1519.32

  root@apoc:~/go/src/github.com/cilium/cilium# netperf -H 10.217.1.19 -t TCP_STREAM -l40 -s2
  MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.217.1.19 () port 0 AF_INET : demo
  Recv   Send    Send
  Socket Socket  Message  Elapsed
  Size   Size    Size     Time     Throughput
  bytes  bytes   bytes    secs.    10^6bits/sec
   87380  16384  16384    40.06     849.96

We are working on a kernel side solution to retain skb->tstamps which
would fix this issue and result in a stable 4Gbit/s rate for this
example. Once that is merged we can reenable BBR from BWM side for
those kernels (and fallback to cubic for those that do not have it).

Related: #15324
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>

pkg/bandwidth/bandwidth.go

@@ -98,8 +98,11 @@ func InitBandwidthManager() {
 		{"net.core.netdev_max_backlog", "1000"},
 		{"net.core.somaxconn", "4096"},
 		{"net.core.default_qdisc", "fq"},
-		{"net.ipv4.tcp_congestion_control", "bbr"},
 		{"net.ipv4.tcp_max_syn_backlog", "4096"},
+		// Temporary disable setting bbr for now until we have a
+		// kernel fix for pacing out of Pods as described in #15324.
+		// Then, kernels with the fix can use bbr, and others cubic.
+		{"net.ipv4.tcp_congestion_control", "cubic"},
 	}
 	for _, s := range baseSettings {
 		log.WithFields(logrus.Fields{