datapath: bigtcp: Fix the IPv4 BIG TCP may not work

The kernel will also update the IPv4 GRO/GSO setting if the new value of
"gso/gro_max_size" isn't greater than 65536:

1. Enable IPv4 and IPv6 BIG TCP firstly:
  a). Dump the agent's log:
	$ kubectl -n kube-system logs cilium-tfhz5  2>&1 | grep "big-tcp"
	level=info msg="  --enable-ipv4-big-tcp='true'" subsys=daemon
	level=info msg="  --enable-ipv6-big-tcp='true'" subsys=daemon
	level=info msg="Setting up BIG TCP" subsys=big-tcp
	level=info msg="Setting IPv4 gso_max_size to 131072 and gro_max_size to 131072" device=enp0 subsys=big-tcp
	level=info msg="Setting IPv6 gso_max_size to 131072 and gro_max_size to 131072" device=enp0 subsys=big-tcp

  b). Check the GSO value on the host's net device:
	$ ip -d -j link show dev enp0 | jq -c '.[0].gso_max_size'
	131072

	$ ip -d -j link show dev enp0 | jq -c '.[0].gso_ipv4_max_size'
	131072

2. Then re-install the cilium by enabling IPv4 BIG TCP only:
  a). Dump the agent's log:
	$ kubectl -n kube-system logs cilium-zwpg6  2>&1 | grep "big-tcp"
	level=info msg="  --enable-ipv4-big-tcp='true'" subsys=daemon
	level=info msg="  --enable-ipv6-big-tcp='false'" subsys=daemon
	level=info msg="Setting up BIG TCP" subsys=big-tcp
	level=info msg="Setting IPv4 gso_max_size to 131072 and gro_max_size to 131072" device=enp0 subsys=big-tcp
	level=info msg="Setting IPv6 gso_max_size to 65536 and gro_max_size to 65536" device=enp0 subsys=big-tcp

  b). Check the GSO value on the host's net device:
	$ ip -d -j link show dev enp0 | jq -c '.[0].gso_max_size'
	65536

	$ip -d -j link show dev enp0 | jq -c '.[0].gso_ipv4_max_size'
	65536

No errors about the BIG TCP setting in cilium agent's log, but the value
of net device's '{gso,gro}_ipv4_max_size' is wrong.

So it needs to handle the IPv6 BIG TCP setting firstly, then IPv4.

Signed-off-by: Haiyue Wang <haiyue.wang@intel.com>
Link: https://lore.kernel.org/netdev/7e1f733cc96c7f7658fbf3276a90281b2f37acd1.1674921359.git.lucien.xin@gmail.com/

pkg/datapath/linux/bigtcp/bigtcp.go

@@ -219,31 +219,34 @@ func InitBIGTCP(bigTCPConfig *Configuration) {
 		// modification and end up with an error, so best to be conservative
 		// and always reset all on error
 		modifiedDevices = append(modifiedDevices, device)
-		if haveIPv4 {
-			err = setGROGSOIPv4MaxSize(device,
-				bigTCPConfig.groIPv4MaxSize, bigTCPConfig.gsoIPv4MaxSize)
+		// For compatibility, the Kernel will also update the net device's "gso/gro_ipv4_max_size",
+		// if the new size of "gso/gro_max_size" isn't greater than 64KB.
+		// So it needs to set the IPv6 firstly, otherwide, the IPv4 BIG TCP value will be reset.
+		if haveIPv6 {
+			err = setGROGSOIPv6MaxSize(device,
+				bigTCPConfig.groIPv6MaxSize, bigTCPConfig.gsoIPv6MaxSize)
 			if err != nil {
-				log.WithError(err).WithField("device", device).Warnf("Could not modify IPv4 gro_max_size and gso_max_size%s",
+				log.WithError(err).WithField("device", device).Warnf("Could not modify IPv6 gro_max_size and gso_max_size%s",
 					disableMsg)
 				option.Config.EnableIPv4BIGTCP = false
 				option.Config.EnableIPv6BIGTCP = false
 				break
 			}
-			log.WithField("device", device).Infof("Setting IPv4 gso_max_size to %d and gro_max_size to %d",
-				bigTCPConfig.gsoIPv4MaxSize, bigTCPConfig.groIPv4MaxSize)
+			log.WithField("device", device).Infof("Setting IPv6 gso_max_size to %d and gro_max_size to %d",
+				bigTCPConfig.gsoIPv6MaxSize, bigTCPConfig.groIPv6MaxSize)
 		}
-		if haveIPv6 {
-			err = setGROGSOIPv6MaxSize(device,
-				bigTCPConfig.groIPv6MaxSize, bigTCPConfig.gsoIPv6MaxSize)
+		if haveIPv4 {
+			err = setGROGSOIPv4MaxSize(device,
+				bigTCPConfig.groIPv4MaxSize, bigTCPConfig.gsoIPv4MaxSize)
 			if err != nil {
-				log.WithError(err).WithField("device", device).Warnf("Could not modify IPv6 gro_max_size and gso_max_size%s",
+				log.WithError(err).WithField("device", device).Warnf("Could not modify IPv4 gro_max_size and gso_max_size%s",
 					disableMsg)
 				option.Config.EnableIPv4BIGTCP = false
 				option.Config.EnableIPv6BIGTCP = false
 				break
 			}
-			log.WithField("device", device).Infof("Setting IPv6 gso_max_size to %d and gro_max_size to %d",
-				bigTCPConfig.gsoIPv6MaxSize, bigTCPConfig.groIPv6MaxSize)
+			log.WithField("device", device).Infof("Setting IPv4 gso_max_size to %d and gro_max_size to %d",
+				bigTCPConfig.gsoIPv4MaxSize, bigTCPConfig.groIPv4MaxSize)
 		}
 	}