RFE: Cilium CNI support

[vincentmli]

The requirement to support Cilium CNI is to create fake FDB for k8s node in BIG-IP, CIS has similar code like below:
https://github.com/F5Networks/k8s-bigip-ctlr/blob/master/pkg/vxlan/vxlanMgr.go#L123-L130

		rec := fdbRecord{}
		for _, addr := range nodeAddrs {
			if addr.Type == addrType {
				rec.Endpoint = addr.Address
				// Initially set the name to a fake MAC (for OpenShift use)
				// For flannel, this will be overwritten with the real MAC
				rec.Name = ipv4ToMac(addr.Address)
			}
		}

https://github.com/F5Networks/k8s-bigip-ctlr/blob/master/pkg/vxlan/vxlanMgr.go#L178-L189

func ipv4ToMac(addr string) string {
	ip := strings.Split(addr, ".")
	if len(ip) != 4 {
		log.Errorf("[VxLAN] Bad IPv4 address format specified for FDB record: %s", addr)
		return ""
	}
	var intIP [4]int
	for i, val := range ip {
		intIP[i], _ = strconv.Atoi(val)
	}
	return fmt.Sprintf("0a:0a:%02x:%02x:%02x:%02x", intIP[0], intIP[1], intIP[2], intIP[3])
}

see the fake fmt.Sprintf("0a:0a:%02x:%02x:%02x:%02x", intIP[0], intIP[1], intIP[2], intIP[3])

so from what I understand, probably https://github.com/f5devcentral/bigip-kubernetes-gateway/blob/master/internal/k8s/k8s.go#L40-L52 section could add cilium annotation check for Cilium k8s node to generate the fake k8s node FDB based on the node ip, and somehow your network rest golang code could send the fake fdb to BIG-IP.

your rest would also be able to create vxlan tunnel profile and tunnel, see the tmsh tunnel command here https://github.com/f5devcentral/f5-ci-docs/blob/master/docs/cilium/cilium-bigip-info.rst

#. Create a VXLAN tunnel profile. The tunnel profile name is fl-vxlan

tmsh create net tunnels vxlan fl-vxlan port 8472 flooding-type multipoint

#. Create a VXLAN tunnel, the tunnel name is ``flannel_vxlan``, in CIS use ``--openshift-sdn-name`` argument

tmsh create net tunnels tunnel flannel_vxlan key 2 profile fl-vxlan local-address 10.169.72.34

#. Create VXLAN tunnel self IP, allow default service, allow none stops self ip ping from working

tmsh create net self 10.1.6.34 address 10.1.6.34/255.255.255.0 allow-service default vlan flannel_vxlan

#. Create a static route to Cilium managed pod CIDR network ``10.0.0.0/16`` through tunnel interface ``flannel_vxlan``
#. DO NOT CREATE STATIC ROUTE THROUGH TUNNEL INTERFACE TO K8S NODE NETWORK/IP

tmsh create net route 10.0.0.0 network 10.0.0.0/16 interface flannel_vxlan

I noticed tmsh could only use modify to add FDB for tunnel, for example
tmsh modify net fdb tunnel flannel_vxlan records add { 0a:0a:0a:a9:48:ea { endpoint 10.169.72.234%0 } }, maybe the rest could simulate tmsh behavior

I am happy to answer any questions for this request :-)

[vincentmli]

Here is how Cilium VXLAN data path to external VTEP device like F5 works

we enable Cilium VTEP feature in Cilium configmap like below:

          enable-vtep:   "true"
          vtep-endpoint: "10.169.72.34            10.169.72.36"
          vtep-cidr:         "10.1.6.0/24                 10.1.5.0/24"
          vtep-mac:      "01:50:56:A0:7D:D8     00:50:56:86:6b:28"
          vtep-mask:     "255.255.255.0"

we can list the enabled Cilium vtep:

kubectl -n kube-system -it <cilium agent pod> -- cilium bpf vtep list

IP PREFIX/ADDRESS   VTEP
10.1.6.0            vtepmac=00:50:56:A0:7D:D8 tunnelendpoint=10.169.72.34
10.1.5.0            vtepmac=00:50:56:86:6b:28 tunnelendpoint=10.169.72.36

in Cilium data path code, we have

#if defined(ENABLE_VTEP)
	{
		struct vtep_key vkey = {};
		struct vtep_value *vtep;

		vkey.vtep_ip = ip4->daddr & VTEP_MASK;
		vtep = map_lookup_elem(&VTEP_MAP, &vkey);
		if (!vtep)
			goto skip_vtep;

		if (vtep->vtep_mac && vtep->tunnel_endpoint) {
			if (eth_store_daddr(ctx, (__u8 *)&vtep->vtep_mac, 0) < 0)
				return DROP_WRITE_ERROR;
			return __encap_and_redirect_with_nodeid(ctx, vtep->tunnel_endpoint,
								SECLABEL, WORLD_ID,
								WORLD_ID, &trace);
		}
	}
skip_vtep:
#endif

when pod send packet to BIGIP tunnel ip, say ip4->daddr =10.1.5.1, Cilium do the ip4->daddr & VTEP_MASK that result in 10.1.5.0, cilium do the map lookup with 10.1.5.0, found key value vtepmac=00:50:56:A0:7D:D8 tunnelendpoint=10.169.72.34, then cilium do the tunnel inner packet destination MAC rewrite to 00:50:56:A0:7D:D8 which matches the flannel_vxlan interface MAC address, also use tunnelendpoint=10.169.72.34 to encapsulate tunnel inner packet with tunnel outer destination IP 10.169.72.34 which is the BIG-IP vlan self ip, this complete the VXLAN encapsulation and send the packet to BIG-IP.

I don't understand why BIG-IP requires fake FDB for k8s node, maybe it is part of VXLAN RFC, but Cilium VXLAN does not require FDB (magic of kernel eBPF I think), only BIG-IP.

[vincentmli]

@zongzw in case you are curious about Cilium VXLAN data path further, I did deep dive here:
https://youtu.be/1jwFEW0Bth0
https://youtu.be/Ocy2pFhNFfE
https://youtu.be/P3hZdX5Cyws

[zongzw]

@vincentmli Sure, thanks for PR, will look into it soon, as well as the provided materials.

[zongzw]

#68