As a follow-up to issue #470 I've changed the address family temporarily and hit a few more issues/concerns/unexpected items, and figured I'd document them here FYI.
First, I started with an existing cluster, that had bridge CNI plugin. The config called out a /72 pod subnet CIDR and the CNI configurations on nodes defined /72 subnets. When I started up kube-router, it created a new CNI config, that created a new bridge and /88 subnet, on each node. Should I have created the cluster w/o specifying the bridge CNI plugin (and static routes for pod networks to communicate with each node)?
Second, it looks like the SyncNodeIPSets() has ipset call create with this /88 info and by default, ipset is doing IPv4. Looks like "family inet6" needs to be specified (thanks @murali-reddy). Here is the error:
Third, there is another error in kube-router, which combined with the previous error causes it to fail to start. The message is:
The netstat output shows some listening (on this host):
On the first issue, in syncNodeIPSets() I tried ensuring that the Create() call would specify that the IP family is IPv6. However, I found that the Create() method was not called (there is already a psSet and it was using ipv4. It then calls Refresh() with the IPv6 cluster CIDRs for the pod ets on each node, and complains of the incompatibility.
Not sure where the initial sets are created, so that they can be made IPv6.
P.S. I'll be on PTO for a week or so, but will resume investigation, when I get back.
Here are some additional things that I see with the current kube-router code...
I've done some hacks to the code to force the above to use IPv6 equivalents.
The current issue I have is that the GBP server does not start. For IPv4 mode, it seems like the local address list passed to the bgp server that is started has [ 10.192.0.2, ::], for IPv4 and IPv6. I see listening on the V4 address port 179, and listening on V6 any address for port 179 and two other ports (I have 3 nodes).
I'm unsure how to set the local address list, when running in IPv6 (only) mode. I've done some guessing, without any success [fd00:20::2, fd00:20::2], [::, ::], [::1, ::1], [fd00:20::2, ::], [::, fd00:20::2], each saying port 179 is in use. If I try to use [0.0.0.0, fd00:20::2], I get a panic saying that the slice bounds is out o range.
Can anyone explain how the local address list should be configured for use in IPv6 mode? Do I need to force an IPv4 address on the interface (it is IPv6 only right now)?
@rearden-steel I belive no. Service proxy part is not yet compatible with IPv6.
https://github.com/cloudnativelabs/kube-router/blob/master/docs/ipv6.md is the up to date status.