Dual-stack with Kube router 2

[ncopa]

Description

Update kube-router to 2.0.1 and enable ipv6

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

How Has This Been Tested?

• Manual test
• Auto test added

Checklist:

• My code follows the style guidelines of this project
• My commit messages are signed-off
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

[github-actions]

The PR is marked as stale since no activity has been recorded in 30 days

[github-actions]

This pull request has merge conflicts that need to be resolved.

[github-actions]

This pull request has merge conflicts that need to be resolved.

[github-actions]

The PR is marked as stale since no activity has been recorded in 30 days

[twz123]

NB: This is the error that's reported by containerd in the failed inttest:

failed to setup network for sandbox: plugin type="bridge" name="kubernetes" failed (add): failed to set bridge addr: could not add IP address to "kube-bridge": permission denied

[github-actions]

This pull request has merge conflicts that need to be resolved.

[twz123]

/xref #3814

[twz123]

How does this work with dynamic configuration? We'd need to restart kubelet, I guess, since we're actively changing flags based on dual stack?

Hmm, what happens for Calico when toggling dual stack? We're not (yet) reconciling the workers at all... (#1806)

[twz123]

Suggested change

	ipv4 = addr.IP
	ipv4 = addr.IP.To4()

[ncopa]

What would this solve?

[twz123]

It would ensure that ipv4 is actually in its four byte form, so that len(ipv4) == IPv4len. Otherwise, it could be in its 16 byte form, i.e. len(ipv4) == IPv6len. This is a nit, but it feels a bit saner to have a v4 IP returned as 4 bytes.

[ncopa]

I think we only care what the address family is. How the buffers are allocated under the hood is completely irrelevant.

[twz123]

This needs attention from kubectl side, as well. I'm a bit on the fence with the IP auto-detection, because this makes it so much harder for k0sctl to do the right thing, but maybe I'm overstating the problem. WDYT @kke?

[ncopa]

This needs attention from kubectl side, as well. I'm a bit on the fence with the IP auto-detection, because this makes it so much harder for k0sctl to do the right thing, but maybe I'm overstating the problem. WDYT @kke?

Would you prefer that we required to specify the ipv4 and ipv6 node address in k0s config? How would that work for dynamic addresses? There is no guarantee that the node gets same ipv4 address after a reboot.

[twz123]

Would you prefer that we required to specify the ipv4 and ipv6 node address in k0s config?

I think there's two separate things which bother me:

First: There's this schism between single-stack and dual-stack. There is IP auto-detection in kubelet for v4 and v6, but not for both at the same time (which would be reasonable in dual-stack mode, IMO). So k0s has to do it, and is responsible for getting it right, even it doesn't really care about the detected IPs itself. Although I would rather see someone else (i.e. upstream) maintain this 🙃, there is still the option to set the IPs via --node-ip as kubelet extra args if users want to set the IPs (or need to set them since the auto-detection doesn't do the right thing for them). I think that's a fair compromise.

Second, and this is something that needs clarification and probably a fix on the k0sctl or on both k0sctl and k0s side: k0sctl sets --node-ip based on privateInterface. This also needs to get dual-stack support. So people will need to specify exactly zero or two IPs in privateInterface if they are using a dual-stack cluster. I don't think k0sctl can accept more than one IP in this setting, so this is a blocker for use cases like k0sproject/k0sctl#736. One way to fix this on k0s's end might be to look at the user-provided --node-ip value and, if it doesn't have IP addresses for v4 and v6, add the missing ones via auto-detection.

How would that work for dynamic addresses? There is no guarantee that the node gets same ipv4 address after a reboot.

Is that even a thing, workers with dynamic addresses? Maybe. Would require a k0s restart on IP address changes, though. Not sure about the running containers. They probably need to be restarted, as well.

[ncopa]

I would rather see someone else (i.e. upstream) maintain this 🙃

I agree. This is something that should be fixed in kubelet. But for now we will have to work around it in k0s.

k0sctl sets --node-ip based on privateInterface. This also needs to get dual-stack support. So people will need to specify exactly zero or two IPs in privateInterface if they are using a dual-stack cluster.

Yes, and this is due to upstream kubelet behavior.

I don't think k0sctl can accept more than one IP in this setting, so this is a blocker for use cases like k0sproject/k0sctl#736. One way to fix this on k0s's end might be to look at the user-provided --node-ip value and, if it doesn't have IP addresses for v4 and v6, add the missing ones via auto-detection.

I'd rather fix kubelet

Is that even a thing, workers with dynamic addresses? Maybe. Would require a k0s restart on IP address changes, though. Not sure about the running containers. They probably need to be restarted, as well.

I would expect most workers use DHCP for ipv4. Not sure if we care about address changes on dhcp lease expiry runtime. But I think that address change on reboot is something to be expected. (which probably means we shouldn't set the address from k0sctl?)