Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPv6 support #307

Open
jpds opened this issue Feb 13, 2018 · 24 comments
Open

IPv6 support #307

jpds opened this issue Feb 13, 2018 · 24 comments
Labels

Comments

@jpds
Copy link
Contributor

@jpds jpds commented Feb 13, 2018

IPv6 should be moving to beta in 1.10 and is in alpha in 1.9 - does kube-router have any integration with the protocol version?

@murali-reddy
Copy link
Member

@murali-reddy murali-reddy commented Feb 14, 2018

@jpds honestly not much thought went in to supporting IPv6 at this point. But looks like at some point kube-router should start supporting it. If there is need from multiple users then cetainly support for IPv6 can be prioiritzed.

@geosword
Copy link

@geosword geosword commented Feb 15, 2018

+1

@uablrek
Copy link
Contributor

@uablrek uablrek commented May 1, 2018

Please consider it. Ipvs, gobgb and ip6tables are supporting ipv6. I think the effort is smaller than one think since the commands are basically the same but with other addresses (don't know about the netlink though).

@murali-reddy
Copy link
Member

@murali-reddy murali-reddy commented May 6, 2018

https://github.com/vishvananda/netlink libray that is used by kube-router supports IPV6 as well.

@telmich
Copy link

@telmich telmich commented May 12, 2018

+1 - looking forward very much to ipv6 support in kube-router (we are on ipv6 only networks)

@murali-reddy
Copy link
Member

@murali-reddy murali-reddy commented May 14, 2018

There is backlog of issue currently for 0.2 release, We can work on this work 0.3 release.

Meanwhile I am happy to accept any partial fixes towards IPv6 support.

@telmich
Copy link

@telmich telmich commented May 16, 2018

What is exactly missing / needed in kube-router to fully support IPv6?

@pmichali
Copy link

@pmichali pmichali commented Jun 20, 2018

As a follow-up to issue #470 I've changed the address family temporarily and hit a few more issues/concerns/unexpected items, and figured I'd document them here FYI.

First, I started with an existing cluster, that had bridge CNI plugin. The config called out a /72 pod subnet CIDR and the CNI configurations on nodes defined /72 subnets. When I started up kube-router, it created a new CNI config, that created a new bridge and /88 subnet, on each node. Should I have created the cluster w/o specifying the bridge CNI plugin (and static routes for pod networks to communicate with each node)?

Second, it looks like the SyncNodeIPSets() has ipset call create with this /88 info and by default, ipset is doing IPv4. Looks like "family inet6" needs to be specified (thanks @murali-reddy). Here is the error:

E0620 11:42:34.532817       1 network_routes_controller.go:110] Failed initial ipset setup: Failed to sync Pod Subnets ipset: ipset v6.34: Syntax error: '88' is out of range 0-32
E0620 11:42:34.551302       1 network_routes_controller.go:129] Failed to enable required policy based routing: Failed to add ip rule due to: exit status 2

Third, there is another error in kube-router, which combined with the previous error causes it to fail to start. The message is:

I0620 11:42:34.557856       1 network_routes_controller.go:194] Starting network route controller
E0620 11:42:34.705180       1 network_routes_controller.go:200] Failed to start node BGP server: Failed to start BGP server due to : listen tcp6 [::]:179: bind: address already in use

The netstat output shows some listening (on this host):

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 fd00:20::2:179          :::*                    LISTEN      -
@pmichali
Copy link

@pmichali pmichali commented Jun 20, 2018

On the first issue, in syncNodeIPSets() I tried ensuring that the Create() call would specify that the IP family is IPv6. However, I found that the Create() method was not called (there is already a psSet and it was using ipv4. It then calls Refresh() with the IPv6 cluster CIDRs for the pod ets on each node, and complains of the incompatibility.

Not sure where the initial sets are created, so that they can be made IPv6.

P.S. I'll be on PTO for a week or so, but will resume investigation, when I get back.

@pmichali
Copy link

@pmichali pmichali commented Jul 9, 2018

Here are some additional things that I see with the current kube-router code...

  • The docker image has iptables, but not ip6tables
  • In controllers/routing/pbr.go it uses ip commands, but is not using the "-6" argument for IPv6.
  • In controllers/routing/pod_egress.go it creates an iptables object, instead of an ip6tables object

I've done some hacks to the code to force the above to use IPv6 equivalents.

The current issue I have is that the GBP server does not start. For IPv4 mode, it seems like the local address list passed to the bgp server that is started has [ 10.192.0.2, ::], for IPv4 and IPv6. I see listening on the V4 address port 179, and listening on V6 any address for port 179 and two other ports (I have 3 nodes).

I'm unsure how to set the local address list, when running in IPv6 (only) mode. I've done some guessing, without any success [fd00:20::2, fd00:20::2], [::, ::], [::1, ::1], [fd00:20::2, ::], [::, fd00:20::2], each saying port 179 is in use. If I try to use [0.0.0.0, fd00:20::2], I get a panic saying that the slice bounds is out o range.

Can anyone explain how the local address list should be configured for use in IPv6 mode? Do I need to force an IPv4 address on the interface (it is IPv6 only right now)?

@pmichali
Copy link

@pmichali pmichali commented Jul 23, 2018

I have not been able to figure out how to pass the right arguments for the local listen addresses to the go routine that starts goBGP, when operating in IPv6 mode.

@murali-reddy
Copy link
Member

@murali-reddy murali-reddy commented Jul 23, 2018

@pmichali ok let me give it a try.

GoBGP folks also has a slack, https://slackin-gobgp.mybluemix.net they are pretty responsive.

@murali-reddy
Copy link
Member

@murali-reddy murali-reddy commented Oct 14, 2018

@aauren
Copy link
Collaborator

@aauren aauren commented Apr 24, 2020

Bumping this up since IPv6 is now fully supported as of Kubernetes 1.16. We'll probably want to revisit this soon.

@rearden-steel
Copy link

@rearden-steel rearden-steel commented May 22, 2020

Is it possible now to use ipv6 as an externalIP with kube-router? @murali-reddy

@murali-reddy
Copy link
Member

@murali-reddy murali-reddy commented May 26, 2020

@rearden-steel I belive no. Service proxy part is not yet compatible with IPv6.

https://github.com/cloudnativelabs/kube-router/blob/master/docs/ipv6.md is the up to date status.

CC: @uablrek

@karibertils
Copy link

@karibertils karibertils commented Aug 29, 2020

+1 looking forward to this one

@kvaps
Copy link
Contributor

@kvaps kvaps commented Aug 29, 2020

BTW I switched from Kube-router to Cilium.
IPv6 and dual stack is made really well there

@telmich
Copy link

@telmich telmich commented Aug 29, 2020

@kvaps did you try ipv6 only in cilium?

@kvaps
Copy link
Contributor

@kvaps kvaps commented Aug 29, 2020

Well there is not much plugins which support IPv6, for now only Calico and Cilium, also Contiv.
Yes. I've tried only cilium.

@Ulexus
Copy link

@Ulexus Ulexus commented Aug 29, 2020

DANM also supports IPv6 very well. I hadn't had much luck with Cilium until 1.8. GoBGP itself supports IPv6 well.

@telmich
Copy link

@telmich telmich commented Aug 31, 2020

@kvaps Sorry, I did not mean to say "did you only test cilium", but I meant to say "did you test IPv6 only" in cilium :)

Because what is of interest for me is IPv6 only k8s clusters without any IPv4 being involved.

@kvaps
Copy link
Contributor

@kvaps kvaps commented Aug 31, 2020

I have tested dual stack, but IPv6-only should be even simpler.
You don't need to enable any future-gates in Kubernetes for that.
I'm sure, IPv6 will work out-of-box.

@telmich
Copy link

@telmich telmich commented Aug 31, 2020

@kvaps Last time I checked all the components had hardcoded IPv4 addresses for discovery, which won't work in IPv6 only environments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet