New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPv6 support #307

Open
jpds opened this Issue Feb 13, 2018 · 13 comments

Comments

Projects
None yet
6 participants
@jpds
Contributor

jpds commented Feb 13, 2018

IPv6 should be moving to beta in 1.10 and is in alpha in 1.9 - does kube-router have any integration with the protocol version?

@murali-reddy

This comment has been minimized.

Member

murali-reddy commented Feb 14, 2018

@jpds honestly not much thought went in to supporting IPv6 at this point. But looks like at some point kube-router should start supporting it. If there is need from multiple users then cetainly support for IPv6 can be prioiritzed.

@geosword

This comment has been minimized.

geosword commented Feb 15, 2018

+1

@uablrek

This comment has been minimized.

Contributor

uablrek commented May 1, 2018

Please consider it. Ipvs, gobgb and ip6tables are supporting ipv6. I think the effort is smaller than one think since the commands are basically the same but with other addresses (don't know about the netlink though).

@murali-reddy

This comment has been minimized.

Member

murali-reddy commented May 6, 2018

https://github.com/vishvananda/netlink libray that is used by kube-router supports IPV6 as well.

@telmich

This comment has been minimized.

telmich commented May 12, 2018

+1 - looking forward very much to ipv6 support in kube-router (we are on ipv6 only networks)

@murali-reddy

This comment has been minimized.

Member

murali-reddy commented May 14, 2018

There is backlog of issue currently for 0.2 release, We can work on this work 0.3 release.

Meanwhile I am happy to accept any partial fixes towards IPv6 support.

@telmich

This comment has been minimized.

telmich commented May 16, 2018

What is exactly missing / needed in kube-router to fully support IPv6?

@pmichali

This comment has been minimized.

pmichali commented Jun 20, 2018

As a follow-up to issue #470 I've changed the address family temporarily and hit a few more issues/concerns/unexpected items, and figured I'd document them here FYI.

First, I started with an existing cluster, that had bridge CNI plugin. The config called out a /72 pod subnet CIDR and the CNI configurations on nodes defined /72 subnets. When I started up kube-router, it created a new CNI config, that created a new bridge and /88 subnet, on each node. Should I have created the cluster w/o specifying the bridge CNI plugin (and static routes for pod networks to communicate with each node)?

Second, it looks like the SyncNodeIPSets() has ipset call create with this /88 info and by default, ipset is doing IPv4. Looks like "family inet6" needs to be specified (thanks @murali-reddy). Here is the error:

E0620 11:42:34.532817       1 network_routes_controller.go:110] Failed initial ipset setup: Failed to sync Pod Subnets ipset: ipset v6.34: Syntax error: '88' is out of range 0-32
E0620 11:42:34.551302       1 network_routes_controller.go:129] Failed to enable required policy based routing: Failed to add ip rule due to: exit status 2

Third, there is another error in kube-router, which combined with the previous error causes it to fail to start. The message is:

I0620 11:42:34.557856       1 network_routes_controller.go:194] Starting network route controller
E0620 11:42:34.705180       1 network_routes_controller.go:200] Failed to start node BGP server: Failed to start BGP server due to : listen tcp6 [::]:179: bind: address already in use

The netstat output shows some listening (on this host):

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 fd00:20::2:179          :::*                    LISTEN      -
@pmichali

This comment has been minimized.

pmichali commented Jun 20, 2018

On the first issue, in syncNodeIPSets() I tried ensuring that the Create() call would specify that the IP family is IPv6. However, I found that the Create() method was not called (there is already a psSet and it was using ipv4. It then calls Refresh() with the IPv6 cluster CIDRs for the pod ets on each node, and complains of the incompatibility.

Not sure where the initial sets are created, so that they can be made IPv6.

P.S. I'll be on PTO for a week or so, but will resume investigation, when I get back.

@pmichali

This comment has been minimized.

pmichali commented Jul 9, 2018

Here are some additional things that I see with the current kube-router code...

  • The docker image has iptables, but not ip6tables
  • In controllers/routing/pbr.go it uses ip commands, but is not using the "-6" argument for IPv6.
  • In controllers/routing/pod_egress.go it creates an iptables object, instead of an ip6tables object

I've done some hacks to the code to force the above to use IPv6 equivalents.

The current issue I have is that the GBP server does not start. For IPv4 mode, it seems like the local address list passed to the bgp server that is started has [ 10.192.0.2, ::], for IPv4 and IPv6. I see listening on the V4 address port 179, and listening on V6 any address for port 179 and two other ports (I have 3 nodes).

I'm unsure how to set the local address list, when running in IPv6 (only) mode. I've done some guessing, without any success [fd00:20::2, fd00:20::2], [::, ::], [::1, ::1], [fd00:20::2, ::], [::, fd00:20::2], each saying port 179 is in use. If I try to use [0.0.0.0, fd00:20::2], I get a panic saying that the slice bounds is out o range.

Can anyone explain how the local address list should be configured for use in IPv6 mode? Do I need to force an IPv4 address on the interface (it is IPv6 only right now)?

@pmichali

This comment has been minimized.

pmichali commented Jul 23, 2018

I have not been able to figure out how to pass the right arguments for the local listen addresses to the go routine that starts goBGP, when operating in IPv6 mode.

@murali-reddy

This comment has been minimized.

Member

murali-reddy commented Jul 23, 2018

@pmichali ok let me give it a try.

GoBGP folks also has a slack, https://slackin-gobgp.mybluemix.net they are pretty responsive.

@murali-reddy

This comment has been minimized.

Member

murali-reddy commented Oct 14, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment