XCAT_IPv6_support_on_Linux

Table of Contents

• Overview
• Prerequisite
• Configure IPv6 addresses and gateway on the compute nodes for Ethernet adapters
• Configure IPv6 addresses on the compute nodes for Ethernet adapters
• Configure IPv6 routing on the compute nodes
• Setup the ipforward for IPv6 on MN
• Other Design Considerations

Design_Warning

Note: this design is only for Linux, IPv6 support on AIX will be covered by a separate design.

Overview

This item icludes only the "black box" model, which means:

• The os provisioning is done through ipv4 network.
• The ethernet adapters and Infiniband adapters on the compute nodes could be configured with ipv4 only, or ipv6 only, or both
• The compute nodes could use ipv6 default gateway to communicate with the external ipv6 nodes.
• The compute nodes could be configured to communicate with different ipv6 subnets through different gateway.
• The management node should have ipv6 forwarding enabled in case the compute nodes will use the management node as the ipv6 gateway.

The specific support xcat needs to provide:

• Use makehosts command to setup IPv6 hostnames in /etc/hosts
• Use confignics postscript to configure the IPv6 addresses on the compute nodes, for both Ethernet and Infiniband
• Use confignics postscript to configure IPv6 gateway on the compute nodes, for both Ethernet and Infiniband
• Use makeroutes to configure IPv6 routing on the compute nodes, for both Ethernet and Infiniband
• xcatconfig will configure IPv6 forwarding if there is any non-link-local IPv6 network setup on the management node.

IPv6 support with Ubuntu is not covered by this doc, it will be addressed in a separate item

Prerequisite

To configure IPv6 environment, the perl IPv6 packages need to be installed on the management node:

 yum install perl-IO-Socket-INET6 perl-IO-Socket-SSL perl-Socket6
 service xcatd stop
 service xcatd start

Configure IPv6 addresses and gateway on the compute nodes for Ethernet adapters

1. Plan the IPv6 addresses for the compute nodes

For the IPv6 addresses allocation, there are two choices: either defining the ip addresses in the nics table manually, or use the DHCPv6/RA to assign dynamic IPv6 addresses and use ddns to map the hostnames and nic. Setting up the DHCPv6/RA to assign IPv6 addresses is beyond the scope of this doc, this doc will only cover the first scenario, i.e., defining the ip addresses in the nics table manually.

2. Specify the node ip addresses and gateway configuration in nics table.

Here is an example on how to configure the IPv4/IPv6 addresses for the Ethernet adapters in the nics table.

 [root@ls21n01 ~]# tabdump nics
 #node,nicips,nichostnamesuffixes,nictypes,niccustomscripts,nicnetworks,nicaliases,comments,disable
 "ipv6cn1","eth0!10.1.89.7|fd56::214:5eff:fe15:849b|2000::214:5eff:fe15:849b,eth1!11.1.89.7|fd57::214:5eff:fe15:849b|2001::214:5eff:fe15:849b,eth2!12.1.89.7|fd58::214:5eff:fe15:849b|2002::214:5eff:fe15:849b","eth0!|-eth0-ipv6-1|-eth0-ipv6-2,eth1!-eth1|-eth1-ipv6-1|-eth1-ipv6-2,eth2!-eth2|-eth2-ipv6-1|-eth2-ipv6-2","eth0!Ethernet,eth1!Ethernet,eth2!Ethernet",,"eth0!10_1_0_0-255_255_0_0|fd56::/64|2000::/64,eth1!11_1_0_0-255_255_0_0|fd57::/64|2001::/64,eth2!12_1_0_0-255_255_0_0|fd58::/64|2002::/64",,,
   [root@ls21n01 ~]# 

3. Add network entries for the nics in the networks table

 [root@ls21n01 ~]# tabdump networks
 #netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,logservers,dynamicrange,staticrange,staticrangeincrement,nodehostname,ddnsdomain,vlanid,domain,comments,disable
 "10_1_0_0-255_255_0_0","10.1.0.0","255.255.0.0","eth1","<xcatmaster>",,"10.1.0.218",,,,,,,,,,"clusters.com",,
 "11_1_0_0-255_255_0_0","11.1.0.0","255.255.0.0","eth1",,,,,,,,,,,,,"clusters.com",,
 "12_1_0_0-255_255_0_0","12.1.0.0","255.255.0.0","eth1",,,,,,,,,,,,,"clusters.com",,
 "fd56::/64","fd56::/64","/64","eth0","fd56::214:5eff:fe15:1",,,,,,,,,,,,"clusters.com",,
 "fd57::/64","fd57::/64","/64","eth1",,,,,,,,,,,,,"clusters.com",,
 "fd58::/64","fd58::/64","/64","eth2",,,,,,,,,,,,,"clusters.com",,
 "2000::/64","2000::/64","/64","eth0",,,,,,,,,,,,,"clusters.com",,
 "2001::/64","2001::/64","/64","eth1",,,,,,,,,,,,,"clusters.com",,
 "2002::/64","2002::/64","/64","eth2",,,,,,,,,,,,,"clusters.com",,
 [root@ls21n01 ~]# 

In this example, the fd56::214:5eff:fe15:1 is used as the IPv6 default gateway for the compute node ipv6cn1. Please be aware that only one IPv6 default gateway could be specified, if you would like to setup different gateways for different IPv6 subnets, see the section "Configure IPv6 routing on the compute nodes" for more details.

4. Setup /etc/hosts for the IPv6 entries

The makehosts is able to setup both the IPv4 hostnames and IPv6 hostnames in /etc/hosts based on information stored in the nics table, here is an example:

4.1. Run makehosts to setup the IPv6 entries in /etc/hosts

 makehosts ipv6cn1

4.2. Check the IPv6 entries are setup correctly in /etc/hosts

 [root@ls21n01 ~]# cat /etc/hosts | grep ipv6cn1
 12.1.89.7 ipv6cn1-eth2 ipv6cn1-eth2.clusters.com  
 fd58::214:5eff:fe15:849b ipv6cn1-eth2-ipv6-1 ipv6cn1-eth2-ipv6-1.clusters.com  
 2002::214:5eff:fe15:849b ipv6cn1-eth2-ipv6-2 ipv6cn1-eth2-ipv6-2.clusters.com  
 11.1.89.7 ipv6cn1-eth1 ipv6cn1-eth1.clusters.com  
 fd57::214:5eff:fe15:849b ipv6cn1-eth1-ipv6-1 ipv6cn1-eth1-ipv6-1.clusters.com  
 2001::214:5eff:fe15:849b ipv6cn1-eth1-ipv6-2 ipv6cn1-eth1-ipv6-2.clusters.com  
 10.1.89.7 ipv6cn1 ipv6cn1.clusters.com  
 fd56::214:5eff:fe15:849b ipv6cn1-eth0-ipv6-1 ipv6cn1-eth0-ipv6-1.clusters.com  
 2000::214:5eff:fe15:849b ipv6cn1-eth0-ipv6-2 ipv6cn1-eth0-ipv6-2.clusters.com  
 [root@ls21n01 ~]# 

5. Configure IPv4/IPv6 addresses and IPv6 default gateway on the compute node

The postscript confignics can configure the IPv4 and IPv6 addresses and gateway on the compute nodes. It could be called through whatever possible ways:

5.1 Setup the IPv4/IPv6 addresses and IPv6 default gateway on the compute node during operating system provisioning

 chdef ipv6cn1 -p postscripts=confignics
 or
 chdef ipv6cn1 -p postscripts="confignics -s"

The "-s" flag specified with confignics indicates that the installation nic should be setup as static configuration using the information in the nics table.

 nodeset ipv6cn1 osimage=xxx
 rpower ipv6cn1 reset

5.2 Setup the IPv4/IPv6 addresses and IPv6 default gateway on the compute node through updatenode

 updatenode ipv6cn1 -P confignics
 or
 updatenode ipv6cn1 -P "confignics -s"

The "-s" flag specified with confignics indicates that the installation nic should be setup as static configuration using the information in the nics table.

6. Verify the configuration on the compute nodes

When the confignics finishes setup the IPv4 and IPv6 addresses and gateway on the compute nodes, the following commands could be used to verify the configuration:

 [root@ipv6cn1 ~]# ifconfig
 eth0    Link encap:Ethernet  HWaddr 42:C6:0A:01:59:07  
         inet addr:10.1.89.7  Bcast:10.1.255.255  Mask:255.255.0.0
         inet6 addr: 2000::214:5eff:fe15:849b/64 Scope:Global
         inet6 addr: fd56::214:5eff:fe15:849b/64 Scope:Global
         inet6 addr: fe80::40c6:aff:fe01:5907/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:14663 errors:0 dropped:0 overruns:0 frame:0
         TX packets:959 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000 
         RX bytes:1053278 (1.0 MiB)  TX bytes:130254 (127.2 KiB)
 
 eth1    Link encap:Ethernet  HWaddr 42:56:0A:01:59:07  
         inet addr:11.1.89.7  Bcast:11.1.255.255  Mask:255.255.0.0
         inet6 addr: fd55:faaf:e1ab:336:4056:aff:fe01:5907/64 Scope:Global
         inet6 addr: fe80::4056:aff:fe01:5907/64 Scope:Link
         inet6 addr: 2001::214:5eff:fe15:849b/64 Scope:Global
         inet6 addr: fd57::214:5eff:fe15:849b/64 Scope:Global
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:12833 errors:0 dropped:0 overruns:0 frame:0
         TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000 
         RX bytes:824536 (805.2 KiB)  TX bytes:1642 (1.6 KiB)
 
 eth2    Link encap:Ethernet  HWaddr 42:0D:0A:01:59:07  
         inet addr:12.1.89.7  Bcast:12.1.255.255  Mask:255.255.0.0
         inet6 addr: 2002::214:5eff:fe15:849b/64 Scope:Global
         inet6 addr: fd58::214:5eff:fe15:849b/64 Scope:Global
         inet6 addr: fe80::400d:aff:fe01:5907/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:34 errors:0 dropped:0 overruns:0 frame:0
         TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000 
         RX bytes:3696 (3.6 KiB)  TX bytes:1226 (1.1 KiB)
 [root@ipv6cn1 ~]# 





 [root@ipv6cn1 ~]# ip -6 route show  default
 default via fd56::214:5eff:fe15:1 dev eth0  metric 1  mtu 1500 advmss 1440 hoplimit 4294967295
 default via fe80::226:88ff:fe57:b7f0 dev eth1  proto kernel  metric 1024  expires 0sec mtu 1500 advmss 1440 hoplimit 64
 [root@ipv6cn1 ~]#

Configure IPv6 addresses on the compute nodes for Ethernet adapters

Here is an example on how to configure the IPv4/IPv6 addresses for the Infiniband adapters in the nics table.

 [root@ls21n01 ~]# tabdump nics
 "dx360m3n06","ib0!11.1.89.10|21.1.89.10|fd57::214:5eff:fe15:8496|2000::214:5eff:fe15:8496,ib1!12.1.89.10|22.1.89.10|fd58::214:5eff:fe15:8496|2001::214:5eff:fe15:8496","ib0!-ib0|-ib0-2|-ib0-ipv6-1|-ib0-ipv6-2,ib1!-ib1|-ib1-2|-ib1-ipv6-1|-ib1-ipv6-2","ib0!Infiniband,ib1!Infiniband",,"ib0!11_1_0_0-255_255_0_0|21_1_0_0-255_255_0_0|fd57::/64|2000::/64,ib1!12_1_0_0-255_255_0_0|22_1_0_0-255_255_0_0|fd58::/64|2001::/64",,,
 [root@ls21n01 ~]# 

3. Add network entries for the nics in the networks table

 [root@ls21n01 ~]# tabdump networks
 #netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,logservers,dynamicrange,staticrange,staticrangeincrement,nodehostname,ddnsdomain,vlanid,domain,comments,disable
 "11_1_0_0-255_255_0_0","11.1.0.0","255.255.0.0","ib0",,,,,,,,,,,,,"clusters.com",,
 "21_1_0_0-255_255_0_0","21.1.0.0","255.255.0.0","ib0",,,,,,,,,,,,,"clusters.com",,
 "12_1_0_0-255_255_0_0","12.1.0.0","255.255.0.0","ib1",,,,,,,,,,,,,"clusters.com",,
 "22_1_0_0-255_255_0_0","22.1.0.0","255.255.0.0","ib1",,,,,,,,,,,,,"clusters.com",,
 "fd57::/64","fd57::/64","/64","ib0",,,,,,,,,,,,,"clusters.com",,
 "fd58::/64","fd58::/64","/64","ib1",,,,,,,,,,,,,"clusters.com",,
 "2000::/64","2000::/64","/64","ib0",,,,,,,,,,,,,"clusters.com",,
 "2001::/64","2001::/64","/64","ib1",,,,,,,,,,,,,"clusters.com",,
 [root@ls21n01 ~]# 

4. Setup /etc/hosts for the Infiniband IPv6 entries

The makehosts is able to setup both the IPv4 hostnames and IPv6 hostnames in /etc/hosts based on information stored in the nics table, here is an example:

4.1. Run makehosts to setup the IPv6 entries in /etc/hosts

 makehosts dx360m3n06

4.2. Check the IPv6 entries are setup correctly in /etc/hosts

 [root@ls21n01 ~]# cat /etc/hosts | grep dx360m3n06
 10.1.0.236 dx360m3n06.clusters.com dx360m3n06
 12.1.89.10 dx360m3n06-ib2 dx360m3n06-ib2.clusters.com 
 22.1.89.10 dx360m3n06-ib2-2 dx360m3n06-ib2-2.clusters.com 
 fd58::214:5eff:fe15:8496 dx360m3n06-ib2-ipv6-1 dx360m3n06-ib2-ipv6-1.clusters.com 
 2001::214:5eff:fe15:8496 dx360m3n06-ib2-ipv6-2 dx360m3n06-ib2-ipv6-2.clusters.com 
 11.1.89.10 dx360m3n06-ib1 dx360m3n06-ib1.clusters.com 
 21.1.89.10 dx360m3n06-ib1-2 dx360m3n06-ib1-2.clusters.com 
 fd57::214:5eff:fe15:8496 dx360m3n06-ib1-ipv6-1 dx360m3n06-ib1-ipv6-1.clusters.com 
 2000::214:5eff:fe15:8496 dx360m3n06-ib1-ipv6-2 dx360m3n06-ib1-ipv6-2.clusters.com 
 [root@ls21n01 ~]# 

5. Configure IPv4/IPv6 addresses for the Infiniband adapters

The postscript confignics can configure the IPv4 and IPv6 addresses for the Infiniband adapters. It could be called through whatever possible ways:

5.1 Setup the IPv4/IPv6 addresses for the Infiniband adapters during operating system provisioning

 chdef dx360m3n06 -p postscripts=confignics


 nodeset ipv6cn1 osimage=xxx
 rpower ipv6cn1 reset

5.2 Setup the IPv4/IPv6 addresses and IPv6 default gateway on the compute node through updatenode

 updatenode ipv6cn1 -P confignics

6. Verify the configuration on the compute nodes

When the confignics finishes setup the IPv4 and IPv6 addresses for the Infiniband adapters, the following commands could be used to verify the configuration:

 [root@dx360m3n06 ~]# ip addr show dev ib0
 9: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP qlen 1024
   link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:01:00:02:c9:03:00:4e:47:cf brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
   inet 11.1.89.10/16 brd 11.1.255.255 scope global ib0
   inet 21.1.89.10/16 brd 21.1.255.255 scope global ib0:2
   inet6 2000::214:5eff:fe15:8496/64 scope global 
      valid_lft forever preferred_lft forever
   inet6 fd57::214:5eff:fe15:8496/64 scope global 
      valid_lft forever preferred_lft forever
   inet6 fe80::202:c903:4e:47cf/64 scope link 
      valid_lft forever preferred_lft forever
 [root@dx360m3n06 ~]# ip addr show dev ib1
 10: ib1: <BROADCAST,MULTICAST,UP> mtu 65520 qdisc pfifo_fast state UNKNOWN qlen 1024
   link/infiniband 80:00:00:49:fe:80:00:00:00:00:00:00:00:02:c9:03:00:4e:47:d0 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
   inet 12.1.89.10/16 brd 12.1.255.255 scope global ib1
   inet 22.1.89.10/16 brd 22.1.255.255 scope global ib1:2
   inet6 2001::214:5eff:fe15:8496/64 scope global tentative 
      valid_lft forever preferred_lft forever
   inet6 fd58::214:5eff:fe15:8496/64 scope global tentative 
      valid_lft forever preferred_lft forever
 [root@dx360m3n06 ~]# 

Note: this item only covers the IPoIB IPv6 configuration on the compute nodes from the IP layer perspective, the IPv6 in IPoIB configuration depends on the Infiniband IPv6 support structure, like Infiniband switches, operating systems and device drivers, you might need to do more configuration for IPv6 work in IPoIB environment, or even worse, it is possible the IPv6 could not work with some specific IPoIB configuration.

Configure IPv6 routing on the compute nodes

The IPv6 routing setup on the compute nodes follows the same procedure as the IPv6 routing setup.

1. Add the network routes in the routes table

 [root@ls21n01 ~]# tabdump routes
 #routename,net,mask,gateway,ifname,comments,disable
 "13route","13.1.0.0","255.255.0.0","11.1.89.1",,,
 "14route","14.1.0.0","255.255.0.0","12.1.89.1",,,
 "fd59route","fd59::/64","/64","fd56::214:5eff:fe15:1","eth0",,
 [root@ls21n01 ~]# 

2. Associate the routes to the compute nodes

 [root@ls21n01 ~]# chdef ipv6cn1 routenames=13route,14route,fd59route
 1 object definitions have been created or modified.
 [root@ls21n01 ~]#  

3. Setup up the routing

To setup the routes on the management node:

 makeroutes -r 13route,14route,fd59route

To setup the routes on the compute nodes during operating system provisioning:

 chdef ipv6cn1 -p postscripts=setroute
 nodeset ipv6cn1 osimage
 rpower ipv6cn1 reset

To setup the routes on the compute nodes when the compute nodes are up and running:

 makeroutes ipv6cn1 -r 13route,14route,fd59route

To remove the routes on the management node:

 makeroutes -d -r 13route,14route,fd59route

To remove the routes on the compute nodes:

 makeroutes ipv6cn1 -d -r 13route,14route,fd59route

4. Verify the routing setup

When the makeroutes finishes the routes setup on the management node or compute nodes, the following commands could be used to verify the routes setup.

 [root@ipv6cn1 ~]# ip route show
 14.1.0.0/16 via 12.1.89.1 dev eth2 
 12.1.0.0/16 dev eth2  proto kernel  scope link  src 12.1.89.7 
 10.1.0.0/16 dev eth0  proto kernel  scope link  src 10.1.89.7 
 13.1.0.0/16 via 11.1.89.1 dev eth1 
 169.254.0.0/16 dev eth0  scope link  metric 1002 
 169.254.0.0/16 dev eth1  scope link  metric 1003 
 169.254.0.0/16 dev eth2  scope link  metric 1004 
 11.1.0.0/16 dev eth1  proto kernel  scope link  src 11.1.89.7 
 [root@ipv6cn1 ~]# 


 [root@ipv6cn1 ~]# ip -6 route show fd59::/64
 fd59::/64 via fd56::214:5eff:fe15:1 dev eth0  metric 1024  mtu 1500 advmss 1440 hoplimit 4294967295
 [root@ipv6cn1 ~]# 

On RedHat:

 [root@ipv6cn1 ~]# cat /etc/sysconfig/static-routes
 # xCAT_CONFIG_START
 any net 13.1.0.0 netmask 255.255.0.0 gw 11.1.89.1 
 any net 14.1.0.0 netmask 255.255.0.0 gw 12.1.89.1 
 # xCAT_CONFIG_END
 [root@ipv6cn1 ~]# cat /etc/sysconfig/static-routes-ipv6 
 # xCAT_CONFIG_START
 eth0 fd59::/64 fd56::214:5eff:fe15:1
 # xCAT_CONFIG_END
 [root@ipv6cn1 ~]# 

On SLES:

 ipv6cn2:~ # cat /etc/sysconfig/network/routes 
 default fd56::214:5eff:fe15:1 - -
 # xCAT_CONFIG_START
 13.1.0.0 11.1.89.1 255.255.0.0
 14.1.0.0 12.1.89.1 255.255.0.0 
 fd59::/64 fd56::214:5eff:fe15:1 - -
 # xCAT_CONFIG_END
 ipv6cn2:~ # 

Setup the ipforward for IPv6 on MN

If there is any IPv6 network entry is defined in the networks table, when the xCAT is installed on the management node, the IPv6 forwarding will be enabled.

To check the IPv6 forwarding on the management node:

 [root@ls21n01 ~]# cat /etc/sysctl.conf | grep net.ipv6.conf.all.forwarding
 net.ipv6.conf.all.forwarding = 1
 [root@ls21n01 ~]# cat /proc/sys/net/ipv6/conf/all/forwarding
 1
 [root@ls21n01 ~]# 

Note: this is a temporary solution, in some futher xCAT release, the network services setup on the management node will be based on the information in the servicenode table.

Other Design Considerations

   *Required approvers:  Bruce Potter
   *Affect on other components:  confignics(configeth/configib) and makeroutes for IPv4
   *External interface changes, documentation, and usability issues: use the existing confignics and makeroutes command. Documentation at [Configuring_IPv6_in_Cluster].
   *Packaging, installation, dependencies:  N/A
   *Portability and platforms (HW/SW) supported:  N/A
   *Performance and scaling considerations:  N/A
   *Migration and coexistence:  N/A
   *Serviceability: N/A
   *Security:  N/A
   *NLS and accessibility:  N/A
   *Invention protection:  N/A