IPaddr2 may fail on LVS IPv6 configuration

[kskmori]

IPaddr2 may failed to assign an IPv6 address with 'dadfailed' status on LVS configuration, which has the same IPv6 address on lo too.

I consider that this is a blocker for the release.

The steps to reproduce is below. Step 4a. or 4b. is a failed scenario. Step 4c. is a succeeded scenario. (thanks to @nozawatm for testing it)

Diagnosis:

It fails when send_ua is called if the VIP is still 'tentative' status (accomplishing the assignment in the kernel).

The cause seems that send_ua is sending ping for waiting the tentative flag is disappeared (i.e. assingment has finished) but the ping packet makes IPv6 DAD (duplicate address detection) protocol fail in the case of LVS configuration.

(L317-L322 in IPaddr2.c)
17d9f6a#L0R317

One strange thing is that this problem was not observed on RHEL5.7 (send_ua suceeds even if it was tentative status). We are unsure that if the kernel behavior does matter or not, but anyway we have to fix it somehow.

Solution:

I think we should take another way for waiting to finish to assign the IPv6 address. Probably we are going to remove the ping loop in send_ua and alternatively use ip command to check tentative flag in the RA.

I and @nozawatm are working on this issue right now.
I would appreceate if you have any comments.

---

steps to reproduce

• 0. tested revision and environment infomation.

# more /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.1 (Santiago)
# uname -a
Linux devnode1 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
# git log -1 --oneline
66f939e Merge pull request #176 from t-matsuo/delete-unnecessary-line

• 1. set up environment variables. 2001:db8:100::101 is the virtual IP address.

# export OCF_ROOT=/usr/lib/ocf
# export OCF_RESKEY_ip=2001:db8:100::101
# export OCF_RESKEY_nic=eth0
# export OCF_RESKEY_cidr_netmask=64
# export OCF_RESKEY_lvs_ipv6_addrlabel=true

• 2. assign VIP to lo as LVS configuration.

# ip -o -6 addr add $OCF_RESKEY_ip/128 dev lo
# ip -o -6 addr show dev lo
1: lo    inet6 2001:db8:100::101/128 scope global \       valid_lft forever preferred_lft forever
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever

• 3. make sure the VIP is not assigned to eth0 yet

# ip -o -6 addr show dev eth0
2: eth0    inet6 2001:db8:100:0:5054:ff:fe11:0/64 scope global dynamic \       valid_lft 86128sec preferred_lft 14128sec
2: eth0    inet6 fe80::5054:ff:fe11:0/64 scope link \       valid_lft forever preferred_lft forever

• 4a. assign the VIP to eth0 (using IPaddr2 RA). FAILED as 'dadfailed'

# /usr/lib/ocf/resource.d/heartbeat/IPaddr2 start ; echo $?
INFO: Adding IPv6 address label prefix 2001:db8:100::101 label 99
INFO: Adding inet6 address 2001:db8:100::101/64 to device eth0
INFO: Bringing device eth0 up
INFO: /usr/lib64/heartbeat/send_ua -i 200 -c 5 2001:db8:100::101 64 eth0
0
# ip -o -6 addr show dev eth0
2: eth0    inet6 2001:db8:100::101/64 scope global tentative dadfailed \       valid_lft forever preferred_lft forever
(...)

• 4b. (alternative way to reproduce) assign the VIP to eth0 using ip command directly. FAILED as 'dadfailed' if send_ua is called when the VIP is tentative statusl.

# ip addr add 2001:db8:100::101/64 dev eth0 ; ip -o -6 addr show dev eth0; /usr/lib64/heartbeat/send_ua 2001:db8:100::101 64 eth0
2: eth0    inet6 2001:db8:100::101/64 scope global tentative \       valid_lft forever preferred_lft forever
(...)
# ip -o -6 addr show dev eth0
2: eth0    inet6 2001:db8:100::101/64 scope global tentative dadfailed \       valid_lft forever preferred_lft forever
(...)

• 4c. SUCCEED if send_ua is called when the VIP has no tentative flag (by waiting in sleep).

# ip addr add 2001:db8:100::101/64 dev eth0 ; sleep 3; ip -o -6 addr show dev eth0; /usr/lib64/heartbeat/send_ua 2001:db8:100::101 64 eth0
2: eth0    inet6 2001:db8:100::101/64 scope global \       valid_lft forever preferred_lft forever
# ip -o -6 addr show dev eth0
2: eth0    inet6 2001:db8:100::101/64 scope global \       valid_lft forever preferred_lft forever
(...)

[dmuhamedagic]

On Sun, Nov 18, 2012 at 11:04:07PM -0800, Keisuke MORI wrote:

IPaddr2 may failed to assign an IPv6 address with 'dadfailed' status on LVS configuration, which has the same IPv6 address on lo too.

I consider that this is a blocker for the release.

The steps to reproduce is below. Step 4a. or 4b. is a failed scenario. Step 4c. is a succeeded scenario. (thanks to @nozawatm for testing it)

Diagnosis:

It fails when send_ua is called if the VIP is still 'tentative' status (accomplishing the assignment in the kernel).

The cause seems that send_ua is sending ping for waiting the tentative flag is disappeared (i.e. assingment has finished) but the ping packet makes IPv6 DAD (duplicate address detection) protocol fail in the case of LVS configuration.

The kernel should know where the ICMP packets are coming from.
Did you watch the network with tcpdump?

(L317-L322 in IPaddr2.c)
17d9f6a#L0R317

One strange thing is that this problem was not observed on RHEL5.7 (send_ua suceeds even if it was tentative status). We are unsure that if the kernel behavior does matter or not, but anyway we have to fix it somehow.

Neither does it happen with SLE11SP2, kernel 3.0.42. I guess it
does depend on the kernel.

Solution:

I think we should take another way for waiting to finish to assign the IPv6 address. Probably we are going to remove the ping loop in send_ua and alternatively use ip command to check tentative flag in the RA.

I guess that that's also a possibility.

[kskmori]

The cause seems that send_ua is sending ping for waiting the tentative flag is disappeared (i.e. assingment has finished) but the ping packet makes IPv6 DAD (duplicate address detection) protocol fail in the case of LVS configuration.

The kernel should know where the ICMP packets are coming from. Did you watch the network with tcpdump?

Yes (attached below). Ping packets themselves work as usual. The problem is a neighbor solicitation packet for DAD can not be seen on the wire in the case of the failed scenario. I suspect that succeeding ping to lo makes the kernel think as the address has already assigned hence DAD fails.

One strange thing is that this problem was not observed on RHEL5.7 (send_ua suceeds even if it was tentative status).

Let me correct this; RHEL5.7 did not work properly either. 'dadfailed' was not displayed on RHEL5.7 but the kernel log reports that it has detected the address duplication and the 'tentative' status remains. The connection between other nodes may work but apparently it is not right status.

I think we should take another way for waiting to finish to assign the IPv6 address. Probably we are going to remove the ping loop in send_ua and alternatively use ip command to check tentative flag in the RA.

I guess that that's also a possibility.

I and @nozawatm are now testing the patch along with this.
Please allow us for one more day.

Regards,

---

Failed scenario. (4b.)

# tcpdump -i lo -n icmp6
10:16:54.490512 IP6 2001:db8:100::101 > 2001:db8:100::101: ICMP6, echo request, seq 0, length 64
10:16:54.490526 IP6 2001:db8:100::101 > 2001:db8:100::101: ICMP6, echo reply, seq 0, length 64

# tcpdump -i eth0 -n icmp6
10:16:54.491331 IP6 2001:db8:100::101 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:db8:100::101, length 32
(5 times)

Succeeded scenario. (4c.)

# tcpdump -i lo -n icmp6
10:18:06.904199 IP6 2001:db8:100::101 > 2001:db8:100::101: ICMP6, echo request, seq 0, length 64
10:18:06.904208 IP6 2001:db8:100::101 > 2001:db8:100::101: ICMP6, echo reply, seq 0, length 64

# tcpdump -i eth0 -n icmp6
10:18:04.534835 IP6 :: > ff02::1:ff00:101: ICMP6, neighbor solicitation, who has 2001:db8:100::101, length 24
10:18:06.904327 IP6 2001:db8:100::101 > ff02::1: ICMP6, neighbor advertisement, tgt is 2001:db8:100::101, length 32
(5 times)

[dmuhamedagic]

On Tue, Nov 20, 2012 at 02:47:48AM -0800, Keisuke MORI wrote:

I and @nozawatm are now testing the patch along with this.
Please allow us for one more day.

Of course.

[kskmori]

Closed as the patch is submitted in pull request #181.

[kskmori]

Just for an additional note:

According to our kernel experts, this issue is fixed by the commit below so the recent kernel should not be affected:

• http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=patch;h=bd015928bb1713691068c4d0d159afccbaf0f8c0

Looking at tentative status should be more reliable than pinging anyway.