Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPv6 RA flood (THC flood_router6) causes network disconnection even after flood ceases #2977

Closed
1 of 2 tasks
cvmiller opened this issue Apr 6, 2016 · 5 comments
Closed
1 of 2 tasks
Labels
bug 🐛 Programming errors, that need preferential fixing network

Comments

@cvmiller
Copy link

cvmiller commented Apr 6, 2016

Submission type

  • Bug report
  • Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

 # /usr/lib/systemd/systemd --version
systemd 229
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

Arch Linux

In case of bug report: Expected behaviour you didn't see

Expected to see number of prefixes limited to 16 (as per /proc/sys/net/ipv6/conf/eth0/max_addresses). Instead saw more than 3000!

In case of bug report: Unexpected behaviour you saw

Ping6 ceased to see echo replies, even after RA flood was stopped. IPv6 network connectivity is lost.

In case of bug report: Steps to reproduce the problem

  1. Run THC IPv6 flood_router6 against network interface
    Used THC IPv6 version 3.0
    THC IPv6
sudo ./flood_router6 eth0 

Starting to flood network with router advertisements on eth0 (Press Control-C to end, a dot is printed for every 1000 packets):
....................................................................................................................................................................................................................................................................................................................................................................................................................^C
  1. Simultaneously run a ping6 from the device under test, and note that the destination becomes unreachable, even after the flood is stopped.
# ping6 2607:c000:8124:ca00:129a:ddff:fe54:b634
PING 2607:c000:8124:ca00:129a:ddff:fe54:b634(2607:c000:8124:ca00:129a:ddff:fe54:b634) 56 data bytes
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=2 ttl=63 time=1.38 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=3 ttl=63 time=7.98 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=4 ttl=63 time=7.95 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=5 ttl=63 time=7.99 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=6 ttl=63 time=8.12 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=7 ttl=63 time=7.98 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=8 ttl=63 time=7.95 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=9 ttl=63 time=7.96 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=10 ttl=63 time=3.61 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=11 ttl=63 time=7.93 ms
64 bytes from 2607:c000:8124:ca00:129a:ddff:fe54:b634: icmp_seq=12 ttl=63 time=7.94 ms
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=13 Destination unreachable: Address unreachable
[   81.287506] neighbour: ndisc_cache: neighbor table overflow!
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=14 Destination unreachable: Address unreachable
[   82.240875] neighbour: ndisc_cache: neighbor table overflow!
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=15 Destination unreachable: Address unreachable
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=16 Destination unreachable: Address unreachable
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=17 Destination unreachable: Address unreachable
[   85.314544] neighbour: ndisc_cache: neighbor table overflow!
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=18 Destination unreachable: Address unreachable
[   86.320098] neighbour: ndisc_cache: neighbor table overflow!
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=19 Destination unreachable: Address unreachable
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=20 Destination unreachable: Address unreachable
[   88.520568] neighbour: ndisc_cache: neighbor table overflow!
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=21 Destination unreachable: Address unreachable
[   89.422210] neighbour: ndisc_cache: neighbor table overflow!
[   89.560333] neighbour: ndisc_cache: neighbor table overflow!
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=22 Destination unreachable: Address unreachable
[   91.513214] neighbour: ndisc_cache: neighbor table overflow!
[   91.578216] neighbour: ndisc_cache: neighbor table overflow!
From 2607:c000:8124:ca05:3522:53a2:2855:a59e icmp_seq=23 Destination unreachable: Address unreachable

--- 2607:c000:8124:ca00:129a:ddff:fe54:b634 ping statistics ---
26 packets transmitted, 11 received, +11 errors, 57% packet loss, time 25264ms
rtt min/avg/max/mdev = 1.389/6.986/8.126/2.167 ms

Additional information

Journal output during flood

Apr 06 22:25:02 alarm systemd-networkd[6119]: eth0: Could not set NDisc route or address: Connection timed out
Apr 06 22:25:02 alarm systemd-networkd[6119]: eth0: Could not set NDisc route or address: Connection timed out
Apr 06 22:25:05 alarm systemd-networkd[6119]: Assertion 'link->state == LINK_STATE_SETTING_ROUTES' failed at src/network/networkd-link.c:586, function link_enter_configured(). Aborting.
Apr 06 22:25:05 alarm systemd[1]: systemd-networkd.service: Main process exited, code=killed, status=6/ABRT
Apr 06 22:25:05 alarm systemd[1]: systemd-networkd.service: Unit entered failed state.
Apr 06 22:25:05 alarm systemd[1]: systemd-networkd.service: Failed with result 'signal'.
Apr 06 22:25:05 alarm systemd[1]: systemd-networkd.service: Service has no hold-off time, scheduling restart.
Apr 06 22:25:05 alarm systemd[1]: Stopped Network Service.
Apr 06 22:25:05 alarm systemd[1]: Starting Network Service...
Apr 06 22:25:07 alarm systemd-networkd[6140]: eth0: Gained IPv6LL

System numbers for kernel config, and what is actually on the system after the RA flood

# cat /proc/sys/net/ipv6/conf/eth0/max_addresses 
16

# ip addr show dev eth0 | grep inet6 | wc -l
3838

# ip -6 route | wc -l
3838

restarting networkd does NOT flush errant prefixes, or restore connectivity. Must reboot!

@poettering poettering added network bug 🐛 Programming errors, that need preferential fixing labels Apr 7, 2016
@ssahani
Copy link
Contributor

ssahani commented Apr 26, 2016

Super interesting

Test results.

  1. for the first time networkd gets OOM and this happens.
Apr 06 22:25:05 alarm systemd-networkd[6119]: Assertion 'link->state == LINK_STATE_SETTING_ROUTES' failed at src/network/networkd-link.c:586, function
and

If you stop the flood attack and restant both the whole system hangs . Kernel hangs with "soft lock"
with latest fedora kernel.

We really don't do FW stuff but yes

Solutions:

There are many ways to stop this attack.

  1. Disabling router discovery
  2. RA Guard:
  3. Using firewall to block such packets.

We can do RA guard let's talk about it .
https://tools.ietf.org/html/rfc6105

Hm, I agree networks should not crash.

But question is should we do this @poettering @teg @keszybz ?

@cvmiller
Copy link
Author

Thanks for looking into this problem. There are some challenges with your ideas towards a solution.

  1. Disabling router discovery - If by this you mean no longer listening to RAs, then the default route would have to be manually configured on every host. Even hosts using DHCPv6 rely on RAs for default route.
  2. RA Guard: - I don't think this applies to a host with a single interface. RA Guard as implemented by Cisco, does not forward RAs from ports that have not been configured as having Routers attached. That said, Cisco's implementation is easily breached by the use of extension headers (which THC supports).
  3. Using firewall to block such packets. - Not sure what you would filter on to prevent rogue RAs from reaching the host. After all, THC is already creating RAs which randomize not only the prefix but the source MAC address. You could filter on the real router MAC address, but there is nothing to stop an attacker from spoofing that MAC address and filling your route table with bad prefixes.

I would suggest a simpler solution. If you look at a Linux disto before systemd, the stack survived this kind of attack (RA flood). I ran this kind of testing at Ciena in 2012. The stack has some configurable items, including max_addresses which limit the table size.

I think having configurable tables sizes for neighbors, routes and addresses, and small defaults would go along way to helping with the problem. Currently, I see a neighbor table size of around 3300, and route table size of 20,000.

Using a smaller table size, and a FIFO (First In, First Out) approach to populating the tables, means that when the RA flood is over, and a real RA arrives, a bogus entry will be pushed out of the table and the real RA will be installed, thus restoring connectivity.

@ssahani
Copy link
Contributor

ssahani commented Apr 28, 2016

Thanks for coming out with nice idea how we can do this.

  1. Disabling router discovery - If by this you mean no longer listening to RAs, then the default route
    would have to be manually configured on every host. Even hosts using DHCPv6 rely on RAs for
    default route

yes . This means manually by user while under attack (not always).

  1. RA Guard: - I don't think this applies to a host with a single interface.

Any specific reason for this ? We receive the packet and before we process anyting we first
do RA Guard. If it's successful it goes to rest of the processing else we drop the packet.

Cisco's implementation is easily breached by the use of extension headers (which THC supports).

Right but yes when trying to identify an ICMPv6 Router Advertisement message,
we travel the all IPv6 headers to a limit. If this limit is exceeded we drop
the packet. These are FW work really. I have not looked deeper into it.

  1. Using firewall to block such packets. - Not sure what you would filter on to prevent rogue RAs from reaching the host. After all, THC is already creating RAs which randomize not o
    nly the prefix but the source MAC address. You could filter on the real router MAC address, but there is nothing to stop an attacker from spoofing that MAC address and filling your rou
    te table with bad prefixes.
    I guess this is not a option for us. Port Security is one of solution.

The stack has some configurable items, including max_addresses which limit the table size.

yes we can put a limit. I have to think more. We need to fix the crash first

@cvmiller
Copy link
Author

cvmiller commented May 3, 2016

I agree that not being fooled by RAs encapsulated in bogus extension headers will be a nice addition.

I don't think requiring a user to add a default route (with a link-local address) is all that practical. It will trip up users when the router is replaced (since the MAC address will change, and so will the link-local of the router), and it doesn't scale well (think: adding default route to a 100 machines).

This is why I suggested a FIFO approach in which the correct default route would be re-established after the attack has ended (by receiving a real RA). That said, there may be other approaches which don't require manual entering of static default routes which are better than the FIFO approach.

keszybz pushed a commit that referenced this issue May 18, 2016
…lingering

This is partial fix for #2228 and #2977, #3204.

bridge-test: netdev ready
docker0: Gained IPv6LL
wlan0: Gained IPv6LL
eth0: Gained IPv6LL
Enumeration completed
bridge-test: netdev exists, using existing without changing its
parameters
vboxnet0: IPv6 enabled for interface: Success
lo: Configured
docker0: Could not drop address: No such process
vboxnet0: Gained carrier
wlan0: Could not drop address: No such process
eth0: Could not drop address: No such process
eth0: Could not drop address: No such process
eth0: Could not drop address: No such process
vboxnet0: Gained IPv6LL
vboxnet0: Could not set NDisc route or address: Invalid argument
vboxnet0: Failed
[New Thread 0x7ffff6505700 (LWP 1111)]
[Thread 0x7ffff6505700 (LWP 1111) exited]
Assertion 'link->state == LINK_STATE_SETTING_ROUTES' failed at
src/network/networkd-link.c:672, function link_enter_configured().
Aborting.

Program received signal SIGABRT, Aborted.
0x00007ffff6dc6a98 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install
iptables-1.4.21-15.fc23.x86_64 libattr-2.4.47-14.fc23.x86_64
libidn-1.32-1.fc23.x86_64 pcre-8.38-7.fc23.x86_64

Debugging

(gdb) bt
"link->state == LINK_STATE_SETTING_ROUTES", file=0x5555556a34c8
"src/network/networkd-link.c", line=672,
    func=0x5555556a56d0 <__PRETTY_FUNCTION__.14850>
"link_enter_configured") at src/basic/log.c:788
src/network/networkd-link.c:672
src/network/networkd-link.c:720
flags=0 '\000', scope=0 '\000', cinfo=0x7fffffffe020) at
src/network/networkd-address.c:344
(rtnl=0x5555556eded0, message=0x55555570ff20, userdata=0x5555556ec590)
at src/network/networkd-manager.c:604
m=0x55555570ff20) at src/libsystemd/sd-netlink/sd-netlink.c:365
at src/libsystemd/sd-netlink/sd-netlink.c:395
ret=0x0) at src/libsystemd/sd-netlink/sd-netlink.c:429
revents=1, userdata=0x5555556eded0) at
src/libsystemd/sd-netlink/sd-netlink.c:723
src/libsystemd/sd-event/sd-event.c:2268
src/libsystemd/sd-event/sd-event.c:2629
timeout=18446744073709551615) at src/libsystemd/sd-event/sd-event.c:2688
bus=0x5555556eeba0, name=0x55555568a2f5 "org.freedesktop.network1",
timeout=30000000,
    check_idle=0x55555556adb6 <manager_check_idle>,
userdata=0x5555556ec590) at src/shared/bus-util.c:134
src/network/networkd-manager.c:1130
src/network/networkd.c:127

(gdb) f 3
src/network/networkd-link.c:672
672         assert(link->state == LINK_STATE_SETTING_ROUTES);
(gdb) p link->state
$1 = LINK_STATE_FAILED

We should not be in this state .

even if vboxnet0 failed we went into this state.

vboxnet0: Could not set NDisc route or address: Invalid argument
vboxnet0: Failed
@cvmiller
Copy link
Author

cvmiller commented Nov 5, 2016

Retested with systemd-networkd version 231. Passed

systemd 231
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

Now neighbour table size is limited to below 1000, and is consistently cleaned up while under RA flood attack.

[root@alarm systemd]# ip neigh | wc
    765    5354   54313
[root@alarm systemd]# ip neigh | wc
    984    6887   69789
[root@alarm systemd]# ip neigh | wc
    753    5270   53456
[root@alarm systemd]# ip neigh | wc
    397    2778   28180

Closing issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐛 Programming errors, that need preferential fixing network
Development

No branches or pull requests

3 participants