New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

respondd: duplicate ipv6 address in nodeinfo #1615

Open
mweinelt opened this Issue Jan 8, 2019 · 5 comments

Comments

Projects
None yet
5 participants
@mweinelt
Copy link
Contributor

mweinelt commented Jan 8, 2019

For a while routers seem to be reporting some of their addresses twice in their nodeinfo response. The issue stems from the nodeinfo response on the router, as can be seen by calling gluon-neighbour-info locally on an affected router.

An excerpt given by Jason from Freifunk Frankfurt:

root@avm4020:~# gluon-neighbour-info -d ::1 -p 1001 -r nodeinfo -c 1
{
  "software": {
    "autoupdater": {
      "branch": "test",
      "enabled": true
    },
    "batman-adv": {
      "version": "2013.4.0",
      "compat": 14
    },
    "fastd": {
      "version": "v18",
      "enabled": false
    },
    "firmware": {
      "base": "gluon-2018.2",
      "release": "v2.5.1-test-0102"
    }
  },
  "network": {
    "addresses": [
      "2a06:8187:fbba:1234:ca0e:14ff:feb5:e1ad",
      "2a06:8187:fbba:1234:ca0e:14ff:feb5:e1ad",
      "fddd:5d16:b5dd::ca0e:14ff:feb5:e1ad",
      "fe80::ca0e:14ff:feb5:e1ad"
    ],
    "mesh": {
      "bat0": {
        "interfaces": {
          "wireless": [
            "be:2e:7d:2c:32:42"
          ],
          "other": [
            "be:2e:7d:2c:32:44",
            "be:2e:7d:2c:32:43",
            "be:2e:7d:2c:32:40"
          ]
        }
      }
    },
    "mac": "c8:0e:14:b5:e1:ad"
  },
  "location": {
    "latitude": 50.12771951,
    "longitude": 8.59648837
  },
  "owner": {
    "contact": "...@gmx.net"
  },
  "system": {
    "site_code": "ffffm"
  },
  "node_id": "c80e14b5e1ad",
  "hostname": "avm4020",
  "hardware": {
    "model": "AVM FRITZ!Box 4020",
    "nproc": 1
  }
}
root@avm4020:~# cat /proc/net/if_inet6
2a068187fbba1234ca0e14fffeb5e1ad 09 40 00 00 br-client
fddd5d16b5dd0000ca0e14fffeb5e1ad 09 40 00 00 br-client
fe80000000000000bc2e7dfffe2c3243 0a 40 20 80 primary0
fe80000000000000bc2e7dfffe2c3240 11 40 20 80  client0
fe80000000000000bc2e7dfffe2c3240 06 40 20 80   br-wan
fe80000000000000ca0e14fffeb5e1ad 0b 40 20 80     bat0
fe80000000000000ca0e14fffeb5e1ad 09 40 20 80 br-client
fe80000000000000144195fffe40f7dc 08 40 20 80 local-node
fe80000000000000bc2e7dfffe2c3242 10 40 20 80    ibss0
00000000000000000000000000000001 01 80 10 80       lo
fddd5d16b5dd00000000000000000001 08 80 00 a0 local-node
fe80000000000000bc2e7dfffe2c3244 04 40 20 80     eth1

The issue could have possibly been introduced in 7408f04 and is not always present. I've only seen it in roughly ~15% of our routers with a recent master build.

@rotanid rotanid added the bug label Jan 9, 2019

@NeoRaider

This comment has been minimized.

Copy link
Member

NeoRaider commented Jan 9, 2019

I don't see how the code could possibly malfunction in a way that would turn 3 matching entries in if_inet6 into 4.

Is the behaviour stable - do the same nodes always exhibit the issue, or does it disappear with a reboot, or at random?

I guess we should switch from /proc/net/if_inet6 to the Netlink-based getifaddrs() in any case - I think the current code is still a remnant of the previous Lua-based respondd implementation.

@blocktrron

This comment has been minimized.

Copy link
Contributor

blocktrron commented Jan 9, 2019

Just some input - another node based on Gluon master:

root@64xxx-5c49793001b4:~# gluon-neighbour-info -d ::1 -p 1001 -r nodeinfo -c 1
{"software":{"autoupdater":{"branch":"stable","enabled":false},"batman-adv":{"version":"openwrt-2018.1-5","compat":15},"fastd":{"version":"v18","enabled":true,"public_key":"692f60178199654162151ddda08d3731778fe85de7698521096e8c5ca1608a92"},"firmware":{"base":"gluon-v2018.1-180-gc739e481","release":"1.3~20190109"}},"network":{"addresses":["2001:67c:2ed8:100e:5e49:79ff:fe30:1b4","2001:67c:2ed8:100e:5e49:79ff:fe30:1b4","fe80::5e49:79ff:fe30:1b4","fd01:67c:2ed8:100e:5e49:79ff:fe30:1b4"],"mesh":{"bat0":{"interfaces":{"wireless":["4a:f8:07:87:6e:c9"],"tunnel":["4a:f8:07:87:6e:cf"],"other":["4a:f8:07:87:6e:cb"]}}},"mac":"5c:49:79:30:01:b4"},"system":{"site_code":"ffda","domain_code":"ffda_64367"},"node_id":"5c49793001b4","hostname":"64xxx-5c49793001b4","hardware":{"model":"AVM FRITZ!Box 4020","nproc":1}}
root@64xxx-5c49793001b4:~# cat /proc/net/if_inet6
2001067c2ed8100e5e4979fffe3001b4 06 40 00 00 br-client
fe8000000000000048f807fffe876ecf 0f 40 20 80 mesh-vpn
fe800000000000005e4979fffe3001b4 0c 40 20 80     bat0
fe800000000000005e4979fffe3001b4 06 40 20 80 br-client
00000000000000000000000000000001 01 80 10 80       lo
fe8000000000000048f807fffe876ec9 0d 40 20 80    mesh0
fe80000000000000d8ff14fffe00ffff 0a 40 20 80 local-node
fd01067c2ed8100e5e4979fffe3001b4 06 40 00 00 br-client
fe8000000000000048f807fffe876ecb 0b 40 20 80 primary0
fe8000000000000048f807fffe876ec8 0e 40 20 80  client0
fd01067c2ed8100e0000000000010001 0a 80 00 a0 local-node
root@64xxx-5c49793001b4:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether ea:ef:ee:a6:d9:e6 brd ff:ff:ff:ff:ff:ff
3: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-wan state DOWN qlen 1000
    link/ether 5c:49:79:30:01:b3 brd ff:ff:ff:ff:ff:ff
4: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN qlen 1000
    link/ether 5c:49:79:30:01:b2 brd ff:ff:ff:ff:ff:ff
6: br-client: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 5c:49:79:30:01:b4 brd ff:ff:ff:ff:ff:ff
    inet6 2001:67c:2ed8:100e:5e49:79ff:fe30:1b4/64 scope global dynamic 
       valid_lft 86399sec preferred_lft 14399sec
    inet6 fd01:67c:2ed8:100e:5e49:79ff:fe30:1b4/64 scope global dynamic 
       valid_lft 86399sec preferred_lft 14399sec
    inet6 fe80::5e49:79ff:fe30:1b4/64 scope link 
       valid_lft forever preferred_lft forever
7: eth1.1@eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-client state LOWERLAYERDOWN qlen 1000
    link/ether 5c:49:79:30:01:b2 brd ff:ff:ff:ff:ff:ff
8: br-wan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 4a:f8:07:87:6e:c8 brd ff:ff:ff:ff:ff:ff
9: local-port@local-node: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-client state UP qlen 1000
    link/ether 5c:49:79:30:01:b4 brd ff:ff:ff:ff:ff:ff
10: local-node@local-port: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether da:ff:14:00:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.84.239.254/20 brd 10.84.239.255 scope global local-node
       valid_lft forever preferred_lft forever
    inet6 fd01:67c:2ed8:100e::1:1/128 scope global deprecated 
       valid_lft forever preferred_lft 0sec
    inet6 fe80::d8ff:14ff:fe00:ffff/64 scope link 
       valid_lft forever preferred_lft forever
11: primary0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1532 qdisc noqueue master bat0 state UNKNOWN qlen 1000
    link/ether 4a:f8:07:87:6e:cb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::48f8:7ff:fe87:6ecb/64 scope link 
       valid_lft forever preferred_lft forever
12: bat0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-client state UNKNOWN qlen 1000
    link/ether 5c:49:79:30:01:b4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5e49:79ff:fe30:1b4/64 scope link 
       valid_lft forever preferred_lft forever
13: mesh0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1532 qdisc noqueue master bat0 state UP qlen 1000
    link/ether 4a:f8:07:87:6e:c9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::48f8:7ff:fe87:6ec9/64 scope link 
       valid_lft forever preferred_lft forever
14: client0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-client state UP qlen 1000
    link/ether 4a:f8:07:87:6e:c8 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::48f8:7ff:fe87:6ec8/64 scope link 
       valid_lft forever preferred_lft forever
15: mesh-vpn: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1312 qdisc fq_codel master bat0 state UNKNOWN qlen 1000
    link/ether 4a:f8:07:87:6e:cf brd ff:ff:ff:ff:ff:ff
    inet6 fe80::48f8:7ff:fe87:6ecf/64 scope link 
       valid_lft forever preferred_lft forever
root@64xxx-5c49793001b4:~# 

blocktrron added a commit to blocktrron/gluon that referenced this issue Jan 10, 2019

gluon-mesh-batman-adv: use getifaddrs
Previously the content of "/proc/net/if_inet6" was parsed to acquire the
nodes IPv6 addresses. With this commit, IPv6 addresses are acquired
using getifaddrs() as proposed by neoraider in freifunk-gluon#1615.

blocktrron added a commit to blocktrron/gluon that referenced this issue Jan 10, 2019

gluon-mesh-batman-adv: use getifaddrs
Previously the content of "/proc/net/if_inet6" was parsed to acquire the
nodes IPv6 addresses. With this commit, IPv6 addresses are acquired
using getifaddrs() as proposed by neoraider in freifunk-gluon#1615.
@kaechele

This comment has been minimized.

Copy link
Contributor

kaechele commented Jan 12, 2019

I can confirm that on affected nodes we do see the same ULA address on br-client twice in ifconfig:

br-client Link encap:Ethernet  HWaddr 14:CC:20:C8:14:90  
          inet6 addr: fddf:ebfd:a801:214:16cc:20ff:fec8:1490/64 Scope:Global
          inet6 addr: fddf:ebfd:a801:214:16cc:20ff:fec8:1490/64 Scope:Global
          inet6 addr: fe80::16cc:20ff:fec8:1490/64 Scope:Link

but not in ip:

6: br-client: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 14:cc:20:c8:14:90 brd ff:ff:ff:ff:ff:ff
    inet6 fddf:ebfd:a801:214:16cc:20ff:fec8:1490/64 scope global dynamic 
       valid_lft 86382sec preferred_lft 14382sec
    inet6 fe80::16cc:20ff:fec8:1490/64 scope link 
       valid_lft forever preferred_lft forever
@rotanid

This comment has been minimized.

Copy link
Member

rotanid commented Jan 13, 2019

interesting finding, maybe there's an upstream bug, too...

blocktrron added a commit to blocktrron/gluon that referenced this issue Jan 13, 2019

gluon-mesh-batman-adv: get ip via netlink
Previously the content of "/proc/net/if_inet6" was parsed to acquire the
nodes IPv6 addresses. With this commit, IPv6 addresses are acquired
using netlink sockets as proposed by neoraider in freifunk-gluon#1615.

blocktrron added a commit to blocktrron/gluon that referenced this issue Jan 13, 2019

gluon-mesh-batman-adv: get ip via netlink
Previously the content of "/proc/net/if_inet6" was parsed to acquire the
nodes IPv6 addresses. With this commit, IPv6 addresses are acquired
using netlink sockets as proposed by neoraider in freifunk-gluon#1615.

blocktrron added a commit to blocktrron/gluon that referenced this issue Jan 13, 2019

gluon-mesh-batman-adv: get ip via netlink
Previously the content of "/proc/net/if_inet6" was parsed to acquire the
nodes IPv6 addresses. With this commit, IPv6 addresses are acquired
using netlink sockets as proposed by neoraider in freifunk-gluon#1615.

blocktrron added a commit to blocktrron/gluon that referenced this issue Jan 13, 2019

gluon-mesh-batman-adv: get ip via netlink
Previously the content of "/proc/net/if_inet6" was parsed to acquire the
nodes IPv6 addresses. With this commit, IPv6 addresses are acquired
using netlink sockets as proposed by neoraider in freifunk-gluon#1615.

blocktrron added a commit to blocktrron/gluon that referenced this issue Jan 13, 2019

gluon-mesh-batman-adv: get IPv6-addresses via netlink
Previously the content of "/proc/net/if_inet6" was parsed to acquire the
nodes IPv6 addresses. With this commit, IPv6 addresses are acquired
using netlink as proposed by neoraider in freifunk-gluon#1615.
@mweinelt

This comment has been minimized.

Copy link
Contributor

mweinelt commented Jan 15, 2019

The issue seems to be device (not model) specific. I have a UAP-AC-M that even after multiple reboots always comes back with the problem.

root@64283-cccda-w:~# cat /proc/net/if_inet6 
fd01067c2ed81001f29fc2fffedec4c5 09 40 00 00 br-client
fe80000000000000f29fc2fffedec4c5 0b 40 20 80     bat0
fe80000000000000f29fc2fffedec4c5 09 40 20 80 br-client
fe80000000000000b8f560fffedd3973 0a 40 20 80 primary0
00000000000000000000000000000001 01 80 10 80       lo
fd01067c2ed810010000000000010001 08 80 00 a0 local-node
fe80000000000000b8f560fffedd3970 0e 40 20 80  client0
fe80000000000000b8f560fffedd3970 0f 40 20 80 vx_mesh_wan
fe80000000000000b8f560fffedd3970 06 40 20 80   br-wan
fe80000000000000d8ff01fffe00ffff 08 40 20 80 local-node
fe80000000000000b8f560fffedd3975 0c 40 20 80    mesh1
fe80000000000000f29fc2fffeddc4c5 0d 40 20 80  client1
2001067c2ed81001f29fc2fffedec4c5 09 40 00 00 br-client

the busybox ifconfig applet, using /proc/net/if_inet6 as well, also exhibits this issue:

root@64283-cccda-w:~# ifconfig -a br-client
br-client Link encap:Ethernet  HWaddr F0:9F:C2:DE:C4:C5  
          inet6 addr: fd01:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
          inet6 addr: fd01:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
          inet6 addr: fe80::f29f:c2ff:fede:c4c5/64 Scope:Link
          inet6 addr: 2001:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:26231 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5786 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2741459 (2.6 MiB)  TX bytes:815270 (796.1 KiB)
root@64283-cccda-w:~# strace ifconfig -a br-client
execve("/sbin/ifconfig", ["ifconfig", "-a", "br-client"], 0x7f806e98 /* 12 vars */) = 0
set_thread_area(0x77917dc0)             = 0
set_tid_address(0x77910d28)             = 10905
open("/etc/ld-musl-mips-sf.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=78096, ...}) = 0
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\10\0\0\0\1\0\0(p\0\0\0004"..., 936) = 936
mmap2(NULL, 147456, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x77848000
mmap2(0x7786b000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x13000) = 0x7786b000
close(3)                                = 0
mprotect(0x45b000, 4096, PROT_READ)     = 0
prctl(PR_SET_NAME, "ifconfig")          = 0
getuid()                                = 0
open("/proc/net/dev", O_RDONLY|O_LARGEFILE) = 3
readv(3, [{iov_base="", iov_len=0}, {iov_base="Inter-|   Receive               "..., iov_len=1024}], 2) = 1024
readv(3, [{iov_base="", iov_len=0}, {iov_base="945492    6896    0    0    0   "..., iov_len=1024}], 2) = 805
_llseek(3, -750, [1079], SEEK_CUR)      = 0
close(3)                                = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
ioctl(3, SIOCGIFFLAGS, {ifr_name="br-client", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
ioctl(3, SIOCGIFHWADDR, {ifr_name="br-client", ifr_hwaddr={sa_family=ARPHRD_ETHER, sa_data=f0:9f:c2:de:c4:c5}}) = 0
ioctl(3, SIOCGIFMETRIC, {ifr_name="br-client", ifr_metric=0}) = 0
ioctl(3, SIOCGIFMTU, {ifr_name="br-client", ifr_mtu=1500}) = 0
ioctl(3, SIOCGIFMAP, {ifr_name="br-client", ifr_map={mem_start=0, mem_end=0, base_addr=0, irq=0, dma=0, port=0}}) = 0
ioctl(3, SIOCGIFTXQLEN, {ifr_name="br-client", ifr_qlen=1000}) = 0
ioctl(3, SIOCGIFADDR, {ifr_name="br-client"}) = -1 EADDRNOTAVAIL (Address not available)
close(3)                                = 0
ioctl(1, TIOCGWINSZ, 0x7fad84a0)        = 0
writev(1, [{iov_base="br-client Link encap:Ethernet  H"..., iov_len=57}, {iov_base="\n", iov_len=1}], 2br-client Link encap:Ethernet  HWaddr F0:9F:C2:DE:C4:C5  
) = 58
open("/proc/net/if_inet6", O_RDONLY|O_LARGEFILE) = 3
readv(3, [{iov_base="", iov_len=0}, {iov_base="fd01067c2ed81001f29fc2fffedec4c5"..., iov_len=1024}], 2) = 767
writev(1, [{iov_base="          inet6 addr: fd01:67c:2"..., iov_len=76}, {iov_base="\n", iov_len=1}], 2          inet6 addr: fd01:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
) = 77
writev(1, [{iov_base="          inet6 addr: fd01:67c:2"..., iov_len=76}, {iov_base="\n", iov_len=1}], 2          inet6 addr: fd01:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
) = 77
writev(1, [{iov_base="          inet6 addr: fe80::f29f"..., iov_len=61}, {iov_base="\n", iov_len=1}], 2          inet6 addr: fe80::f29f:c2ff:fede:c4c5/64 Scope:Link
) = 62
readv(3, [{iov_base="", iov_len=0}, {iov_base="", iov_len=1024}], 2) = 0
writev(1, [{iov_base="          inet6 addr: 2001:67c:2"..., iov_len=76}, {iov_base="\n", iov_len=1}], 2          inet6 addr: 2001:67c:2ed8:1001:f29f:c2ff:fede:c4c5/64 Scope:Global
) = 77
close(3)                                = 0
writev(1, [{iov_base="          UP BROADCAST RUNNING M"..., iov_len=60}, {iov_base="\n", iov_len=1}], 2          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
) = 61
writev(1, [{iov_base="          RX packets:28120 error"..., iov_len=64}, {iov_base="\n", iov_len=1}], 2          RX packets:28120 errors:0 dropped:0 overruns:0 frame:0
) = 65
writev(1, [{iov_base="          TX packets:6896 errors"..., iov_len=65}, {iov_base="\n", iov_len=1}], 2          TX packets:6896 errors:0 dropped:0 overruns:0 carrier:0
) = 66
writev(1, [{iov_base="          collisions:0 txqueuele"..., iov_len=39}, {iov_base="\n", iov_len=1}], 2          collisions:0 txqueuelen:1000 
) = 40
writev(1, [{iov_base="          RX bytes:3887581 (3.7 "..., iov_len=65}, {iov_base="\n", iov_len=1}], 2          RX bytes:3887581 (3.7 MiB)  TX bytes:945492 (923.3 KiB)
) = 66
writev(1, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2
) = 1
exit_group(0)                           = ?
+++ exited with 0 +++

We did a single device testrun with blocktrron@71eb8ba yesterday and it looked promising. Tonight we will roll this out to another 30 nodes and see if the issue is gone entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment