ot-br-posix docker: can't reach docker network from the thread network

[lcoudeville]

I'm having an issue where I can't reach the docker network from the (open)thread network. I can ping the docker network from a bash shell in the docker container. But when I try to ping the address using ot-ctl it doesn't work. Investigation with tcpdump learned that in both cases the ping message are sent and respondend with a different ip address (which is ofcourse definitely normal).

On the native machine (docker -> bash)

$ ping -c 1 fd11:db8:1::1
PING fd11:db8:1::1(fd11:db8:1::1) 56 data bytes
64 bytes from fd11:db8:1::1: icmp_seq=1 ttl=64 time=0.025 ms
--- fd11:db8:1::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms

in parallel, I ran:

$ tcpdump -ni eth0 icmp6 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:46:45.792485 IP6 fd11:db8:1::2 > fd11:db8:1::1: ICMP6, echo request, seq 1, length 64
18:46:45.792504 IP6 fd11:db8:1::1 > fd11:db8:1::2: ICMP6, echo reply, seq 1, length 64

from the ot-ctl shell

$ ot-ctl ping fd11:db8:1::1
1 packets transmitted, 0 packets received. Packet loss = 100.0%.
Done

tcpdump on the eth0, also in parallel:

$ tcpdump -ni eth0 icmp6 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:48:13.587013 IP6 fdf2:2776:958:1:3fe9:18f8:1c19:9e80 > fd11:db8:1::1: ICMP6, echo request, seq 3, length 16
18:48:13.587045 IP6 fd11:db8:1::1 > fdf2:2776:958:1:3fe9:18f8:1c19:9e80: ICMP6, echo reply, seq 3, length 16

tcpdump on the wpan0 interface:

$ tcpdump -ni wpan0 icmp6 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wpan0, link-type RAW (Raw IP), capture size 262144 bytes
18:51:52.289403 IP6 fdf2:2776:958:1:3fe9:18f8:1c19:9e80 > fd11:db8:1::1: ICMP6, echo request, seq 6, length 16

For me, it looks a bit strange that the icmp6 message is also emitted on the wpan0 interface, however I'm not skilled enough in ot-br-posix, so this might be normal. But the ping request is routed to the eth0 interface, also an answer is received but doesn't seem to reach its destination 'the ot-ctl shell'.

Here are the steps to reproduce the issue.

docker network create --ipv6 --subnet fd11:db8:1::/64 -o com.docker.network.bridge.name=otbr0 otbr

sudo modprobe ip6table_filter
docker run -it --rm --privileged --name otbr --network otbr -p 8080:80 --sysctl "net.ipv6.conf.all.disable_ipv6=0 net.ipv4.conf.all.forwarding=1 net.ipv6.conf.all.forwarding=1" --volume /dev/ttyACM0:/dev/radio nrfconnect/otbr:9185bda --radio-url spinel+hdlc+uart:///dev/radio?uart-baudrate=1000000

(also tried with openthread/otbr:latest).

Enable/start the thread border router:

docker exec -it otbr bash -c "ot-ctl dataset init new && ot-ctl dataset commit active && ot-ctl ifconfig up && ot-ctl thread start"

The routing table in my docker environment

root@a31a21b3758c:/app# ip -6 r
fd11:db8:1::/64 dev eth0 proto kernel metric 256 pref medium
fd66:cfb6:d113:3fc0::/64 dev eth0 proto kernel metric 256 expires 1651sec pref medium
fda6:7394:fdef:8044::/64 dev wpan0 proto kernel metric 256 pref medium
fdf2:2776:958:1::/64 dev wpan0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev wpan0 proto kernel metric 256 pref medium
default via fd11:db8:1::1 dev eth0 metric 1024 pref medium

On my native machine, I've added a default route ipv6 to the docker network

default via fd11:db8:1::2 dev otbr0 metric 1024 pref medium

Additionally, I added another thread router (not a border router). When I ping from my native machine to that thread router I can see that my ping message arrives but they are not responded (probably because it lacks a route to my docker network?). I played around with routes and prefixes but without luck. But I guess that resolving the first issue might also resolve this issue.

Hope anybody can help, thank you in advance.

[lcoudeville]

I was able to resolve this by manually adding a route to my docker network in the open thread router:

docker exec -it otbr ot-ctl route add fd11:db8:1:0::/64 s high

followed by

docker exec -it otbr ot-ctl netdata register

I also noticed that a route was added for my network:

$ ip -6 r
fd11:db8:1::/64 dev eth0 proto kernel metric 256 pref medium
fd11:db8:1::/64 dev wpan0 metric 65535 pref medium
...

Now I can ping other thread devices, from my native machine that hosts the border router, in my thread network. I still can't ping fd11:db8:1::1 from my border router with ot-ctl. However, that's something that I can understand... the preferred system route for that network is eth0 not wpan0. But from another router in the thread network I can now ping fd11:db8:1::1. So mission accomplished...

However, I still don't understand why I had to add this route manually. Anyone that can explain why this is necessary or what I'm (still) doing wrong?

[jwhui]

fd11:db8:1::/64 prefix appears to be a manually-configured prefix. You should be using the IPv6 prefix that is automatically managed by the border router instead, such that routes are automatically configured and advertised. For more information, see the Thread Border Router - Bidirectional IPv6 Connectivity and DNS-Based Service Discovery codelab.