End-to-End MPLS Not Working On Docker Container

[anspectrum]

Hello,
I've following setup:
Host machine is Ubuntu 16.04 LTS (Linux OS 4.15.0-58-generic #64~16.04.1-Ubuntu SMP Wed Aug 7 14:10:35 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux). Installed docker on it ( Version: 18.09.7). Also have GNS3 simulator (2.1.21).

Pulled FRR image from Docker (docker pull frrouting/frr)
MPLS modules (mpls_iptunnel mpls_router) are enabled on host machine. Also enabled following in /etc/sysctl.conf on host machine:

# Enable MPLS Label processing on all interfaces
net.mpls.platform_labels=100000
net.mpls.conf.lo.input=1
net.mpls.conf.enp0s25.input=1
net.mpls.conf.docker0.input = 1

A simple MPLS 3 x router topology

Everything works fine like OSPF, ISIS, BGP. However, when I enable MPLS LDP on interconnecting interfaces, frr-1 can not ping frr-3 loopback and vide versa. I've done packer captures and it seems that frr-1 and frr-3 both send the correct MPLS labelled packets but frr-2 keeps dropping them. Don't know why its like that. Here is the configs:

hostname frr-1
!
!
interface eth0
 ip address 10.1.1.1/30
 ip ospf network point-to-point
!
interface lo
 ip address 1.1.1.1/32
!
!
 !
router ospf
 network 0.0.0.0/0 area 0
!
mpls ldp
 router-id 1.1.1.1
 !       
 address-family ipv4
  discovery transport-address 1.1.1.1
  !
  interface eth0
  !
  interface lo
  !
 exit-address-family
 !
!
line vty
 exec-timeout 0 0
!
end


hostname frr-2
!
interface eth0
 ip address 10.1.1.2/30
 ip ospf network point-to-point
!
interface eth1
 ip address 10.1.1.5/30
 ip ospf network point-to-point
!
interface lo
 ip address 1.1.1.2/32
!
router ospf
 network 0.0.0.0/0 area 0
!
mpls ldp
 router-id 1.1.1.2
 !       
 address-family ipv4
  discovery transport-address 1.1.1.2
  !
  interface eth1
  !
  interface eth0
  !
  interface lo
  !
 exit-address-family
 !
line vty
!
end


hostname frr-3
!
!
interface eth1
 ip address 10.1.1.6/30
 ip ospf network point-to-point
!
interface lo
 ip address 1.1.1.3/32
!
 !
!
router ospf
 network 0.0.0.0/0 area 0
!
mpls ldp
 router-id 1.1.1.3
 !       
 address-family ipv4
  discovery transport-address 1.1.1.3
  !
  interface eth1
  !
  interface lo
  !
 exit-address-family
 !
!
line vty
 exec-timeout 0 0
!
end

Please suggest where can I look to fix it.

[anspectrum]

I've also checked this issue on latest released Alpine Linux and the behavior remains the same i.e., labelled packets can't get through.

[mkbt]

I vaguely remember having a similar issue with MPLS inside containers. And I think Donald directed me to the topotests in https://github.com/FRRouting/frr/blob/master/tests/topotests This has some of the flags https://github.com/FRRouting/frr/blob/master/tests/topotests/bgp_l3vpn_to_bgp_direct/customize.py https://github.com/i-maravic/MPLS-Linux and Linux VRF txt was the other thing I followed. (bearing in mind your post is about IPoMPLS via not VPNv4 over LDP)

…

On Wed, 28 Aug 2019 at 12:27, anspectrum ***@***.***> wrote: I've also checked this issue on latest released Alpine Linux and the behavior remains the same i.e., labelled packets can't get through. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

-- Regards, Mark Tees

[mkbt]

Just looking closer at what you said, are you sure are not more veth's that need MPLS enabled on in /proc/sys/net/mpls/conf/ ?

…

On Wed, 28 Aug 2019 at 12:46, Mark Tees ***@***.***> wrote: I vaguely remember having a similar issue with MPLS inside containers. And I think Donald directed me to the topotests in https://github.com/FRRouting/frr/blob/master/tests/topotests This has some of the flags https://github.com/FRRouting/frr/blob/master/tests/topotests/bgp_l3vpn_to_bgp_direct/customize.py https://github.com/i-maravic/MPLS-Linux and Linux VRF txt was the other thing I followed. (bearing in mind your post is about IPoMPLS via not VPNv4 over LDP) On Wed, 28 Aug 2019 at 12:27, anspectrum ***@***.***> wrote: > > I've also checked this issue on latest released Alpine Linux and the behavior remains the same i.e., labelled packets can't get through. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub, or mute the thread. -- Regards, Mark Tees

-- Regards, Mark Tees

[anspectrum]

@mkbt Thanks for reading and replying.
During my testing I've also created containers with "--privileged" flag and once I spawn them I can see that new vethxxxxxxxx interfaces are created on host machine. Then I enabled mpls processing in container (/proc/sys/net/mpls/conf/vethxxxxx/input) as we all on the host machine but still its not working. Although when I capture traffic on those vethxxxxx interfaces on host machine using tcpdump, I can see MPLS labeled packets.
Really stuck in here :((

((NOTE: Removed BGP config from OP as it was not required))

[anspectrum]

After much hit and trial and searching information, the FRR is working perfectly now. Here is what I did (for others to be benefited)

Need to enable MPLS processing in Docker container Kernel:

# cat /etc/sysctl.conf 
# content of this file will override /etc/sysctl.d/*
net.mpls.conf.lo.input = 1
net.mpls.conf.eth0.input = 1
net.mpls.conf.eth1.input = 1
net.mpls.conf.eth2.input = 1
net.mpls.platform_labels = 100000

After enabling it execute sysctl -p

VRF Interface Addition on FRR CLI

ip link add ABC type vrf table 10
ip link set dev ABC up
ip link set eth0 master ABC

FIB Entries in Linux for proper working of VRF
((Got this from Andrea Dainese from http://www.routereflector.com/2016/11/working-with-vrf-on-linux/))

ip rule add iif ABC table 10
ip rule add oif ABC table 10

To confirm rules have been added

ip rule show
0:	from all lookup local
32764:	from all oif ABC lookup 10
32765:	from all iif ABC lookup 10
32766:	from all lookup main
32767:	from all lookup default

Sample working config of frr-1

hostname frrr-1
!
enable password zebra
password zebra
!
interface eth1 vrf ABC
 ip address 20.1.1.1/30
!
interface eth0
 ip address 10.1.1.1/30
!
interface lo
 ip address 1.1.1.1/32
!
router bgp 1
 bgp router-id 1.1.1.1
 neighbor 1.1.1.3 remote-as 1
 neighbor 1.1.1.3 update-source lo
 !
 address-family ipv4 unicast
  neighbor 1.1.1.3 next-hop-self
 exit-address-family
 !
 address-family ipv4 vpn
  neighbor 1.1.1.3 activate
  neighbor 1.1.1.3 next-hop-self
 exit-address-family
!
router bgp 1 vrf ABC
 !
 address-family ipv4 unicast
  redistribute connected
  label vpn export auto
  rd vpn export 1:1
  rt vpn both 1:1
  export vpn
  import vpn
 exit-address-family
!
router ospf
 network 0.0.0.0/0 area 0
!
mpls ldp
 router-id 1.1.1.1
 !
 address-family ipv4
  discovery transport-address 1.1.1.1
  !
  interface eth0
  !
  interface lo
  !
 exit-address-family
 !
!
line vty
!
end

[dipankarshaw]

i ran into the same problem and MPLS was not showing enabled. I could by pass the problem
this is how i did it.
i was using a GNS3VM on my PC to host Frr routing as docker containers.
now, when i was checking the command :
frr_d-5# show mpls status
MPLS support enabled: no (mpls kernel extensions not detected)

this shows me MPLS support not enabled.
in the command sysctl -a, i could see there was no mpls variables.
so i logged in the GNS3vm
gns3@gns3vm:~$ uname -r
5.3.0-51-generic
<<<<<<<< and i installed the additional modules).

root@gns3vm:# apt-get install linux-modules-extra-5.3.0-51-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
linux-modules-extra-5.3.0-51-generic
0 upgraded, 1 newly installed, 0 to remove and 18 not upgraded.
Need to get 37.6 MB of archives.
After this operation, 181 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-security/main amd64 linux-modules-extra-5.3.0-51-generic amd64 5.3.0-51.4418.04.2 [37.6 MB]
Fetched 37.6 MB in 5s (6,890 kB/s)
Selecting previously unselected package linux-modules-extra-5.3.0-51-generic.
(Reading database ... 32545 files and directories currently installed.)
Preparing to unpack .../linux-modules-extra-5.3.0-51-generic_5.3.0-51.4418.04.2_amd64.deb ...
Unpacking linux-modules-extra-5.3.0-51-generic (5.3.0-51.4418.04.2) ...
Setting up linux-modules-extra-5.3.0-51-generic (5.3.0-51.4418.04.2) ...
Processing triggers for linux-image-5.3.0-51-generic (5.3.0-51.4418.04.2) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.3.0-51-generic
/etc/kernel/postinst.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.3.0-51-generic
Found initrd image: /boot/initrd.img-5.3.0-51-generic
done
root@gns3vm:#
root@gns3vm:#
root@gns3vm:#
root@gns3vm:#
root@gns3vm:~# sudo modprobe mpls-router mpls-iptunnel

now..i can see some entries in the gns3vm
net.mpls.conf.docker0.input = 0
net.mpls.conf.eth0.input = 0
net.mpls.conf.eth1.input = 0
net.mpls.conf.lo.input = 0
net.mpls.conf.virbr0.input = 0
net.mpls.conf.virbr0-nic.input = 0
net.mpls.default_ttl = 255
net.mpls.ip_ttl_propagate = 1
net.mpls.platform_labels = 0

so i went to the docker frr rouitng and set the core interfaces to be mpls enabled
frr_d-4# exit
/ # cat /etc/sysctl.conf
net.mpls.conf.lo.input=1
net.mpls.conf.eth0.input=1
net.mpls.conf.eth1.input=1
net.mpls.conf.eth2.input=1
net.mpls.platform_labels=100000
/ # sysctl -p
net.mpls.conf.lo.input = 1
net.mpls.conf.eth0.input = 1
net.mpls.conf.eth1.input = 1
net.mpls.conf.eth2.input = 1
net.mpls.platform_labels = 100000
/ # sysctl -a | grep mpls
sysctl: error reading key 'net.ipv6.conf.all.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.default.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.eth0.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.eth1.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.eth2.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.eth3.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.eth4.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.eth5.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.eth6.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.eth7.stable_secret': I/O error
sysctl: error reading key 'net.ipv6.conf.lo.stable_secret': I/O error
net.mpls.conf.eth0.input = 1
net.mpls.conf.eth1.input = 1
net.mpls.conf.eth2.input = 1
net.mpls.conf.eth3.input = 0
net.mpls.conf.eth4.input = 0
net.mpls.conf.eth5.input = 0
net.mpls.conf.eth6.input = 0
net.mpls.conf.eth7.input = 0
net.mpls.conf.lo.input = 1
net.mpls.default_ttl = 255
net.mpls.ip_ttl_propagate = 1
net.mpls.platform_labels = 100000
/ #

now, i rebooted the docker container and i can see sr mpls tables.
frr_d-4# show mpls status
MPLS support enabled: yes
frr_d-4# show mpls table
Inbound Label Type Nexthop Outbound Label

20001 SR (OSPF) 14.0.0.1 20001
20002 SR (OSPF) 24.0.0.1 20002
20003 SR (OSPF) 24.0.0.1 20003
20003 SR (OSPF) 14.0.0.1 20003
20004 SR (OSPF) lo -
20005 SR (OSPF) 24.0.0.1 20005
20005 SR (OSPF) 14.0.0.1 20005
50000 SR (OSPF) 14.0.0.1 implicit-null
50001 SR (OSPF) 14.0.0.1 implicit-null
50002 SR (OSPF) 24.0.0.1 implicit-null
50003 SR (OSPF) 24.0.0.1 implicit-null

frr_d-4#

[anp135]

Docker does not want you have corect container's lo (loopback) interface working.

In case, when all frr from upper picture based on personal virtual machines, their loopbacks is unreacheable. Containers use ipvlan l2 mode for direct connection to the network adapter. Instead macvlan mode because basically promisc didn't allowed by hypervisor. So containers have the same MAC with Host... When destination IP in incoming packets is own by any container - no problem, packets come to container and everything works properly. But when we setup lo or dummy or other network adapter (isolated/not isolated bridge/something else) - nothing works, packets, wihicn come for the same MAC didn't reach container - docker inspect and not found reason for put this packets to container... stupid DPI -(((.. And nobody can't change it.
When we attach the same /32 ip address to the ipvlan interface of container, so there docker switching come works...

Host ip config does not matter. If parent ipvlan interface have ip - then Host will be ICMP reply "router changed to ..container_IP, while no ip address attached - just silently drop.. of couse..

Of cource fogot explicit-null also, because MPLS packets fullmissunderstood by docker the same -)).

[Pcrossley69]

I have the same problem - frr 8.1 docker containers under GNS3 2.2.29
Kernel modules are installed
sysctl has been modified

I am using explicit null labels.

"show mpls status" shows mpls is enabled
"show mpls table" and "show mpls ldp bindings" shows that the labels are being allocated

doing a ping test: wireshark shows label swapping occurring at intermediate hops but there are no replies to pings from the final destination.

I've tried final destination as "lo" and as eth0
without mpls everything works fine

[Pcrossley69]

I couldn't quite understand the comment from anp135.
However it did make me wonder if the problem was my use of explicit nulls.

I tried again using implicit null labels and it now works.
I've tested it with a final destination of "lo" and "eth0" both work fine!

It appears that docker containers can't handle explicit null labels.

[taspelund]

I haven't tried this in a container, but I recall having blackholing issues with explicit-null labels until I enabled net.mpls.conf.lo.input = 1 in sysctl.
@Pcrossley69 maybe that would help in your situation as well?

[mjstapp]

There's also the long-standing issue that FRR zebra has when terminating LSPs. Historically, we've relied on PHP it seems, and zebra is not really willing to terminate an LSP locally. Implicit-null works for us for this reason. There may be some container-specific issue too, not aware of that, but in general at this time zebra doesn't really handle tunnel termination completely.