Fix route reconciliation by enabling fpm

[l0wl3vel]

Description

Add a frr.conf template including fpm address 127.0.0.1 to connect frr to the SONiC Forwarding Plane.

SONiC at some point in H2 2024 moved to their own FRR FPM module(Forwarding Plane Manager) called dplane_fpm_sonic. (Design Doc) Unsure if route propagation using the frr.conf template used by metal-core worked before that, but from this point on fpm address 127.0.0.1 was strictly required for FRR/Zebra to push routes to the SONiC FPM.

This lead to broken route propagation from FRR to the sonic forwarding plane, which was worked around by restarts of FRR and metal-core, that for some reason I do not yet understand, forced propagation into the forwarding plane. Maybe there is a workaround in metal-core that periodically/at restart copies routes from Zebra to the SONiC data plane?

Old SONiC distributions from before that change may or may not be affected. Our Dell Sonic versions always used fpm, not dplane_fpm_nl or dplane_fpm_sonic, so are not affected by the switch as far as I can tell.

Adding the config file override as a workaround for now until we figure out which SONiC versions and distributions are affected. Goal is to get this into the frr.conf templates in metal-roles/metal-core asap.

Used AI-Tools ✨

• Claude Opus 4.8 used for launch.py startup adjustments

[Gerrit91]

Pretty insane finding if this is true! 💪

At least the pipeline works without any metal-core restarts, which is proof enough for me. 😅

I would merge it immediately, but maybe someone who knows something about SONiC and networking should have a look at it, too.

[majst01]

If this is the only difference, it might be worth to check of this could/must be added to our general metal-core frr template instead of copying to this repo.

[l0wl3vel]

yep, it is the only difference. Took the metal-core default template and added one line.

Agreed. This belongs in metal-core/metal-roles asap. Wanted to merge this here first to make it available, since my gatewayapi PR network restructuring broke our old workarounds in some funny new way that this fully fixes.

IMO we need validation on in production use sonic flavors (dell, edgecore, broadcom) and versions before metal-roles. Can we do the validation in a seperate high-prio metal-core/metal-roles issue and then remove this workaround?

[majst01]

since which version is this configuration available ?

[Sven-Ric]

Community/Edgecore 202411. Dell and Broadcom are still on fpm in the latest release. The fpm equivalent of this command is also slightly different, so fpm address 127.0.0.1 wouldn't be accepted on switches running with -M fpm

[l0wl3vel]

@Sven-Ric community sonic is also launching FRR with -M fpm, meaning that the launch arg will not show you which module is loaded. The module is being compiled into You can check which module is loaded using docker exec -it bgp vtysh -c 'show zebra dplane providers'

# docker exec -it bgp vtysh -c 'show zebra dplane providers'
dataplane Incoming Queue from Zebra: 0
Zebra dataplane providers:
  Kernel (1): in: 55, q: 0, q_max: 5, out: 55, q: 0, q_max: 5
  dplane_fpm_sonic (2): in: 55, q: 0, q_max: 5, out: 55, q: 0, q_max: 5
dataplane Outgoing Queue to Zebra: 0

And @Sven-Ric is right:
dplane_fpm_sonic and dplane_fpm_nl use fpm address A.B.C.D for endpoint configuration, while fpm uses fpm connection ip A.B.C.D. (docs)We need to check which module is really loaded on the distros and use the right syntax depending on which one is used.

We can tell if anything is needed at all by looking at the factory settings on a freshly installed switch. An unmodified frr.conf on a sonic community switch will have fpm address 127.0.0.1 present, for example, which tipped me of to the issue in the first place.

[Sven-Ric]

@l0wl3vel Maybe I am missing something, but on a 202511 community SONiC container I get the following.

admin@sonic:~$ docker exec bgp ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  3.8  0.8  41328 35496 pts/0    Ss+  18:34   0:00 /usr/bin/python3 /usr/local/bin/supervisord
root          37  0.2  0.1 221840  4724 pts/0    Sl   18:34   0:00 /usr/sbin/rsyslogd -n -iNONE
frr           41  1.0  0.3  19332 12664 pts/0    S    18:34   0:00 /usr/lib/frr/mgmtd -A 127.0.0.1 -P 0
frr           42  1.0  0.4 655184 18772 pts/0    Sl   18:34   0:00 /usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M dplane_fpm_sonic -M snmp --asic-offload=notify_on_offload
frr           57  0.2  0.2  15800  8640 pts/0    S    18:34   0:00 /usr/lib/frr/staticd -A 127.0.0.1 -P 0
frr           58  2.7  1.1 290516 45304 pts/0    Sl   18:34   0:00 /usr/lib/frr/bgpd -A 127.0.0.1 -P 0 -M snmp
root          60  4.2  0.9 127524 37348 pts/0    Sl   18:34   0:00 /usr/bin/python3 /usr/local/bin/bgpcfgd
root          61  0.5  0.5  38088 21364 pts/0    S    18:34   0:00 /usr/bin/python3 /usr/local/bin/bgpmon
root          65  0.0  0.2  96736  7984 pts/0    Sl   18:34   0:00 fpmsyncd
root          66  0.2  0.5  38584 22836 pts/0    S    18:34   0:00 /usr/bin/python3 /usr/local/bin/staticroutebfd
root         109  0.0  0.2  97212  9340 pts/0    Sl   18:34   0:00 /usr/bin/supervisor-proc-exit-listener-rs --container-name bgp
root         112 75.0  0.1   8096  4208 ?        Rs   18:34   0:00 ps aux

The dplane_fpm_sonic module is explicitly used here.

[Sven-Ric]

We can tell if anything is needed at all by looking at the factory settings on a freshly installed switch. An unmodified frr.conf on a sonic community switch will have fpm address 127.0.0.1 present, for example, which tipped me of to the issue in the first place.

The show commands for each module are also different. No factory settings necessary.

On community SONiC with the new dplane_fpm_sonic module loaded.

sonic# show fpm status
 Address to connect to         127.0.0.1
 Port                          2620
 Connected                     Yes
 Use Nexthop Groups            No
 Use Route Replace Semantics   Yes
 Disabled                      No

On enterprise SONiC with the old fpm module

sonic# show fpm status
% Unknown command: show fpm status

[l0wl3vel]

yup, remembered it wrong, my bad. Thought the module was overwritten instead of added to FRR. The -M parameter shows you the module loaded.

yeah, but the factory default settings might tell us what is broken in other sonic distributions, like dell_sonic. Because the working theories are 1. dell sonic is just has broken route propagation for bgp, which I find hard to believe for a intensively used feature in a commercially sold product in version 4.x.x or 2. We have a similar issue at our hands, cause we misconfigure FRR as well, but differently, manifesting in the same problems of routes not propagating.

I think 2 is much more likely than 1 due to the similar symptoms/workarounds across platforms and the fact we replace parts of the sonic runtime template configuration with our own templates.

[Sven-Ric]

Yes, looks good. The old fpm module defaults to the correct address without explicit configuration. Looks like the new dplane_fpm_sonic only defaults the port but not the address.