[21.05] Improve router behaviour during failovers

[sysvinit]

We've had some problems with temporary (ca. 20s) loss of connectivity when performing failovers with NixOS routers in VXLAN (both between Gentoo and NixOS and with NixOS pairs). For comparison, in the pre-VXLAN world failover between Gentoo routers was near-instantaneous.

One half of this problem is that by default we enable ARP and ND suppression on VXLAN interfaces. This is recommended by most of the tutorials I've seen for setting up VXLAN, as this reduces the amount of flooding traffic which needs to be replicated to all VTEPs in the fabric. However, flood suppression means that the gratuitous ARPs and NDs sent by keepalived when it's promoted from backup to primary router are dropped, which prevents the new primary router from being able to promptly steer outgoing traffic away from the demoted router.

The other half of this problem is that the firewall rule for allowing incoming TFTP packets was only enabled for the primary specialisation, and not the base specialisation. This means that switching between specialisations would cause a firewall reload, and reloading the NixOS firewall causes all incoming packets to be dropped for a few seconds. This is enough to interrupt BFD and BGP sessions, which (depending on the local configuration) can cause routers to temporarily lose their default route. If the router has just been promoted to primary, then the entire site loses connectivity until the BGP session comes back up and the default routes are re-learned.

This change splits the VXLAN bridge port configuration into separate units for flood suppression and MAC learning suppression, and then disables the former in the router role only. This means that flood suppression will stay enabled on all other physical hosts. Additionally, the firewall rules for TFTP are enabled unconditionally, irrespective of whether the router is the primary or not, in order to prevent a firewall reload when performing a failover.

PL-132482

@flyingcircusio/release-managers

Release process

Impact: internal

Changelog: none

PR release workflow (internal)

• PR has internal ticket
• internal issue ID (PL-…) part of branch name
• internal issue ID mentioned in PR description text
• ticket is on Platform agile board
• ticket state set to Pull request ready
• if ticket is more urgent than within the next few days, directly contact a member of the Platform team

Design notes

• Provide a feature toggle if the change might need to be adjusted/reverted quickly depending on context. Consider whether the default should be on or off. Example: rate limiting.
• All customer-facing features and (NixOS) options need to be discoverable from documentation. Add or update relevant documentation such that hosted and guided customers can understand it as well.

Security implications

• Security requirements defined? (WHERE)
  • Flood suppression is generally desirable to prevent packets being flooded unnecessarily, especially on multi-tenant hosts such as KVM servers. Suppression is hence only disabled on routers.
  • No other requirements, this change improves the integration of the router role in the platform.
• Security requirements tested? (EVIDENCE)
  • Manually verified in DEV. Failing over between the NixOS and Gentoo router causes no visible disruption in an mtr trace left running on a VM in DEV.