Sougatab/bfd adminshut periodic tx#22643
Conversation
Problem ------- When a BFD session (or a whole BFD profile) is administratively shut down, bfd_set_shutdown() moved the session to AdminDown, deleted the transmit timer, and sent exactly ONE Control packet carrying State=AdminDown. After that the session went silent. If that single AdminDown packet is dropped, or simply loses the race with the remote's detection timer (detect_mult * negotiated interval), the peer never learns we went AdminDown. Instead it expires the session with diag "Control Detection Time Expired", declares the session Down, and tears down whatever protocol BFD protects. For BGP this means the peer sends a Cease/BFD Down NOTIFICATION and our session resets. This is nondeterministic and gets worse at scale and with short timers: shutting a profile fans out an AdminDown burst across many sessions in a few milliseconds, so a handful reliably lose the race. Observed on a 304-session box: 6-9 BGP sessions reset per profile shut/no-shut with 50ms/mult-3 (150ms detection) timers. RFC recommendation ------------------ RFC 5880 Section 6.8.16 (Administrative Control): "BFD Control packets SHOULD be transmitted for at least a Detection Time after transitioning to AdminDown state in order to ensure that the remote system is aware of the state change. BFD Control packets MAY be transmitted indefinitely after transitioning to AdminDown state in order to maintain session state in each system." Sending a single packet transmits for ~0 time, violating this SHOULD. Solution -------- Do not delete the transmit timer on shutdown. Keep advertising State=AdminDown, but slow the transmit rate to BFD_DEF_SLOWTX (1s) since the session is no longer Up (RFC 5880 Section 6.8.3), mirroring the existing ptm_bfd_sess_dn() DOWN handling. Echo and the receive/detection timers are still disabled - we only need to keep informing the peer, not detect it, while administratively down. The remote now reliably receives AdminDown (diag "Neighbor signaled session down", RFC 5880 Section 6.8.6) and holds the protocol session up instead of timing it out. Cost is 1 packet/s per session while shut. For offloaded (data-plane) sessions the same guarantee must be enforced in the data-plane program (keep emitting AdminDown); a comment marks that path. Signed-off-by: Sougata Barik <sougatab@nvidia.com>
Add a topotest that guards the BFD administrative-shutdown transmit behaviour fixed in the preceding commit. Problem it reproduces --------------------- Before the fix, bfd_set_shutdown() deleted the transmit timer and sent a single Control packet with State=AdminDown, then went silent. A lost AdminDown (or one that lost the race with the peer's detection timer) left the peer to time the session out and tear down BGP with a Cease/BFD Down NOTIFICATION. RFC 5880 Section 6.8.16 says AdminDown "SHOULD be transmitted for at least a Detection Time" so the remote reliably learns of the state change. What the test checks -------------------- Topology r1 --- r2 with eBGP + BFD (300ms/mult-3). * test_reproduce_send_once_regression: after admin-shutdown on r1, capture outbound BFD with tcpdump and require MORE THAN ONE AdminDown Control packet. Against the old send-once code exactly one (or zero) is seen, so this fails and reproduces the bug. * test_fix_periodic_admindown_and_bgp_stable: end-to-end - r2 observes the remote AdminDown, r1 keeps re-transmitting AdminDown while shut, BGP stays Established on both routers, connectionsDropped is unchanged across the shut/no-shut cycle, and the session recovers. Signed-off-by: Sougata Barik <sougatab@nvidia.com>
Greptile SummaryThis PR changes BFD administrative shutdown to keep advertising AdminDown. The main changes are:
Confidence Score: 4/5This is close, but the offloaded shutdown path should be fixed before merging.
bfdd/bfd.c Important Files Changed
Prompt To Fix All With AIFix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
bfdd/bfd.c:1795-1800
**Offload still exits early**
When `bs->bdc` is set, this branch still sends one dataplane update and returns before the new slow AdminDown transmit path is reached. That leaves the offloaded case dependent on an external dataplane behavior that this code does not program or verify. If the dataplane only reacts to `SESSION_SHUTDOWN` once, the peer can still miss the AdminDown packet, expire detection, and reset the protected protocol. Please make this branch explicitly program periodic AdminDown transmission for offloaded sessions, or route it through an equivalent periodic sender.
Reviews (2): Last reviewed commit: "tests: bfdd: topotest for periodic Admin..." | Re-trigger Greptile |
c8741cc to
d54fd6b
Compare
|
CI:rerun |
|
Can you take your username out of the title and make it a little more descriptive? |
riw777
left a comment
There was a problem hiding this comment.
Should there also be a new state in YANG and show commands to tell the operator the session is shut down but still sending admin shut packets? Or is this state too short to be worried about it?
| hostname r1 | ||
| ! | ||
| interface lo | ||
| ip address 1.1.1.1/32 |
There was a problem hiding this comment.
Please use documentation addresses rather than publicly routeable.
|
Will we continue sending the admin down notifications "permanently" with this fix? Should we stop after detection time? |
Uh oh!
There was an error while loading. Please reload this page.