bgpd: unremovable neighbors

[tigranmartirosyan]

Hi

we faced some strange thing with Cumulus Quagga, and suspect same problem can be in the FRR and would like to fix for the FRR actually,
but not sure how to reproduce / collect info.

There is one bgp neighbor which is not possible to delete

pmx2-lc(config-router)# no neighbor 169.254.3.158

this command passes without any error, but neighbor stays in the config

pmx2-lc(config-router)# do sh ip bgp sum
BGP router identifier 185.47.156.72, local AS number 65002 vrf-id 0
BGP table version 10800
RIB entries 36, using 4320 bytes of memory
Peers 15, using 242 KiB of memory
Peer groups 1, using 56 bytes of memory
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
sw-r02-lc(eth0) 4 65001 2274657 2302512        0    0    0 11w1d16h        2
sw-r01-lc(eth1) 4 65001 2153318 2181383        0    0    0 01w6d06h        2
lc-asp-03.inn.local(tap142i0)
               4 65501    5252    5250        0    0    0 01:55:45        3
kafka3-lc(tap156i0)
               4 65501 1386887 1387056        0    0    0 06w6d00h        1
graphite12-lc(tap159i0)
               4 65501  155671  155670        0    0    0 5d06h52m        2
169.254.0.22    4 65501 2563492 2254328        0    0    0 06w5d23h        1
169.254.2.178   4 65501    2694    3669        0    0    0 00:09:25 Active    
169.254.3.110   4 65501 2506996 2203248        0    0    0 6d06h59m        1
169.254.3.114   4 65501 2506872 2203170        0    0    0 6d06h58m        1
169.254.3.118   4 65501 2506884 2203161        0    0    0 6d06h58m        1
169.254.3.134   4 65501 2565467 2254302        0    0    0 01w6d05h        1
169.254.3.138   4 65501 2564879 2254199        0    0    0 01w6d05h        1
169.254.3.158   4 65501 2561502 2248955        0    0    0 02:07:54 Active       <--- THIS one  - UNREMOVABLE neighbor
169.254.3.162   4 65501 2567715 2255136        0    0    0 11w0d19h        1
169.254.3.170   4 65501 2579032 2265582        0    0    0 02w0d15h        1 (edited)

That neighbor is not recognized by configurator which allows to add second neighbor with the same IP

169.254.3.138   4 65501 2565041 2254344        0    0    0 01w6d05h        1
169.254.3.158   4 65501 2561502 2248955        0    0    0 02:15:03 Active        <-------- undeletable neighbor
169.254.3.158   4   888       0       0        0    0    0 never    Active        <-------- new neighbor
169.254.3.162   4 65501 2567879 2255281        0    0    0 11w0d19h        1

then deleting will remove only new one

pmx2-lc(config-router)# no neighbor 169.254.3.158
pmx2-lc(config-router)# no neighbor 169.254.3.158
pmx2-lc(config-router)# no neighbor 169.254.3.158

169.254.3.138   4 65501 2565041 2254344        0    0    0 01w6d05h        1
169.254.3.158   4 65501 2561502 2248955        0    0    0 02:15:03 Active         <-------- undeletable neighbor
169.254.3.162   4 65501 2567879 2255281        0    0    0 11w0d19h        1

The problem was solved after quagga daemon had been restarted.

[donaldsharp]

@dwalton76 I thought you had fixed this already?

[tigranmartirosyan]

No, it is not fixed yet. We cannot reproduce this problem on our dev environment. 3 days ago we had such kind of problem for one of our customer. We did not restart the quagga daemon to collect more information regarding this issue.
What information should I collect to better troubleshoot the source of the issue ?
Could you also suggest me in which part of the source code should I look at ?

[tigranmartirosyan]

Could you please assign ticket to me. I am trying to reproduce problem on our dev environment.

[dwalton76]

@tigranmartirosyan can you send me the steps on how to reproduce this one?

[tigranmartirosyan]

@dwalton76 we are still in process to reproduce the problem on our dev environment. The problem is still actual as it was noticed by our customer a couple of time during last 2 weeks. I will update this topic when I have some news.

[tigranmartirosyan]

So it is happening but can't constantly reproduce. Hope to find the right trigger next week. Will keep you posted. Kind of ghost bug.

[dwalton76]

gotcha, thanks @tigranmartirosyan

[dwalton76]

@tigranmartirosyan any luck figuring out how to reproduce this one?

[tigranmartirosyan]

No luck to reproduce. We have not seen this problem again with frr.

[dwalton76]

Ack, will close this, please re-open if you see it again.