Add graceful-restart support #2386

karampok · 2024-04-26T11:40:37Z

/kind feature

What this PR does / why we need it:
Issue covers #2368

Special notes for your reviewer:

Release note:

TODO

karampok · 2024-04-26T13:20:33Z

Note to discuss:

Graceful restart configuration changed, reset this peer to take effect
% The Graceful Restart command used is not valid at this moment.
line 76: Failure to communicate[13] to bgpd, line:  neighbor 11.11.11.1 graceful-restart

[311|bfdd] done
[294|zebra] done
[300|bgpd] Configuration file[/etc/frr/frr.conf] processing failure: 13
2024-04-26 13:18:54,515 WARNING: frr-reload.py failed due to
vtysh (exec file) exited with status 13
Failed to fully apply configuration file 1 seconds
Caught SIGHUP and acquired lock! Reloading FRR..
Checking the configuration file syntax

it seems it fails, and retries to apply that the speaker/frr-reloader.

karampok · 2024-04-29T14:10:04Z

The above might be related to FRRouting/frr#8403

 vtysh -c configure -c 'router bgp 7003' -c 'bgp graceful-restart'
Graceful restart configuration changed, reset all peers to take effect
k00-worker:/# echo $?
0
k00-worker:/# vtysh -c configure -c 'router bgp 7003' -c 'neighbor 11.11.11.1 graceful-restart'
Graceful restart configuration changed, reset this peer to take effect
% The Graceful Restart command used is not valid at this moment.
k00-worker:/# echo $?
1

karampok · 2024-04-30T14:42:06Z

In my test: I achieved zero downtime if I have graceful restart either in global bgp graceful restarst or in neighbor mode neighbor x.y.z.k graceful restart and having that in bgp graceful-restart preserve-fw-state.

(Test= in runtime I delete the speaker and see if the client loses connectivity)

The second command that enables the F -bit is only available in global mode, not available per neighbor. Therefore a solution which we add graceful-restart in neighbor level and f-bit in global (because is not safe to just have that there by default) seem not optimal.

@fedepaol How much do you object to have it global (one for enable gracefull restart/one to on/off f-bit), and also have the option to opt out per neighbor? Any other ideas?

karampok · 2024-04-30T14:46:14Z

FRRouting/frr#15880 created this issue upstream, even though we are not really blocked on that (well we will get an error but doe not look like it has an impact)

fedepaol · 2024-05-07T14:18:58Z

The second command that enables the F -bit is only available in global mode, not available per neighbor. Therefore a solution which we add graceful-restart in neighbor level and f-bit in global (because is not safe to just have that there by default) seem not optimal.

@fedepaol How much do you object to have it global (one for enable gracefull restart/one to on/off f-bit), and also have the option to opt out per neighbor? Any other ideas?

Won't the global F-bit affect only peers with gr enabled? If so, doesn't sound too terrible to always enable it.

karampok · 2024-05-07T20:28:04Z

My preference would be to be configurable to be in the safe side. I cannot find argument against always having it (or being default on) but no idea for which case this option exists. Should we go always having it in the config?

In the same topic, should we not allow tuning the timer time? default is 120sec but seems long (ideally we need < holdtime)

fedepaol · 2024-05-08T13:23:28Z

My preference would be to be configurable to be in the safe side. I cannot find argument against always having it (or being default on) but no idea for which case this option exists. Should we go always having it in the config?

I'd try to keep the ux as best as we can, which means not having to set an extra parameter. We can either set the global when at least one peer needs GR, or just always enable it. I am leaning towards the second, as we'll have to deal with peers without GR and the global anyway, so if it doesn't work we'll have a problem.
Also, hopefully CI will tell us if there's a problem.

In the same topic, should we not allow tuning the timer time? default is 120sec but seems long (ideally we need < holdtime)

You mean the graceful restart timer? That should be meaningful only for the helper side I guess (but we may need it in frr-k8s where we might be helpers).

fedepaol · 2024-05-08T13:51:35Z

cc @oribon

api/v1beta2/bgppeer_types.go

e2etest/bgptests/bgp.go

api/v1beta2/bgppeer_types.go

internal/config/validation.go

e2etest/bgptests/bgp.go

fedepaol · 2024-05-27T15:21:39Z

e2etest/pkg/metallb/metallb.go

+
+		ret := true
+		for _, p := range pods {
+			ret = ret && k8s.PodIsReady(p)


can you just exit early if the pod is not ready?

e2etest/pkg/metallb/metallb.go

fedepaol · 2024-05-27T15:26:49Z

e2etest/pkg/metallb/metallb.go

+		}
+	}
+
+	f := func(context.Context) (bool, error) {


this can be embedded as anonymous function in the call below. No need to assign it to a variable.

done, but is there a drawback to have it assigned to variable?

e2etest/bgptests/bgp.go

e2etest/bgptests/metrics.go

e2etest/bgptests/bgp.go

fedepaol · 2024-05-30T12:30:22Z

e2etest/bgptests/bgp.go

+				}
+
+				if err := c(); err != nil {
+					if !errors.Is(err, ErrStaleRoute) {


why is staleroute skipped?

during the controlplane reboot (when peer has started graceful restart timers), the routes are stale and that is okay(happy path), we should ignore them during that time. We should NOT when we generally check for svc to be validated.

e2etest/bgptests/bgp.go

BGP graceful restart functionality as defined in RFC-4724 defines the mechanisms that allows BGP speaker to continue to forward data packets along known routes while the routing protocol information is being restored. This allows DS to be updated without routes being retracted in the peer side. We enable by default if GR then the F-bit (preserve-fw-state) to be set. We make gracefulRestart immutable according to https://kubernetes.io/blog/2022/09/29/enforce-immutability-using-cel/#immutablility-after-first-modification Signed-off-by: karampok <karampok@gmail.com>

Signed-off-by: karampok <karampok@gmail.com>

karampok · 2024-05-31T13:11:50Z

@fedepaol I have completed the last commit with adding the e2e test (plus the reverse to see a failure without graceful restart). I think I addressed all your comments. Let me know if I miss something. Thanks

e2etest/pkg/metallb/metallb.go

e2etest/bgptests/bgp.go

karampok · 2024-06-10T11:50:34Z

Another version pushed that does make the function not depend on external variable

e2etest/bgptests/bgp.go

fedepaol · 2024-06-12T18:34:31Z

e2etest/pkg/wget/wget.go

@@ -25,7 +25,7 @@ func Do(address string, exc executor.Executor) error {

 	// Retry loop to handle wget NetworkFailure errors
 	for {
-		out, err = exc.Exec("wget", "-O-", "-q", address, "-T", "60")
+		out, err = exc.Exec("wget", "-O-", "-q", address, "-T", "5")


is this change related?

yes a timeout of 60 seconds could block the serial test and hide a potential downtime

fedepaol · 2024-06-12T18:35:28Z

e2etest/pkg/frr/provider/provider.go

 	if !ok {
-		return nil, fmt.Errorf("speakers %s/%s not found in known speakers %v", ns, name, f.speakers)
+		return nil, fmt.Errorf("speakers %s/%s not found in known speakers %v", ns, name, speakers)


nit: speaker %s not found in..

fedepaol · 2024-06-12T18:37:30Z

e2etest/pkg/jigservice/jig.go

@@ -415,7 +415,7 @@ func (j *TestJig) UpdateService(ctx context.Context, update func(*v1.Service)) (

 // WaitForLoadBalancer waits the given service to have a LoadBalancer, or returns an error after the given timeout
 func (j *TestJig) WaitForLoadBalancer(ctx context.Context, timeout time.Duration) (*v1.Service, error) {
-	ginkgo.GinkgoWriter.Printf("Waiting up to %v for service %q to have a LoadBalancer", timeout, j.Name)
+	ginkgo.GinkgoWriter.Printf("\tWaiting up to %v for service %q to have a LoadBalancer\n", timeout, j.Name)


is this related? If not but makes sense, please add it in a separte commit

fedepaol · 2024-06-12T18:38:03Z

e2etest/pkg/metallb/metallb.go

+		}
+		return true, nil
+	})
+


nit: unnecessary line

fedepaol · 2024-06-12T18:38:47Z

few nits, almost there!

- Restarts speakers pod in non-blocking - Wait speakers to be ready and monitor at the same time if downtime BDD Description looks like BGP GracefulRestart, when speakers restart and when GR enabled dataplane should keep working BGP GracefulRestart, when speakers restart when GR disabled dataplane should have a downtime Signed-off-by: karampok <karampok@gmail.com>

Signed-off-by: karampok <karampok@gmail.com>

karampok

What about now :) let me know if you want to rebase. Thanks

karampok · 2024-06-13T07:04:46Z

e2etest/pkg/metallb/metallb.go

+		}
+		return true, nil
+	})
+


karampok · 2024-06-13T07:04:52Z

e2etest/pkg/wget/wget.go

@@ -25,7 +25,7 @@ func Do(address string, exc executor.Executor) error {

 	// Retry loop to handle wget NetworkFailure errors
 	for {
-		out, err = exc.Exec("wget", "-O-", "-q", address, "-T", "60")
+		out, err = exc.Exec("wget", "-O-", "-q", address, "-T", "5")


yes a timeout of 60 seconds could block the serial test and hide a potential downtime

karampok · 2024-06-13T07:06:11Z

e2etest/pkg/frr/provider/provider.go

 	if !ok {
-		return nil, fmt.Errorf("speakers %s/%s not found in known speakers %v", ns, name, f.speakers)
+		return nil, fmt.Errorf("speakers %s/%s not found in known speakers %v", ns, name, speakers)


karampok · 2024-06-13T07:15:36Z

e2etest/pkg/jigservice/jig.go

@@ -415,7 +415,7 @@ func (j *TestJig) UpdateService(ctx context.Context, update func(*v1.Service)) (

 // WaitForLoadBalancer waits the given service to have a LoadBalancer, or returns an error after the given timeout
 func (j *TestJig) WaitForLoadBalancer(ctx context.Context, timeout time.Duration) (*v1.Service, error) {
-	ginkgo.GinkgoWriter.Printf("Waiting up to %v for service %q to have a LoadBalancer", timeout, j.Name)
+	ginkgo.GinkgoWriter.Printf("\tWaiting up to %v for service %q to have a LoadBalancer\n", timeout, j.Name)


fedepaol · 2024-06-13T09:18:11Z

What about now :) let me know if you want to rebase. Thanks

rebase is not necessary as we use merge queues.
LGTM, sending to merge queue, thanks!

github-actions bot added the kind/feature label Apr 26, 2024

karampok force-pushed the main branch 4 times, most recently from 14ff603 to 0b43ca8 Compare May 13, 2024 13:32

karampok marked this pull request as ready for review May 13, 2024 14:03

karampok requested review from fedepaol, russellb, gclawes and oribon as code owners May 13, 2024 14:03

fedepaol reviewed May 14, 2024

View reviewed changes

api/v1beta2/bgppeer_types.go Outdated Show resolved Hide resolved

api/v1beta2/bgppeer_types.go Show resolved Hide resolved

e2etest/bgptests/bgp.go Show resolved Hide resolved

karampok force-pushed the main branch 2 times, most recently from 72e3c27 to d65bb88 Compare May 14, 2024 11:47

oribon reviewed May 15, 2024

View reviewed changes

api/v1beta2/bgppeer_types.go Outdated Show resolved Hide resolved

internal/config/validation.go Outdated Show resolved Hide resolved

karampok force-pushed the main branch 4 times, most recently from 50f6988 to 363fa20 Compare May 21, 2024 14:11

karampok force-pushed the main branch 4 times, most recently from 63944e7 to a7cbd33 Compare May 23, 2024 12:54

karampok force-pushed the main branch 2 times, most recently from 95e4970 to e57aee6 Compare May 27, 2024 08:40

fedepaol reviewed May 27, 2024

View reviewed changes

karampok force-pushed the main branch 2 times, most recently from 7459922 to f93f754 Compare May 30, 2024 07:27

fedepaol reviewed May 30, 2024

View reviewed changes

karampok added 2 commits May 30, 2024 14:40

E2E: add test for gracefulRestart

e1fb4d1

Signed-off-by: karampok <karampok@gmail.com>

karampok force-pushed the main branch 3 times, most recently from f59de1e to a286563 Compare May 31, 2024 12:12

fedepaol reviewed Jun 4, 2024

View reviewed changes

karampok force-pushed the main branch 2 times, most recently from 9e0c354 to 5925259 Compare June 10, 2024 09:48

fedepaol reviewed Jun 10, 2024

View reviewed changes

e2etest/bgptests/bgp.go Outdated Show resolved Hide resolved

karampok force-pushed the main branch 2 times, most recently from 09b162b to 8fed043 Compare June 10, 2024 14:16

fedepaol reviewed Jun 12, 2024

View reviewed changes

karampok added 2 commits June 13, 2024 09:13

Improve print output of jig pkg

2d52178

Signed-off-by: karampok <karampok@gmail.com>

karampok force-pushed the main branch from 8fed043 to 2d52178 Compare June 13, 2024 07:15

karampok commented Jun 13, 2024

View reviewed changes

fedepaol added this pull request to the merge queue Jun 13, 2024

Merged via the queue into metallb:main with commit e9aec9c Jun 13, 2024
34 of 35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add graceful-restart support #2386

Add graceful-restart support #2386

karampok commented Apr 26, 2024

karampok commented Apr 26, 2024

karampok commented Apr 29, 2024 •

edited

karampok commented Apr 30, 2024

karampok commented Apr 30, 2024

fedepaol commented May 7, 2024 •

edited

karampok commented May 7, 2024 •

edited

fedepaol commented May 8, 2024

fedepaol commented May 8, 2024

fedepaol May 27, 2024

karampok May 30, 2024

fedepaol May 27, 2024

karampok May 30, 2024

fedepaol May 30, 2024

karampok May 30, 2024

karampok commented May 31, 2024

karampok commented Jun 10, 2024

fedepaol Jun 12, 2024

karampok Jun 13, 2024

fedepaol Jun 12, 2024

karampok Jun 13, 2024

fedepaol Jun 12, 2024

karampok Jun 13, 2024

fedepaol Jun 12, 2024

karampok Jun 13, 2024

fedepaol commented Jun 12, 2024

karampok left a comment

karampok Jun 13, 2024

karampok Jun 13, 2024

karampok Jun 13, 2024

karampok Jun 13, 2024

fedepaol commented Jun 13, 2024

Add graceful-restart support #2386

Add graceful-restart support #2386

Conversation

karampok commented Apr 26, 2024

karampok commented Apr 26, 2024

karampok commented Apr 29, 2024 • edited

karampok commented Apr 30, 2024

karampok commented Apr 30, 2024

fedepaol commented May 7, 2024 • edited

karampok commented May 7, 2024 • edited

fedepaol commented May 8, 2024

fedepaol commented May 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karampok commented May 31, 2024

karampok commented Jun 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedepaol commented Jun 12, 2024

karampok left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedepaol commented Jun 13, 2024

karampok commented Apr 29, 2024 •

edited

fedepaol commented May 7, 2024 •

edited

karampok commented May 7, 2024 •

edited