Skip to content

Commit

Permalink
refactor retry to close watchFactory
Browse files Browse the repository at this point in the history
leverage stop channel on watch factory when periodic retry stopchannel is called

Problem: Hybrid test was creating creating an ovn-controller for some tests. The tests could not use the waitGroup in the test
so the afterEach function (cleanup function after each test) could execute while leaving controllers up. In the tests, we were leveraging the code in ovn.go to run the controllers and creating a watcher pod on them. This issue was just a symptom of the main issue which was that we were using the incorrect stopChannel in the retry logic.

Solution: The stopChannel that we use in the watchFactory is private, so we created a function that when called will close the watcher stopChannel, closing the watcher. We then call this function in the select (basically a switch statement in go but for channels) to close the watcher, because when we stop watching an object, that means we no longer care about it and should stop retrying it.

Signed-off-by: Ben Pickard <bpickard@redhat.com>
Closes: #3203
  • Loading branch information
bpickard22 committed Oct 21, 2022
1 parent 6e00308 commit 01622a4
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 1 deletion.
5 changes: 5 additions & 0 deletions go-controller/pkg/factory/factory.go
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,11 @@ func NewNodeWatchFactory(ovnClientset *util.OVNClientset, nodeName string) (*Wat
return wf, nil
}

func (wf *WatchFactory) WaitForWatchFactoryStopChannel(stopChan chan struct{}) {
<-wf.stopChan
close(stopChan)
}

func (wf *WatchFactory) Shutdown() {
close(wf.stopChan)

Expand Down
6 changes: 5 additions & 1 deletion go-controller/pkg/retry/obj_retry.go
Original file line number Diff line number Diff line change
Expand Up @@ -364,6 +364,10 @@ func (r *RetryFramework) iterateRetryResources() {
// RetryObjInterval seconds or when requested through retryChan.
func (r *RetryFramework) periodicallyRetryResources() {
timer := time.NewTicker(RetryObjInterval)
waitCh := make(chan struct{})
go func() {
r.watchFactory.WaitForWatchFactoryStopChannel(waitCh)
}()
defer timer.Stop()
for {
select {
Expand All @@ -375,7 +379,7 @@ func (r *RetryFramework) periodicallyRetryResources() {
r.iterateRetryResources()
timer.Reset(RetryObjInterval)

case <-r.StopChan:
case <-waitCh:
klog.V(5).Infof("Stop channel got triggered: will stop retrying failed objects of type %s", r.ResourceHandler.ObjType)
return
}
Expand Down

0 comments on commit 01622a4

Please sign in to comment.