Skip to content

stupid simple retry #1084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

stupid simple retry #1084

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a705245
stupid simple retry
paulgmiller Nov 4, 2021
a54d93e
go lint fixes
paulgmiller Nov 4, 2021
92dd540
missed one //
paulgmiller Nov 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions cns/service/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@ const (

// 720 * acn.FiveSeconds sec sleeps = 1Hr
maxRetryNodeRegister = 720
// 10 * 30 seconds = 5 minutes of retrying before we crash and backoff.
maxRetryInit = 10
retryIntervalInit = 30 * time.Second
)

var (
Expand Down Expand Up @@ -884,9 +887,18 @@ func InitializeCRDState(ctx context.Context, httpRestService cns.HTTPService, cn
}
}()

err = initCNS(ctx, scopedcli, httpRestServiceImplementation)
if err != nil {
return errors.Wrap(err, "failed to initialize CNS state")
// apiserver nnc might not be registered or api server might be down and crashloop backof puts us outside of 5-10 minutes we have for
// aks addons to come up so retry a bit more aggresively here.
for tryNum := 0; tryNum <= maxRetryInit; tryNum++ {
err = initCNS(ctx, scopedcli, httpRestServiceImplementation)
if err == nil {
break
}
logger.Errorf("[Azure CNS] failed to init cns: %v", err)
if tryNum >= maxRetryInit {
return errors.Wrap(err, "failed to initialize CNS state")
}
time.Sleep(retryIntervalInit)
}

manager, err := ctrl.NewManager(kubeConfig, ctrl.Options{
Expand Down