Skip to content

fix: Using retryOnConflict to fix v4overlay scale tests failing. #2314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 27, 2023
Merged

fix: Using retryOnConflict to fix v4overlay scale tests failing. #2314

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
19028c6
fix: ctry using retryOnConflict for v4overlay scale test
nddq Oct 19, 2023
de7ddad
fix: typo
nddq Oct 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pipelines/singletenancy/dualstack-overlay/dualstackoverlay-e2e-step-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,5 +154,5 @@ steps:
cd test/integration/datapath
sudo -E env "PATH=$PATH" go test -count=1 datapath_windows_test.go -timeout 3m -tags connection -restartKubeproxy true -run ^TestDatapathWin$
name: "WindowsDualStackOverlayDatapathTests"
displayName: "Windows DaulStack Overlay Datapath Tests"
displayName: "Windows DualStack Overlay Datapath Tests"
Copy link
Contributor

@jpayne3506 jpayne3506 Oct 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good eye.

Copy link
Member Author

@nddq nddq Oct 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after looking at a lot of pipelines running, it started to bug me 😂

retryCountOnTaskFailure: 3
28 changes: 21 additions & 7 deletions test/internal/kubernetes/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/tools/remotecommand"
"k8s.io/client-go/util/homedir"
k8sRetry "k8s.io/client-go/util/retry"
)

const (
Expand Down Expand Up @@ -301,14 +302,27 @@ func WaitForPodDaemonset(ctx context.Context, clientset *kubernetes.Clientset, n
}

func MustUpdateReplica(ctx context.Context, deploymentsClient typedappsv1.DeploymentInterface, deploymentName string, replicas int32) {
deployment, err := deploymentsClient.Get(ctx, deploymentName, metav1.GetOptions{})
if err != nil {
panic(errors.Wrapf(err, "could not get deployment %s", deploymentName))
}
retryErr := k8sRetry.RetryOnConflict(k8sRetry.DefaultRetry, func() error {
// Get the latest Deployment resource.
deployment, getErr := deploymentsClient.Get(ctx, deploymentName, metav1.GetOptions{})
if getErr != nil {
return fmt.Errorf("failed to get deployment: %w", getErr)
}

// Modify the number of replicas.
deployment.Spec.Replicas = Int32ToPtr(replicas)

// Attempt to update the Deployment.
_, updateErr := deploymentsClient.Update(ctx, deployment, metav1.UpdateOptions{})
if updateErr != nil {
return fmt.Errorf("failed to update deployment: %w", updateErr)
}

return nil // No error, operation succeeded.
})

deployment.Spec.Replicas = Int32ToPtr(replicas)
if _, err := deploymentsClient.Update(ctx, deployment, metav1.UpdateOptions{}); err != nil {
panic(errors.Wrapf(err, "could not update deployment %s", deploymentName))
if retryErr != nil {
panic(errors.Wrapf(retryErr, "could not update deployment %s", deploymentName))
}
}

Expand Down