Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

retry Update requests #901

Closed
f41gh7 opened this issue Mar 12, 2024 · 1 comment
Closed

retry Update requests #901

f41gh7 opened this issue Mar 12, 2024 · 1 comment
Assignees
@f41gh7
Labels
enhancement New feature or request

Comments

@f41gh7
Copy link
Collaborator

f41gh7 commented Mar 12, 2024

Currently, operator may produce a conflict error if object was already modified by some kubernetes controller or any other external controller.

This error can be retried with the following code:

err := retry.RetryOnConflict(retry.DefaultRetry, func() error {    var res apiv1.MyResource    err := r.Get(ctx, types.NamespacedName{        Name:      resourceName,        Namespace: resourceNamespace,    }, &res)    if err != nil {        return err    }    res.Status.Replias = readyReplicas        return r.Status().Update(ctx, &res)})if err != nil {    return fmt.Errorf("failed to update resource status: %w", err)}

https://alenkacz.medium.com/kubernetes-operators-best-practices-understanding-conflict-errors-d05353dff421
https://pkg.go.dev/k8s.io/client-go/util/retry#RetryOnConflict
@f41gh7 f41gh7 added the enhancement New feature or request label Mar 12, 2024
@f41gh7 f41gh7 self-assigned this Apr 2, 2024
f41gh7 added a commit that referenced this issue Apr 12, 2024
@f41gh7
operator: changes errors handling for reconcile loop
bc09a9b 
* supress conflict errors and retry it in 5 seconds by default
* supress context.Cancelled errors, it either timeout or operator gracefull shutdown
* supress notFound object errors. Most probably it's race condition at kubernetes state and operator cache
* create kubernetes API event with object link in case of reconcile error. It must help investigate operator issue faster
* adds retry.OnConflict for statefulset and deployment updates

#900
#901
@f41gh7
Copy link
Collaborator Author

f41gh7 commented Apr 18, 2024

Operator performs proper checks for conflict error and retry requests.

Must be fixed at v0.43.0 release

@f41gh7 f41gh7 closed this as completed Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@f41gh7 f41gh7

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@f41gh7