Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

e2e flake fix: Namespace controller error handling improvements #22067

Merged
merged 1 commit into from
Feb 26, 2016
Merged

e2e flake fix: Namespace controller error handling improvements #22067

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions pkg/controller/namespace/namespace_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,16 @@ func (nm *NamespaceController) worker() {
if err := nm.syncNamespaceFromKey(key.(string)); err != nil {
if estimate, ok := err.(*contentRemainingError); ok {
go func() {
defer utilruntime.HandleCrash()
t := estimate.Estimate/2 + 1
glog.V(4).Infof("Content remaining in namespace %s, waiting %d seconds", key, t)
time.Sleep(time.Duration(t) * time.Second)
nm.queue.Add(key)
}()
} else {
// rather than wait for a full resync, re-add the namespace to the queue to be processed
nm.queue.Add(key)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually wait (I mean not wait with this PR, it's correct, but wait with #22045).

Now I think, it that was something different, because even if we don't readd it here, we will sec the namespace every resync period, which is now 5 minutes:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/options/options.go#L61

If that's the case, if we didn't remove the namespace for 1h30m (which was the case in that issue), this would mean that we had errors that we didn't handled correctly ~100 times, which seems quite non-realistic.
@derekwaynecarr - am I missing something?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping the logging would have shown the underlying cause of 22045, but I am fine removing the fixes until we have that logging.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - I thought that I missed something.

utilruntime.HandleError(err)
}
}
}()
Expand Down
6 changes: 6 additions & 0 deletions pkg/controller/namespace/namespace_controller_utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,12 @@ func syncNamespace(kubeClient clientset.Interface, versions *unversioned.APIVers
// we have removed content, so mark it finalized by us
result, err := retryOnConflictError(kubeClient, namespace, finalizeNamespaceFunc)
if err != nil {
// in normal practice, this should not be possible, but if a deployment is running
// two controllers to do namespace deletion that share a common finalizer token it's
// possible that a not found could occur since the other controller would have finished the delete.
if errors.IsNotFound(err) {
return nil
}
return err
}

Expand Down