Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compensate for raft/cache delay in namespace admission #32719

Merged
merged 1 commit into from Sep 21, 2016
Merged

compensate for raft/cache delay in namespace admission #32719

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
23 changes: 22 additions & 1 deletion plugin/pkg/admission/namespace/lifecycle/admission.go
View file
Open in desktop
Expand Up @@ -21,6 +21,7 @@ import (
"io"
"time"

"github.com/golang/glog"
lru "github.com/hashicorp/golang-lru"

"k8s.io/kubernetes/pkg/client/cache"
Expand All @@ -38,6 +39,12 @@ const (
PluginName = "NamespaceLifecycle"
// how long a namespace stays in the force live lookup cache before expiration.
forceLiveLookupTTL = 30 * time.Second
// how long to wait for a missing namespace before re-checking the cache (and then doing a live lookup)
// this accomplishes two things:
// 1. It allows a watch-fed cache time to observe a namespace creation event
// 2. It allows time for a namespace creation to distribute to members of a storage cluster,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: there is a case 3 which is non-HA, namespace is marked for deletion, and the admission controller local cache is purged. the next attempt to create a resource in that namespace will wait as well and reduce our flakiness in e2e tests.

// so the live lookup has a better chance of succeeding even if it isn't performed against the leader.
missingNamespaceWait = 50 * time.Millisecond
)

func init() {
Expand Down Expand Up @@ -113,6 +120,19 @@ func (l *lifecycle) Admit(a admission.Attributes) error {
return errors.NewInternalError(err)
}

if !exists && a.GetOperation() == admission.Create {
// give the cache time to observe the namespace before rejecting a create.
// this helps when creating a namespace and immediately creating objects within it.
time.Sleep(missingNamespaceWait)
namespaceObj, exists, err = l.namespaceInformer.GetStore().Get(key)
if err != nil {
return errors.NewInternalError(err)
}
if exists {
glog.V(4).Infof("found %s in cache after waiting", a.GetNamespace())
}
}

// forceLiveLookup if true will skip looking at local cache state and instead always make a live call to server.
forceLiveLookup := false
lruItemObj, ok := l.forceLiveLookupCache.Get(a.GetNamespace())
Expand All @@ -123,14 +143,15 @@ func (l *lifecycle) Admit(a admission.Attributes) error {

// refuse to operate on non-existent namespaces
if !exists || forceLiveLookup {
// in case of latency in our caches, make a call direct to storage to verify that it truly exists or not
// as a last resort, make a call directly to storage
namespaceObj, err = l.client.Core().Namespaces().Get(a.GetNamespace())
if err != nil {
if errors.IsNotFound(err) {
return err
}
return errors.NewInternalError(err)
}
glog.V(4).Infof("found %s via storage lookup", a.GetNamespace())
}

// ensure that we're not trying to create objects in terminating namespaces
Expand Down