unable to get: x/y because of unknown namespace for the cache

[barkbay]

I observed this error in our e2e tests, it seems to be harmless but it makes the logs harder to read and I think we should understand where it comes from:

{
	"log.level": "error",
	"@timestamp": "2021-01-28T01:42:34.794Z",
	"log.logger": "manager.eck-operator.controller.beat-es-association-controller",
	"message": "Reconciler error",
	"service.version": "1.5.0-SNAPSHOT+7257dd51",
	"service.type": "eck",
	"ecs.version": "1.4.0",
	"name": "filebeat",
	"namespace": "e2e-xdhm8",
	"error": "unable to get: e2e-xdhm8/filebeat because of unknown namespace for the cache",
	"error.stack_trace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.1/pkg/internal/controller/controller.go:252\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.1/pkg/internal/controller/controller.go:215\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99"
}

[pebrc]

This message comes from the multi-namespace cache in controller runtime. Any user running ECK in restricted mode with access to only a subset of namespaces will run into this.

So it seems that the controllers get the events for edits on all instances of our CRDs no matter whether they are inside one of the relevant namespaces. We could implement our own Predicate that filters out any event for irrelevant namespaces. Also I wonder if this is something we should raise as an issue against controller runtime.

[ferozsalam]

I would appreciate this being filtered out by default - it causes a large amount of noise which makes other issues more difficult to identify.

[barkbay]

Just to add some context and explain what is happening (it took me a moment to get it): this issue happens iif the following 2 conditions are met:

1. the operator is managing a restricted set of namespaces
2. storage class validation is enabled

In that particular case we add an empty namespace ("") to the list of managed namespaces:

log.Info("Operator configured to manage multiple namespaces", "namespaces", managedNamespaces, "operator_namespace", operatorNamespace)
// The managed cache should always include the operator namespace so that we can work with operator-internal resources.
managedNamespaces = append(managedNamespaces, operatorNamespace)


// Add the empty namespace to allow watching cluster-scoped resources if storage class validation is enabled.
if viper.GetBool(operator.ValidateStorageClassFlag) {
    managedNamespaces = append(managedNamespaces, "")
}


opts.NewCache = cache.MultiNamespacedCacheBuilder(managedNamespaces)

The operator then watches all the resources on the cluster (even the ones which are not managed) while the multiNamespaceCache contains only cache entries for a limited set of namespaces:

Note the odd entry "" in the namespaceToCache map ^

[thbkrkr]

Relates to #3810 and #3868.

Note that kubernetes-sigs/controller-runtime#244 is solved thanks to kubernetes-sigs/controller-runtime#1435 and kubernetes-sigs/controller-runtime#1602.

[barkbay]

kubernetes-sigs/controller-runtime#1435 introduced Options.SelectorsByObject as a cache options, but cache.Options is hard coded:

// New constructs a brand new cluster.
func New(config *rest.Config, opts ...Option) (Cluster, error) {
...
	options := Options{}
	for _, opt := range opts {
		opt(&options)
	}
	options = setOptionsDefaults(options)
...

	// Create the cache for the cached read client and registering informers
	cache, err := options.NewCache(config, cache.Options{Scheme: options.Scheme, Mapper: mapper, Resync: options.SyncPeriod, Namespace: options.Namespace})
	if err != nil {
		return nil, err
	}

🤔

[naemono]

Closing as completed/resolved in #5187 and #6013