Operator chart multi-tenancy does not work with existing namespaces

[michaellzc]

Bug Report

What did you do? Trying to install ECK Operator with softMultiTenancy enabled

What did you expect to see? It works

What did you see instead? Under which circumstances?

We try to deploy the eck-operator helm chart with the following values.

managedNamespaces: ["team_1", "team_2"]

kubeAPIServerIP: "api_server_url"

createClusterScopedResources: true

refs:
  enforceRBAC: true

webhook:
  enabled: true

softMultiTenancy:
  enabled: true

Assuming team_1 and team_2 are two existing namespaces, the helm install will fail with

Error: rendered manifests contain a resource that already exists. Unable to continue with install: Namespace "team_1r" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "eck-operator"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "elastic-system"
  helm.go:81: [debug] Namespace "team_1" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "eck-operator"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "elastic-system"
  rendered manifests contain a resource that already exists. Unable to continue with install

It is most likely caused by https://github.com/elastic/cloud-on-k8s/blob/master/deploy/eck-operator/templates/managed-namespaces.yaml where it is trying to create namespaces that already exist.

Moreover, according to helm/helm#3503, I don't think it does everything that you intended to do. Yes, the namespace will be created, but metadata will not be added. So stuff like eck.k8s.elastic.co/tenant: {{ $namespace }} and {{- include "eck-operator.labels" $ | nindent 4 }} are not added.

I don't think helm does a good job at handling inter-namespace resources. IMO, the better approach is to leave the namespace creating and labelling to end-users. The instruction can be added to the post-installation NOTES.txt

Environment

• ECK version:

  Not applicable. ECK Operator chart cannot be installed.

• Kubernetes information:

  insert any information about your Kubernetes environment that could help us:

  • On premise: No
  • Cloud: IBM Cloud
  • Kubernetes distribution: IBM Cloud-managed

  for each of them please give us the version you are using

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.15+IKS", GitCommit:"6aaff2c2e616933799bce6aea81abb00454376de", GitTreeState:"clean", BuildDate:"2021-01-15T04:16:00Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Resource definition:

if relevant insert the resource definition

• Logs:

insert operator logs or any relevant message to the issue here

[charith-elastic]

Moreover, according to helm/helm#3503, I don't think it does everything that you intended to do. Yes, the namespace will be created, but metadata will not be added. So stuff like eck.k8s.elastic.co/tenant: {{ $namespace }} and {{- include "eck-operator.labels" $ | nindent 4 }} are not added.

That Helm issue is about the installation namespace -- which is special. Charts can create other namespaces just like any other resource.

The soft-multi-tenancy profile is an opinionated example of how the operator can be installed to support a particular form of soft-multi-tenancy. It expects exclusive access to the managed namespaces because it creates network policies that block all other traffic -- which makes it difficult for other non-Elastic applications to be deployed to them anyway.

If you do want to use existing empty namespaces, you can "adopt" them through labelling. However, I'd advise you to test the whole multi-tenancy profile in a non-production cluster to make sure that it matches your requirements. It is not a general-purpose solution that could be adapted to all environments.

[michaellzc]

Thanks for the clarification.

It would have been nice to document it somewhere about such behaviour, explaining that this is an opinionated approach and the operator expects exclusive access to managed namespaces.

Not everyone is running a K8S cluster exclusively for a bunch of ES instances. One might want to deploy an ES instance to an existing namespace that already has other resources.