operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)

[wangjunjack]

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

1. when i create Tenant that will be "error syncing 'minio-tenant-1/tenant1': secrets "operator-tls" not found"
2. how i do it?

Context

Regression

Your Environment

• Version used (minio-operator):
• Environment name and version (e.g. kubernetes v1.18.1

): * Server type and version: * Operating System and version (`uname -a`): * Link to your deployment file:

[dvaldivia]

@wangjunjack can you restart the Operator pod once and share the logs

[wangjunjack]

I0319 08:54:46.314441 1 main.go:72] Starting MinIO Operator

I0319 08:54:47.018108 1 main.go:139] caBundle on CRD updated

I0319 08:54:47.019008 1 main-controller.go:251] Setting up event handlers

I0319 08:54:47.019125 1 main-controller.go:635] Starting Tenant controller

I0319 08:54:47.019143 1 main-controller.go:638] Waiting for informer caches to sync

I0319 08:54:47.049571 1 main-controller.go:599] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)

I0319 08:54:47.096629 1 csr.go:217] Start polling for certificate of csr/operator-minio-operator-csr, every 5s, timeout after 20m0s

I0319 08:54:47.419474 1 main-controller.go:643] Starting workers

[wangjunjack]

@dvaldivia

[wangjunjack]

@dvaldivia when i Execute “kubectl minio init”, that will be "operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)"

[wangjunjack]

when i create Minio Tenant , it will be

`E0319 09:00:25.324901 1 main-controller.go:721] error syncing 'minio-tenant-1/tenant1': secrets "operator-tls" not found

E0319 09:00:35.226868 1 main-controller.go:721] error syncing 'minio-tenant-1/tenant1': secrets "operator-tls" not found`

the command is

kubectl minio tenant create tenant1 --namespace minio-tenant-1 --storage-class longhorn --servers 4 --volumes 4 --capacity 480Gi

[dvaldivia]

@wangjunjack are you able to see any CSR in your cluster from operator? if the secret is missing it means the Certificate Signing Request never succeeded and the TLS certificate for operator didn't install

[wangjunjack]

@dvaldivia yes, i can see it.

[wangjunjack]

@dvaldivia

[dvaldivia]

@wangjunjack and still the operator is not starting?

[wangjunjack]

@dvaldivia the operator is started, but the log of "operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)"

[dvaldivia]

@wangjunjack did you by any chance install operator to a custom namespace? What are the logs of the operator after restart? does it says soming like " polling for certificate" ?

[wangjunjack]

my step :

1. kubectl minio init
   I0322 05:15:10.756044 1 main.go:72] Starting MinIO Operator
   2021/3/22 下午1:15:11 I0322 05:15:11.396504 1 main.go:139] caBundle on CRD updated
   2021/3/22 下午1:15:11 I0322 05:15:11.397358 1 main-controller.go:251] Setting up event handlers
   2021/3/22 下午1:15:11 I0322 05:15:11.397465 1 main-controller.go:635] Starting Tenant controller
   2021/3/22 下午1:15:11 I0322 05:15:11.397487 1 main-controller.go:638] Waiting for informer caches to sync
   2021/3/22 下午1:15:11 I0322 05:15:11.426485 1 main-controller.go:599] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)
   2021/3/22 下午1:15:11 I0322 05:15:11.498451 1 main-controller.go:643] Starting workers
   2021/3/22 下午1:15:11 I0322 05:15:11.505655 1 csr.go:217] Start polling for certificate of csr/operator-minio-operator-csr, every 5s, timeout after 20m0s

the namepsace is default "minio-operator".

The log of "operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)", i don't know where is it, i check secrets there is not find any "operator-tls" secrets

[wangjunjack]

@dvaldivia
the start log
[root@ds-n1 ~]# kubectl logs minio-operator-9c79bb668-tlqz4 -n minio-operator I0322 05:15:10.756044 1 main.go:72] Starting MinIO Operator I0322 05:15:11.396504 1 main.go:139] caBundle on CRD updated I0322 05:15:11.397358 1 main-controller.go:251] Setting up event handlers I0322 05:15:11.397465 1 main-controller.go:635] Starting Tenant controller I0322 05:15:11.397487 1 main-controller.go:638] Waiting for informer caches to sync I0322 05:15:11.426485 1 main-controller.go:599] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found) I0322 05:15:11.498451 1 main-controller.go:643] Starting workers I0322 05:15:11.505655 1 csr.go:217] Start polling for certificate of csr/operator-minio-operator-csr, every 5s, timeout after 20m0s

[dvaldivia]

can you delete the CSR (certificate signging request) and reinstall operator this way

kubectl apply -k github.com/minio/operator/\?ref\=v4.0.3

[wangjunjack]

@dvaldivia i have sloved problem. thank you

[dvaldivia]

@wangjunjack how did you solved it? was it the reinstall? or a missing cluster role permission?

[wangjunjack]

you need add some config for kubernetes when you init kubernetes cluster. the config example:

[wangjunjack]

@dvaldivia thank you for your help

[Davidsv]

How did you add the config? I read here https://docs.min.io/minio/k8s/deployment/deploy-minio-operator.html#kubernetes-tls-certificate-api that I need to check kube-controller-manager, but there is nothing like that on my cluster. EDIT: ok now I understood that kube-controller-manager runs on the etcd nodes and not the cluster per se.

[ctml91]

I am facing this same issue with this operator deployed via OLM on OpenShift, the proposed solution doesn't sound applicable.

Should be noted on OpenShift when you deploy the operator via OLM for "all namespaces", it installs the operator into the openshift-operators namespace. The first error is a missing secret, the second is failure to create csr. OpenShift might implement the certificate signing process for operators differently, though it is just a hunch.

I0309 18:28:03.059435       1 main.go:74] Starting MinIO Operator
I0309 18:28:03.424865       1 main.go:146] caBundle on CRD updated
I0309 18:28:03.425287       1 main-controller.go:272] Setting up event handlers
I0309 18:28:03.425349       1 main-controller.go:656] Starting Tenant controller
I0309 18:28:03.425353       1 main-controller.go:659] Waiting for informer caches to sync
I0309 18:28:03.500277       1 main-controller.go:620] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)
E0309 18:28:03.558448       1 operator.go:104] Unexpected error during the creation of the csr/operator-openshift-operators-csr: the server could not find the requested resource
I0309 18:28:03.558465       1 main-controller.go:623] Waiting for the operator certificates to be issued the server could not find the requested resource
I0309 18:28:03.825603       1 main-controller.go:664] Starting workers
I0309 18:28:08.699035       1 main-controller.go:737] Successfully synced 'openshift-migration/minio'
I0309 18:28:13.563961       1 main-controller.go:620] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)
E0309 18:28:13.566544       1 operator.go:104] Unexpected error during the creation of the csr/operator-openshift-operators-csr: the server could not find the requeste

Issue 2:
I also tried deploying in a single namespace, but the operator doesn't seem to correctly use the WATCH_NAMESPACE environment variable and still attempts to list resources at the cluster scope which results in RBAC errors.

https://sdk.operatorframework.io/docs/building-operators/golang/operator-scope/#configuring-watch-namespaces-dynamically

E0309 18:24:44.146930       1 main.go:138] Error getting CRD for adding caBundle: customresourcedefinitions.apiextensions.k8s.io "tenants.minio.min.io" is forbidden: User "system:serviceaccount:test:minio-operator" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
I0309 18:24:44.147390       1 main-controller.go:272] Setting up event handlers
I0309 18:24:44.147444       1 main-controller.go:656] Starting Tenant controller
I0309 18:24:44.147449       1 main-controller.go:659] Waiting for informer caches to sync
E0309 18:24:44.148817       1 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:serviceaccount:test:minio-operator" cannot list resource "services" in API group "" at the cluster scope
E0309 18:24:44.149791       1 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:test:minio-operator" cannot list resource "deployments" in API group "apps" at the cluster scope
E0309 18:24:44.149796       1 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:test:minio-operator" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0309 18:24:44.150581       1 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v2.Tenant: failed to list *v2.Tenant: tenants.minio.min.io is forbidden: User "system:serviceaccount:test:minio-operator" cannot list resource "tenants" in API group "minio.min.io" at the cluster scope
E0309 18:24:44.150689       1 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:test:minio-operator" cannot list resource "jobs" in API group "batch" at the cluster scope
I0309 18:24:44.208747       1 main-controller.go:620] operator TLS secret not found%!(EXTRA string=secrets "operator-tls" not found)

[dvaldivia]

@ctml91 do you see any logs showing an error writing a secret to the tenant namespace? What version of the operator do you have?

[ctml91]

@ctml91 do you see any logs showing an error writing a secret to the tenant namespace? What version of the operator do you have?

Thanks for the lightning quick reply. Updated original comment with specific error, we installed 4.0.9.

[dvaldivia]

@ctml91 that's a little bit old, can you try the 4.4.9re elase from oeprator hub? https://operatorhub.io/operator/minio-operator

[ctml91]

@ctml91 that's a little bit old, can you try the 4.4.9re elase from oeprator hub? https://operatorhub.io/operator/minio-operator

This is the latest available from OpenShifts integrated operator hub. Is it possible to publish the updated versions to their repo?

https://github.com/redhat-openshift-ecosystem/certified-operators