Stuck on "Provisioning MinIO Headless Service" #632
Comments
|
Forgot to mention the CSR exists and is approved: |
|
@djhoese can you share both the logs of the |
Operator LogAnd this continues. Tenant YAML |
|
Same problem here with minio-operator 4.0.10. |
Remove your existing CSRs or disable autoCert to deploy. This will generate new CSRs that would fix the operator-tls secret @puertal |
|
@harshavardhana Does that apply to both of us? I tried deleting the CSR, it was recreated, but no change on the operator-tls secret error. I tried editing the |
You generate |
|
Had a small hiccup, the Modified resource YAMLapiVersion: minio.min.io/v2
kind: Tenant
metadata:
creationTimestamp: null
name: minio-tenant-1
namespace: minio-tenant-1
scheduler:
name: ""
spec:
certConfig: {}
console:
consoleSecret:
name: minio-tenant-1-console-secret
image: minio/console:v0.6.8
replicas: 2
resources: {}
credsSecret:
name: minio-tenant-1-creds-secret
image: minio/minio:RELEASE.2021-04-06T23-11-00Z
imagePullSecret: {}
mountPath: /export
pools:
- affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: v1.min.io/tenant
operator: In
values:
- minio-tenant-1
topologyKey: kubernetes.io/hostname
resources: {}
servers: 3
volumeClaimTemplate:
apiVersion: v1
kind: persistentvolumeclaims
metadata:
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "1431655765"
storageClassName: local-path
status: {}
volumesPerServer: 4
requestAutoCert: false
status:
availableReplicas: 0
certificates: {}
currentState: ""
pools: null
revision: 0
syncVersion: ""
---
apiVersion: v1
data:
accesskey: ZjEyYmE5MDItMjMyMi00NTk5LThkZjgtMjJkNDYwMzJhZGE2
secretkey: NDQ1MjE3YjQtOGU2ZS00YWY1LWEzMDAtOTE1MDI1ZTRhMjZj
kind: Secret
metadata:
creationTimestamp: null
name: minio-tenant-1-creds-secret
namespace: minio-tenant-1
---
apiVersion: v1
data:
CONSOLE_ACCESS_KEY: YWRtaW4=
CONSOLE_PBKDF_PASSPHRASE: ZjhmYmVlNmUtZmE5YS00MjlkLWE0OTAtZDExMTAzMzI1ZDNh
CONSOLE_PBKDF_SALT: MDE4OTQ3YzMtNDQ2Zi00OGU5LWI0ZTEtZGRlOWFkZTQ5NDE4
CONSOLE_SECRET_KEY: NDQyMDIwMjQtMDkxMC00ZmVjLWFhM2QtMGMxNWY5YThmN2Yx
kind: Secret
metadata:
creationTimestamp: null
name: minio-tenant-1-console-secret
namespace: minio-tenant-1
Did |
This is because there is now operator-csr that is approved or not approved - which is necessary for operator-tls to be present. if the CSR is already approved then we need to wait create the secret associated with it. the CSR problem today is that once its approved - k8s doesn't give out new approvals so it has to be manually purged.
|
|
For example the correct things happening at the operator level Once certs are issued the operator-tls should be present And then you deploy your tenant And now this cert granting takes godforsaken amount of time on k8s :-) |
|
We've seen this on RKE because when you create the RKE cluster you need to configure the CSR feature https://rancher.com/docs/rke/latest/en/installation/certs/ The symptom is that the CSR shows the CSR approved but not issues |
|
Ah I didn't realize the CSR stuff was needed/expected before the tenant should be created. At least that takes the tenant out of the puzzle. However, in my last comment I mentioned that I had tried deleting the CSR and I didn't see a change. Hopefully I'm not missing something. Delete: Operator logs: And so on and on (put my kid to bed) and on (made a drink) and on.. Regarding your comment @dvaldivia, you're saying what I'm seeing is a symptom, meaning that deleting the csr should work as a workaround, right? If I need to configure my RKE cluster a certain way, how should it be done? |
|
I still have the same problem as @djhoese : "Start polling for certificate..." message forever and no "Starting API service" message. To give more info, I am deploying minio-operator using your Helm Chart package v4.0.10 downloaded from: https://github.com/minio/operator/tree/master/helm-releases And I configured it with: Checking the "operator-deployment.yaml" file inside the TGZ file I cannot see much other configuration options. If I delete the CSR, it is always created when Deployment is created, but the "operator-tls" Secret is never created. No Tenant defined at this point. |
|
Well it looks like I need to join this party as well.. The |
|
I've had some more time to play around with this and I confirm @dvaldivia's comment:
If you follow down the links in rancher/rancher#14041 you'll eventually end up at cockroachdb/cockroach#28075 (comment) which suggests: -- cluster.yaml file
services:
kube-controller:
extra_args:
cluster-signing-cert-file: "/etc/kubernetes/ssl/kube-ca.pem"
cluster-signing-key-file: "/etc/kubernetes/ssl/kube-ca-key.pem"Once you add the extra arguments to the #!/bin/bash
NAMESPACE=minio-operator (or whatever)
# Go to the Rancher UI, get to a project, copy the project id from the url:
# i.e. https://domain.com/p/a-bbbbbb:c-dddddd/workloads
RANCHER_PROJECT_ID=a-bbbbbb:c-dddddd
kubectl delete csr/operator-$NAMESPACE-csr
kubectl delete secrets $(kubectl get secrets --namespace=$NAMESPACE | grep -iE "minio|console" | awk '{print $1}' | xargs)
sleep 5
kubectl minio delete --namespace $NAMESPACE
kubectl minio init --image minio/operator:v4.0.11 --namespace $NAMESPACE
# Associate the namespace with a rancher project.
if [[ $NAMESPACE != "default" && $NAMESPACE != "kube-system" && $RANCHER_PROJECT_ID != "" ]]; then
kubectl annotate namespace $NAMESPACE field.cattle.io/projectId=$RANCHER_PROJECT_ID --overwrite
fi
kubectl minio tenant create $NAMESPACE --servers 4 --volumes 4 --capacity 10Gi --storage-class local-path --namespace $NAMESPACEI waited for a minute or so and all worked as expected. :) |
|
This needs to be reported upstream to rancher. Looks like quite a lot of other projects have been affected as well. |
Closing this as per this comment. |
|
@ravindk89 we may have to document this #632 (comment) |
|
Acknowledged - We can integrate this into the dedicated platform-based installation documentation. Though this is a longstanding issue, so I'll prioritize something for Rancher. |
|
@harshavardhana I agree - the RKE documentation is a bit confusing on this topic. You could have Rancher setup via a LB (i.e. Traefik) which generates valid ingress certificates. This might mislead you into thinking you don't need to do additional configuration parameters (which is what I mistakenly thought). Furthermore, these parameters aren't actually mentioned at all in he docs which is weird. I've opened a ticket there as well rancher/rke#2550 |
|
For those visiting this topic via github search, please see our docs for additional guidance. In my recent testing with the latest stable K3S, I did not encounter issues provisioning TLS certificates. This might vary depending on how you set up K3s or Rancher. |
I'm attempting to install MinIO Operator on a new test cluster at my work. The cluster is made by rancher (RKE) and has a default StorageClass using rancher's local-path-provisioner. After following the README (using the krew plugin) to install the operator and create a tenant the status of the tenant stays at "Provisioning MinIO Headless Service". There is a good chance I'm doing something wrong, but it isn't clear to me what. The web UI says "Unable to get tenant usage" when I go to the page for the tenant.
Expected Behavior
Create tenant, see resources being created in the tenant workspace (PV/PVCs), and see the status of the tenant in the MinIO Operator web UI.
Current Behavior
See above. Tenant stuck in "Provisioning MinIO Headless Service".
Additionally, checking the logs of the operator show:
Possible Solution
May be related to #483
Steps to Reproduce (for bugs)
kubectl minio initkubectl minio tenant create minio-tenant-1 --servers 3 --volumes 12 --capacity 16Gi --namespace minio-tenant-1 --storage-class local-pathContext
Note I have very little experience with MinIO. I used the old helm chart once and am trying to switch to the Operator now.
My work just started using rancher as an on-premises solution for creating k8s clusters. I'm helping our IT department out by testing some things on a "test" cluster (3 very small nodes) that I want to use in our operational clusters.
Side note: I'm very confused by the certificate/TLS stuff in https://github.com/minio/operator/blob/master/README.md#3-connect-to-the-tenant. Is that on the operator pod? Is that supposed to be on cluster nodes? When I tried to do this on the operator pod (kubectl exec) the destination directory doesn't exist and can't be created).
Your Environment
minio-operator): krew plugin says v4.0.9uname -a):The text was updated successfully, but these errors were encountered: