Minio Console not creating for a new Tenant

[djciaro]

Hello,

I am running a POC for all things K8s and one of the items is S3 storage.
We run our infra on VMware VCF.

I setup MinIO and created the first tenant based on the documented procedure: https://docs.min.io/minio/vsphere/tutorials/deploy-minio-tenant.html#monitor-tenant-creation.

The tenant is stuck at Current State: Waiting for Pods to be ready

The MinIO endpoint is created and ready, I have access to it and can create buckets etc, but no sign of the Console.

I used:
minio-release:20200508
• minio/k8s-operator:v3.0.28
• minio/console:v0.4.3
• minio/vsphere:v1.0.5

vCenter is 7.0U2 Build 18455184

kubectl get pods -n minio-domain-c1004
NAME READY STATUS RESTARTS AGE
masterproxy-minio-b6plt 1/1 Running 0 2d5h
masterproxy-minio-f5759 1/1 Running 0 2d5h
masterproxy-minio-lmqtr 1/1 Running 0 2d5h
minio-minio-operator-5db6ff9cb6-dqp9d 1/1 Running 0 2d5h
minio-vsphere-855b8d4d68-9s6wt 2/2 Running 0 2d5h
minio-vsphere-855b8d4d68-lwwnl 2/2 Running 0 2d5h
minio-vsphere-855b8d4d68-rdhp2 2/2 Running 0 2d5h
minio-vsphere-controller-86c449bfc-jlt6q 1/1 Running 0 2d5h

kubectl get pods -n minio
NAME READY STATUS RESTARTS AGE
s3-storage-zone-0-0 1/1 Running 0 2d
s3-storage-zone-0-1 1/1 Running 0 2d
s3-storage-zone-0-2 1/1 Running 0 2d
s3-storage-zone-0-3 1/1 Running 0 2d

I need some help on how to check the console and what is blocking it from coming up.

Regards
Ciarán

[harshavardhana]

@dvaldivia ^^

[dvaldivia]

Can you describe one of the pods and post the events? That will tell us why the pods are not scheduling, perhaps they have unfulfilled PVCs, so also attach the list of PVCs for this tenant and a the events of a describe for one of them please.

[djciaro]

Hello,

Below you can see one of the 4 storage pods - this a 4 node VxRail cluster.
I redacted some entries - where it says REDACTED there was a valid entry.
Below that are the events for the same pod

PVC look good I have 64 in bound state, the error at end suggests insufficient resources but the cluster has plenty and NS is unlimited.

kubectl describe pod dwh-s3-storage-zone-0-0 -n minio
Name: dwh-s3-storage-zone-0-0
Namespace: minio
Priority: 0
Node: REDACTED
Start Time: Tue, 16 Nov 2021 15:21:40 +0000
Labels: appplatform.vmware.com/extension-id=io.min.vsphere
appplatform.vmware.com/instance-id=dwh-s3-storage__minio
controller-revision-hash=dwh-s3-storage-zone-0-7d9487995
statefulset.kubernetes.io/pod-name=dwh-s3-storage-zone-0-0
v1.min.io/tenant=dwh-s3-storage
v1.min.io/version=v3.0.28
v1.min.io/zone=zone-0
Annotations: appplatform.vmware.com/vsphere-emm-integrated: true
attachment_id: 79be87ba-800e-401d-9cc3-d61e32f0aabc
kubernetes.io/psp: wcp-default-psp
mac: 04:50:56:00:90:17
vlan: None
vmware-system-ephemeral-disk-uuid: 6000C29a-17cd-69c9-5655-de133a2de179
vmware-system-image-references: {"minio":"minio-release-3a42526071500b0944ecfee024a5d2613e770356-v52242"}
vmware-system-vm-moid: vm-5416:874ac13d-8a35-4818-bbf3-c0d388a67b50
vmware-system-vm-uuid: 50168263-6fae-129d-4358-8d169273dd74
Status: Running
IP: REDACTED
IPs:
IP: REDACTED
Controlled By: StatefulSet/dwh-s3-storage-zone-0
Containers:
minio:
Container ID: 441995a8-7d52-467f-81ff-a7608cea6e3c
Image: REDACTED/minio/minio/minio-release:20200508
Image ID: minio-release-3a42526071500b0944ecfee024a5d2613e770356-v52242
Port: 9000/TCP
Host Port: 0/TCP
Args:
server
--certs-dir
/tmp/certs
State: Running
Started: Tue, 16 Nov 2021 15:21:42 +0000
Ready: True
Restart Count: 0
Limits:
memory: 16Gi
Requests:
memory: 16Gi
Environment:
MINIO_STORAGE_CLASS_STANDARD: EC:2
MINIO_UPDATE: on
MINIO_UPDATE_MINISIGN_PUBKEY: REDACTED
MINIO_ARGS: <set to the key 'MINIO_ARGS' in secret 'operator-webhook-secret'> Optional: false
MINIO_ENDPOINTS: https://dwh-s3-storage-zone-0-{0...3}.dwh-s3-storage-hl.minio.svc.cluster.local/export{0...15}
MINIO_OPERATOR_VERSION: v3.0.28
MINIO_ACCESS_KEY: <set to the key 'accesskey' in secret 'dwh-s3-storage-secret'> Optional: false
MINIO_SECRET_KEY: <set to the key 'secretkey' in secret 'dwh-s3-storage-secret'> Optional: false
Mounts:
/export0 from data0 (rw)
/export1 from data1 (rw)
/export10 from data10 (rw)
/export11 from data11 (rw)
/export12 from data12 (rw)
/export13 from data13 (rw)
/export14 from data14 (rw)
/export15 from data15 (rw)
/export2 from data2 (rw)
/export3 from data3 (rw)
/export4 from data4 (rw)
/export5 from data5 (rw)
/export6 from data6 (rw)
/export7 from data7 (rw)
/export8 from data8 (rw)
/export9 from data9 (rw)
/tmp/certs from dwh-s3-storage-tls (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-444m7 (ro)
Conditions:
Type Status
PodScheduled True
Initialized True
ContainersReady True
Ready True
Volumes:
data7:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data7-dwh-s3-storage-zone-0-0
ReadOnly: false
data10:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data10-dwh-s3-storage-zone-0-0
ReadOnly: false
data2:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data2-dwh-s3-storage-zone-0-0
ReadOnly: false
data3:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data3-dwh-s3-storage-zone-0-0
ReadOnly: false
data11:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data11-dwh-s3-storage-zone-0-0
ReadOnly: false
data14:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data14-dwh-s3-storage-zone-0-0
ReadOnly: false
data0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data0-dwh-s3-storage-zone-0-0
ReadOnly: false
data4:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data4-dwh-s3-storage-zone-0-0
ReadOnly: false
data8:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data8-dwh-s3-storage-zone-0-0
ReadOnly: false
data5:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data5-dwh-s3-storage-zone-0-0
ReadOnly: false
data6:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data6-dwh-s3-storage-zone-0-0
ReadOnly: false
data12:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data12-dwh-s3-storage-zone-0-0
ReadOnly: false
data13:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data13-dwh-s3-storage-zone-0-0
ReadOnly: false
data15:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data15-dwh-s3-storage-zone-0-0
ReadOnly: false
data1:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data1-dwh-s3-storage-zone-0-0
ReadOnly: false
data9:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data9-dwh-s3-storage-zone-0-0
ReadOnly: false
dwh-s3-storage-tls:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: dwh-s3-storage-tls
SecretOptionalName:
default-token-444m7:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-444m7
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

---

API: SYSTEM()
Time: 18:29:53 UTC 11/18/2021
DeploymentID: 488312ad-62e8-40da-8c2f-da2d66c05923
Error: Storage resources are insufficient for the read operation.
8: github.com/minio/minio@/cmd/config-common.go:40:cmd.readConfig()
7: github.com/minio/minio@/cmd/config-migrate.go:2504:cmd.checkConfigVersion()
6: github.com/minio/minio@/cmd/config-migrate.go:2477:cmd.migrateMinioSysConfig()
5: github.com/minio/minio@/cmd/config.go:241:cmd.initConfig()
4: github.com/minio/minio@/cmd/config.go:211:cmd.(*ConfigSys).Init()
3: github.com/minio/minio@/cmd/server-main.go:305:cmd.initAllSubsystems()
2: github.com/minio/minio@/cmd/server-main.go:230:cmd.initSafeMode()
1: github.com/minio/minio@/cmd/server-main.go:521:cmd.serverMain()
Waiting for all MinIO sub-systems to be initialized.. possible cause (Unable to initialize config system: Storage resources are insufficient for the read operation.)
Waiting for all MinIO sub-systems to be initialized.. lock acquired

[harshavardhana]

API: SYSTEM()
Time: 18:29:53 UTC 11/18/2021
DeploymentID: 488312ad-62e8-40da-8c2f-da2d66c05923
Error: Storage resources are insufficient for the read operation.
8: github.com/minio/minio@/cmd/config-common.go:40:cmd.readConfig()
7: github.com/minio/minio@/cmd/config-migrate.go:2504:cmd.checkConfigVersion()
6: github.com/minio/minio@/cmd/config-migrate.go:2477:cmd.migrateMinioSysConfig()
5: github.com/minio/minio@/cmd/config.go:241:cmd.initConfig()
4: github.com/minio/minio@/cmd/config.go:211:cmd.(*ConfigSys).Init()
3: github.com/minio/minio@/cmd/server-main.go:305:cmd.initAllSubsystems()
2: github.com/minio/minio@/cmd/server-main.go:230:cmd.initSafeMode()
1: github.com/minio/minio@/cmd/server-main.go:521:cmd.serverMain()
Waiting for all MinIO sub-systems to be initialized.. possible cause (Unable to initialize config system: Storage resources are insufficient for the read operation.)

Most probably MinIO has no read permissions on the drives

[djciaro]

Hi Harshavardhana,

If there were no permissions on the drives then surely I would not be able to use buckets, write, read etc.

I can create buckets, upload file, download, delete etc. via the MinIO browser.

I deleted the tenant and started a new one.

For some reason the stateful pod does not append the correct path to the internal harbor repository and the Pods remain stuck trying to pull an image with incorrect path.

I am able to correct this (note I remove IPs in this post)
kubectl edit statefulset dwh-s3-storagepoc-zone-0 -n minio
I add the IP of harbor.
delete the pod and when it recreates it finds the path to Harbor

Controlled By: StatefulSet/dwh-s3-storagepoc-zone-0
Containers:
minio:
Container ID: 9907bf0-a139-4bd3-8c7e-59f76070a8dd
Image: HARBORIP/minio/minio/minio-release:20200508
Image ID: minio-release-3a42526071500b0944ecfee024a5d2613e770356-v10630
Port: 9000/TCP
Host Port: 0/TCP

all 4 pods then get to running state:

NAME READY STATUS RESTARTS AGE
dwh-s3-storagepoc-zone-0-0 1/1 Running 0 13m
dwh-s3-storagepoc-zone-0-1 1/1 Running 0 14m
dwh-s3-storagepoc-zone-0-2 1/1 Running 0 15m
dwh-s3-storagepoc-zone-0-3 1/1 Running 0 56m

but the MinIO tenant setup remains with the same step:

I believe just like for the release the system does not have the correct path to the harbor registry. Where can I check the path it is trying to use to get the console image from?

[dvaldivia]

I think what's happening is you have a really old version of MinIO and modern releases have the console built it, so operator doesn't create the separate console deployment anymore, only the service

[djciaro]

I got this working with the following combination of releases:

minio:release.2020-10-18t21-54-12
• minio/k8s-operator:v3.0.28
• minio/console:v0.4.3
• minio/vsphere:v1.0.5
vCenter 7.0U2 Build 18455184