Pods stuck at ContainerCreating with MinIO Operator

[MikeLReminderM]

When I apply the yaml I got from running:

kubectl minio tenant create --name tenant1 --servers 4 --volumes 16 --capacity 16Ti -o > tenant.yaml

the pods are not getting past the ContainerCreating phase.

Expected Behavior

The pods should reach the Running status.

Current Behavior

The pods created do not get past ContainerCreating.

I see messages like the following in the minio-operator logs, these just keep repeating:

kubectl logs minio-operator-85ccdcfb6-r77qt

I1014 15:51:06.641634 1 main-controller.go:761] Successfully synced 'default/tenant-test1'
E1014 15:51:08.712430 1 main-controller.go:766] error syncing 'default/tenant-test1': Operation cannot be fulfilled on tenants.minio.min.io "tenant-test1": the object has been modified; please apply your changes to the latest version and try again
I1014 15:51:10.753121 1 main-controller.go:761] Successfully synced 'default/tenant-test1'
I1014 15:51:12.789315 1 main-controller.go:761] Successfully synced 'default/tenant-test1'

Possible Solution

Not sure. I was stuck at Pending earlier but creating the PVs got past that. I can see the volume claims have
all gone through successfully:

kubectl get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
0-tenant-test1-zone-0-0 Bound minio-tenant-2-3 1Gi RWO 24m
0-tenant-test1-zone-0-1 Bound minio-tenant-2-4 1Gi RWO 24m
1-tenant-test1-zone-0-0 Bound minio-tenant-1-3 1Gi RWO 24m
1-tenant-test1-zone-0-1 Bound minio-tenant-1-1 1Gi RWO 24m
2-tenant-test1-zone-0-0 Bound minio-tenant-2-2 1Gi RWO 24m
2-tenant-test1-zone-0-1 Bound minio-tenant-2-1 1Gi RWO 24m
3-tenant-test1-zone-0-0 Bound minio-tenant-1-4 1Gi RWO 24m
3-tenant-test1-zone-0-1 Bound minio-tenant-1-2 1Gi RWO 24m

Steps to Reproduce (for bugs)

1. installed and initialized the minio kubectl plugin per: https://github.com/minio/operator/blob/master/README.md

2. created 8 PVs (for the 2 servers that use 4 volumes each) using the CSI driver. A sample of the yaml I used for one:

apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-tenant-1-3
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
csi:
driver: direct.csi.min.io
volumeHandle: data
readOnly: false
fsType: ext4

3. generate yaml for a tenant with: kubectl minio tenant create --name tenant1 --servers 4 --volumes 16 --capacity 16Ti -o > tenant.yaml

Modified that yaml to set the size to 512Mi and the number of servers to 2. Changed the name to tenant-test1.

4. kubectl apply -f tenant.yaml

5. "kubectl get pods" shows the pods are stuck at ContainerCreating. Those pods have no logs.

kubectl get pods

NAME READY STATUS RESTARTS AGE
minio-operator-85ccdcfb6-r77qt 1/1 Running 1 22h
tenant-test1-zone-0-0 0/1 ContainerCreating 0 26m
tenant-test1-zone-0-1 0/1 ContainerCreating 0 26m

6. the minio operator logs is repeating lines like this:

I1014 15:51:06.641634 1 main-controller.go:761] Successfully synced 'default/tenant-test1'
E1014 15:51:08.712430 1 main-controller.go:766] error syncing 'default/tenant-test1': Operation cannot be fulfilled on tenants.minio.min.io "tenant-test1": the object has been modified; please apply your changes to the latest version and try again
I1014 15:51:10.753121 1 main-controller.go:761] Successfully synced 'default/tenant-test1'
I1014 15:51:12.789315 1 main-controller.go:761] Successfully synced 'default/tenant-test1'

Context

Trying to bring up a minio cluster in kubernetes cluster for use with staging and then production applications.

Regression

No.

Your Environment

• Version used (minio --version):
• Server setup and configuration:
• Operating System and version (uname -a):

from the yaml: image: minio/minio:RELEASE.2020-09-26T03-44-56Z

the setup is a 2 node test kubernetes cluster set up with kubeadm. it is on the latest stable version, v1.19.2.

the operating system is Ubuntu 18.04.5, running on VMs.

[harshavardhana]

you should describe the pod and see what is going on. Most probably you don't have enough resources on the system.

[MikeLReminderM]

you should describe the pod and see what is going on. Most probably you don't have enough resources on the system.

It looks like it is not mounting the PV - I see this from a describe pod:

Warning FailedMount 10m (x6 over 37m) kubelet Unable to attach or mount volumes: unmounted volumes=[0 1 2 3], unattached volumes=[tenant-test1-tls default-token-4jmf5 0 1 2 3]: timed out waiting for the condition
Warning FailedMount 47s (x34 over 55m) kubelet MountVolume.MountDevice failed for volume "minio-tenant-1-3" : rpc error: code = NotFound desc = volumes.direct.csi.min.io "data" not found

data is the volumeHandle I used in the yaml to create the PVs.

What should that be called for the tenant pods to be able to pick it up?

[harshavardhana]

Warning FailedMount 10m (x6 over 37m) kubelet Unable to attach or mount volumes: unmounted volumes=[0 1 2 3], unattached volumes=[tenant-test1-tls default-token-4jmf5 0 1 2 3]: timed out waiting for the condition
Warning FailedMount 47s (x34 over 55m) kubelet MountVolume.MountDevice failed for volume "minio-tenant-1-3" : rpc error: code = NotFound desc = volumes.direct.csi.min.io "data" not found

Well you don't have CSI installed so it won't work.

[MikeLReminderM]

I do have CSI installed:

direct-csi direct-csi-controller-min-io-cd598c4b-78xgn 2/2 Running 2 22h
direct-csi direct-csi-controller-min-io-cd598c4b-8cs4l 2/2 Running 3 22h
direct-csi direct-csi-controller-min-io-cd598c4b-jl8q4 2/2 Running 2 22h
direct-csi direct-csi-min-io-p7xdf 3/3 Running 3 22h
direct-csi direct-csi-min-io-rsqd8 3/3 Running 3 22h

This is from the shell history:

kubectl apply -k github.com/minio/direct-csi

Also:

kubectl get storageclass

NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
direct.csi.min.io direct.csi.min.io Delete Immediate false

[MikeLReminderM]

Look at the driver I used for the PV yaml files above. Those were all created successfully and the volume claims were successful as well.

apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-tenant-1-3
spec:
capacity:
storage: 1Gi
accessModes:

• ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
  driver: direct.csi.min.io
  volumeHandle: data
  readOnly: false
  fsType: ext4

[harshavardhana]

2. created 8 PVs (for the 2 servers that use 4 volumes each) using the CSI driver. A sample of the yaml I used for one:

apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-tenant-1-3
spec:
capacity:
storage: 1Gi
accessModes:

• ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
  driver: direct.csi.min.io
  volumeHandle: data
  readOnly: false
  fsType: ext4

you are not supposed to create PVs yourself @MikeLReminderM we do that automatically you are mistaken on how to set this up with DirectCSI - all you need is to simply just have directCSI running rest is done by the operator / kubectl plugin.

[MikeLReminderM]

Where is that documented?

Your docs list the following as prerequisites:

https://github.com/minio/operator/blob/master/README.md

Prerequisites
Kubernetes >= v1.17.0.
Create PVs.

When I didn't have the PVs created the pods were stuck in the Pending state. Nothing was set up by the operator.

I do have CSI running as my output above shows.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[nitisht]

Documentation is updated now https://github.com/minio/operator/blob/master/README.md#prerequisites . Is this still an issue @MikeLReminderM ?

[nitisht]

Closing this due to no response. Feel free to reopen.