add init container for sanity checks #255

kerneltime · 2020-08-13T09:11:29Z

During testing, we suspect an edge case where the volume mount might not be ready before minio starts up. Adding these init containers to ensure that volumes and DNS names are ready.

pkg/resources/statefulsets/minio-statefulset.go

nitisht

This somehow seems to interfere with args for MinIO Server. MinIO pods are not starting after initcontainers are done.

$ kubectl get pods
NAME             READY   STATUS             RESTARTS   AGE
minio-zone-0-0   0/1     CrashLoopBackOff   7          14m
minio-zone-0-1   0/1     CrashLoopBackOff   7          14m
minio-zone-0-2   0/1     CrashLoopBackOff   7          14m
minio-zone-0-3   0/1     CrashLoopBackOff   7          14m

$ kubectl logs minio-zone-0-0
NAME:
  minio server - start object storage server

USAGE:
  minio server [FLAGS] DIR1 [DIR2..]
  minio server [FLAGS] DIR{1...64}
  minio server [FLAGS] DIR{1...64} DIR{65...128}

DIR:
  DIR points to a directory on a filesystem. When you want to combine
  multiple drives into a single large system, pass one directory per
  filesystem separated by space. You may also use a '...' convention
  to abbreviate the directory arguments. Remote directories in a
  distributed setup are encoded as HTTP(s) URIs.

FLAGS:
  --address value              bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname (default: ":9000")
  --certs-dir value, -S value  path to certs directory (default: "/root/.minio/certs")
  --quiet                      disable startup information
  --anonymous                  hide sensitive information from logging
  --json                       output server logs and startup information in json format
  --help, -h                   show help
  
EXAMPLES:
  1. Start minio server on "/home/shared" directory.
     $ minio server /home/shared

  2. Start single node server with 64 local drives "/mnt/data1" to "/mnt/data64".
     $ minio server /mnt/data{1...64}

  3. Start distributed minio server on an 32 node setup with 32 drives each, run following command on all the nodes
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...32}.example.com/mnt/export{1...32}

  4. Start distributed minio server in an expanded setup, run the following command on all the nodes
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...16}.example.com/mnt/export{1...32} \
            http://node{17...64}.example.com/mnt/export{1...64}

kerneltime · 2020-08-13T15:36:45Z

Which deployment did you use?

This somehow seems to interfere with args for MinIO Server. MinIO pods are not starting after initcontainers are done.

$ kubectl get pods
NAME             READY   STATUS             RESTARTS   AGE
minio-zone-0-0   0/1     CrashLoopBackOff   7          14m
minio-zone-0-1   0/1     CrashLoopBackOff   7          14m
minio-zone-0-2   0/1     CrashLoopBackOff   7          14m
minio-zone-0-3   0/1     CrashLoopBackOff   7          14m

$ kubectl logs minio-zone-0-0
NAME:
  minio server - start object storage server

USAGE:
  minio server [FLAGS] DIR1 [DIR2..]
  minio server [FLAGS] DIR{1...64}
  minio server [FLAGS] DIR{1...64} DIR{65...128}

DIR:
  DIR points to a directory on a filesystem. When you want to combine
  multiple drives into a single large system, pass one directory per
  filesystem separated by space. You may also use a '...' convention
  to abbreviate the directory arguments. Remote directories in a
  distributed setup are encoded as HTTP(s) URIs.

FLAGS:
  --address value              bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname (default: ":9000")
  --certs-dir value, -S value  path to certs directory (default: "/root/.minio/certs")
  --quiet                      disable startup information
  --anonymous                  hide sensitive information from logging
  --json                       output server logs and startup information in json format
  --help, -h                   show help
  
EXAMPLES:
  1. Start minio server on "/home/shared" directory.
     $ minio server /home/shared

  2. Start single node server with 64 local drives "/mnt/data1" to "/mnt/data64".
     $ minio server /mnt/data{1...64}

  3. Start distributed minio server on an 32 node setup with 32 drives each, run following command on all the nodes
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...32}.example.com/mnt/export{1...32}

  4. Start distributed minio server in an expanded setup, run the following command on all the nodes
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...16}.example.com/mnt/export{1...32} \
            http://node{17...64}.example.com/mnt/export{1...64}

pkg/resources/statefulsets/minio-statefulset.go

pkg/apis/minio.min.io/v1/names.go

harshavardhana · 2020-08-13T22:07:57Z

This somehow seems to interfere with args for MinIO Server. MinIO pods are not starting after initcontainers are done.

$ kubectl get pods
NAME             READY   STATUS             RESTARTS   AGE
minio-zone-0-0   0/1     CrashLoopBackOff   7          14m
minio-zone-0-1   0/1     CrashLoopBackOff   7          14m
minio-zone-0-2   0/1     CrashLoopBackOff   7          14m
minio-zone-0-3   0/1     CrashLoopBackOff   7          14m

$ kubectl logs minio-zone-0-0
NAME:
  minio server - start object storage server

USAGE:
  minio server [FLAGS] DIR1 [DIR2..]
  minio server [FLAGS] DIR{1...64}
  minio server [FLAGS] DIR{1...64} DIR{65...128}

DIR:
  DIR points to a directory on a filesystem. When you want to combine
  multiple drives into a single large system, pass one directory per
  filesystem separated by space. You may also use a '...' convention
  to abbreviate the directory arguments. Remote directories in a
  distributed setup are encoded as HTTP(s) URIs.

FLAGS:
  --address value              bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname (default: ":9000")
  --certs-dir value, -S value  path to certs directory (default: "/root/.minio/certs")
  --quiet                      disable startup information
  --anonymous                  hide sensitive information from logging
  --json                       output server logs and startup information in json format
  --help, -h                   show help
  
EXAMPLES:
  1. Start minio server on "/home/shared" directory.
     $ minio server /home/shared

  2. Start single node server with 64 local drives "/mnt/data1" to "/mnt/data64".
     $ minio server /mnt/data{1...64}

  3. Start distributed minio server on an 32 node setup with 32 drives each, run following command on all the nodes
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...32}.example.com/mnt/export{1...32}

  4. Start distributed minio server in an expanded setup, run the following command on all the nodes
     $ export MINIO_ACCESS_KEY=minio
     $ export MINIO_SECRET_KEY=miniostorage
     $ minio server http://node{1...16}.example.com/mnt/export{1...32} \
            http://node{17...64}.example.com/mnt/export{1...64}

I think this must be because of MinIO version you used in your tenant.yaml

nitisht · 2020-08-14T04:56:04Z

I think this must be because of MinIO version you used in your tenant.yaml

Yes, great catch. Let me try with edge

harshavardhana · 2020-08-14T05:02:34Z

I think this must be because of MinIO version you used in your tenant.yaml

Yes, great catch. Let me try with edge

No just latest release is fine the tenant.yaml was not updated which is now..

nitisht

LGTM & Tested. Works fine. Sorry for the confusion

Signed-off-by: Ritesh H Shukla <ritesh@minio.io>

harshavardhana · 2020-08-18T20:31:42Z

PTAL @nitisht again

kerneltime requested review from harshavardhana, nitisht and dvaldivia August 13, 2020 09:11

nitisht reviewed Aug 13, 2020

View reviewed changes

pkg/resources/statefulsets/minio-statefulset.go Show resolved Hide resolved

nitisht suggested changes Aug 13, 2020

View reviewed changes

kerneltime force-pushed the statefulsets-init branch 3 times, most recently from d2cb981 to 227f028 Compare August 13, 2020 18:06

harshavardhana requested changes Aug 13, 2020

View reviewed changes

pkg/resources/statefulsets/minio-statefulset.go Show resolved Hide resolved

pkg/apis/minio.min.io/v1/names.go Outdated Show resolved Hide resolved

kerneltime force-pushed the statefulsets-init branch from 227f028 to 4359b84 Compare August 13, 2020 22:28

nitisht previously approved these changes Aug 14, 2020

View reviewed changes

kerneltime dismissed nitisht’s stale review via c7bc8b9 August 17, 2020 21:30

kerneltime force-pushed the statefulsets-init branch from 024e1ae to c7bc8b9 Compare August 17, 2020 21:30

Add init containers to wait for DNS and volumes

b201bcb

Signed-off-by: Ritesh H Shukla <ritesh@minio.io>

kerneltime force-pushed the statefulsets-init branch from c7bc8b9 to b201bcb Compare August 17, 2020 21:32

harshavardhana changed the title ~~Create init~~ add init container for sanity checks Aug 18, 2020

harshavardhana approved these changes Aug 18, 2020

View reviewed changes

nitisht approved these changes Aug 20, 2020

View reviewed changes

nitisht merged commit 3c1908d into minio:master Aug 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add init container for sanity checks #255

add init container for sanity checks #255

kerneltime commented Aug 13, 2020 •

edited

nitisht left a comment •

edited

kerneltime commented Aug 13, 2020

harshavardhana commented Aug 13, 2020

nitisht commented Aug 14, 2020

harshavardhana commented Aug 14, 2020

nitisht left a comment

harshavardhana commented Aug 18, 2020

add init container for sanity checks #255

add init container for sanity checks #255

Conversation

kerneltime commented Aug 13, 2020 • edited

nitisht left a comment • edited

Choose a reason for hiding this comment

kerneltime commented Aug 13, 2020

harshavardhana commented Aug 13, 2020

nitisht commented Aug 14, 2020

harshavardhana commented Aug 14, 2020

nitisht left a comment

Choose a reason for hiding this comment

harshavardhana commented Aug 18, 2020

kerneltime commented Aug 13, 2020 •

edited

nitisht left a comment •

edited