Minio on Kubernetes requires service names with full cluster domain

[jbw976]

From attempting to fix rook/rook#2258, it appears that Minio requires service names for the distributed server list that include the full cluster domain. In some environments, this cluster domain may have been changed from the default cluster.local, but a user will not know the customized value. There are 2 key elements to reproducing this issue:

1. The server address list does not contain the cluster.local cluster domain
2. The StatefulSet uses a name other than the default of minio.

Expected Behavior

It is expected that the Minio server will be able to determine a given Kubernetes service address is a local address without requiring the full cluster domain. For example:

• http://my-store-{0...3}.minio.default.svc/data should work, but does not
• http://my-store-{0...3}.minio.default.svc.cluster.local/data does work, note the required service DNS name that includes the full cluster.local domain

Current Behavior

If the full service name including cluster domain is not included, the following error is returned from the minio process:

ERROR Invalid command line arguments: no endpoint pointing to the local machine is found.
      > Please provide correct combination of local/remote paths.
      HELP:
        For more information, please refer to https://docs.minio.io/docs/minio-erasure-code-quickstart-guide

Possible Solution

I'm not quite sure what's going on with DNS/lookups that is making Minio server think an endpoint isn't pointing to the local machine, so I don't have a suggestion unfortunately.

Steps to Reproduce (for bugs)

1. Generate distributed mode config from https://www.minio.io/kubernetes.html
2. Remove cluster domain cluster.local from the generated config
3. Change name of Minio statefulset to another value like my-store
4. Change host name in server address arg to also match the stateful set name my-store, e.g.: http://my-store-{0...3}.minio.default.svc/data, in accordance with StatefulSet stable network IDs
5. kubectl create -f https://gist.githubusercontent.com/jbw976/eba906f6645712605ec196d3bbf1e5ef/raw/39e62dd6a7bbbecb8ea8108b60f48f4c355a493b/minio-deployment.yaml

Note the example deployment that reproduces this issue: https://gist.githubusercontent.com/jbw976/eba906f6645712605ec196d3bbf1e5ef/raw/39e62dd6a7bbbecb8ea8108b60f48f4c355a493b/minio-deployment.yaml

Regression

Not sure

Your Environment

• Version used (minio version):

Version: 2018-11-06T01:01:02Z
Release-Tag: RELEASE.2018-11-06T01-01-02Z
Commit-ID: eb1f9c9916cdd2d1031b76dfc4e539fd643de6bc

• Environment name and version (e.g. nginx 1.9.1): Kubernetes v1.12.2 on Minikube v0.30.0
• Operating System and version (uname -a):

Linux minikube 4.15.0 #1 SMP Fri Oct 5 20:44:14 UTC 2018 x86_64 GNU/Linux

• Link to your project:
  https://github.com/rook/rook
  https://rook.io/docs/rook/master/minio-object-store.html
  https://rook.io/docs/rook/master/minio-object-store-crd.html

[harshavardhana]

This is expected and is well documented https://docs.minio.io/docs/deploy-minio-on-kubernetes.html - there is nothing we are doing to handle this.

IF the DNS resolution works properly with default.svc it would work automatically.

[jbw976]

This issue does not reproduce when cluster.local is removed from the server address args, but the statefulSet name of minio is kept. cluster.local is not needed in that case, everything works OK. Could there be some hard-coded assumption in the minio code base that the statefulSet (or pod name) is always going to be minio?

This yaml file works for me (note it is not using cluster.local): https://gist.githubusercontent.com/jbw976/b3b1beccb530ccfc452c7c3267793076/raw/3839fa99f874ba4f5e49804009041f4a08df46ae/minio-deployment.yaml

[harshavardhana]

There is no such assumption in minio it is perhaps how the pod is deployed in kubernetes. For minio these are just DNS names.

[nitisht]

@jbw976 I tried the below yaml and it works on local Minikube setup

apiVersion: v1
kind: Service
metadata:
  name: minio
  labels:
    app: minio
spec:
  clusterIP: None
  ports:
    - port: 9000
      name: minio
  selector:
    app: minio
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: my-store
spec:
  serviceName: minio
  replicas: 4
  template:
    metadata:
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        env:
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        image: minio/minio
        args:
        - server
        - http://my-store-{0...3}.minio.default.svc/data
        ports:
        - containerPort: 9000
          # This ensures containers are allocated on separate hosts. Remove hostPort to allow multiple Minio containers on one host
        # These volume mounts are persistent. Each pod in the PetSet
        # gets a volume mounted based on this field.
        volumeMounts:
        - name: data
          mountPath: /data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      # Uncomment and add storageClass specific to your requirements below. Read more https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1
      #storageClassName:
---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
spec:
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

[nitisht]

@jbw976 closing this for now as this seems to be working, feel free to reopen/get back to us if there are any further issues

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.