large bucket support

Command line

NAME:
  minio server - Start object storage server.

USAGE:
  minio server [FLAGS] DIR1 [DIR2..]
  minio server [FLAGS] DIR{1...64}

DIR:
  DIR points to a directory on a filesystem. When you want to combine multiple drives
  into a single large system, pass one directory per filesystem separated by space.
  You may also use a `...` convention to abbreviate the directory arguments. Remote
  directories in a distributed setup are encoded as HTTP(s) URIs.
 
FLAGS:
  --address value               Bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname. (default: ":9000")
  --config-dir value, -C value  Path to configuration directory. (default: "/home/harsha/.minio")
  --quiet                       Disable startup information.
  --help, -h                    Show help.

Most common

minio server dir1 - good
minio server dir1 dir2 dir3 dir4 - good
minio server dir{1...64} - good
minio server http://host{1...64}/export{1...64} - good

Other advanced usages

Distributed setup host, export variations

minio server http://host{1...4}/export{1...8} - good

Change default erasure data/parity count

export MINIO_ERASURE=8x4
minio server dir{1...64} - good (Means 8 data, 4 parity split)

Similarity distributed

minio server http://host{1...64}/export{1...64} - good (same as {1...32} {33...64})
minio server http://host{1...32}/export{1...32} http://host{33...64}/export{33...64} - good (same as {1...64})

Similarity standalone

minio server dir{1...64} dir{65...128} - good (same as {1...128})
minio server dir{1...128} - good (same as {1...64} {65...128})

Cases which should fail

minio server dir{1...64} dir{1...64} - fail duplicates
minio server http://192.168.{1...4}.{1...4}/export{1...4} - fail parse error

New format.json

{
  "version": "1",
  "format": "xl",
  "xl": {
    "version": "2",
    "this": "4ec63786-3dbd-4a9e-96f5-535f6e850fb1",
    "sets": [][
      "4ec63786-3dbd-4a9e-96f5-535f6e850fb1",
      "1f3cf889-bc90-44ca-be2a-732b53be6c9d",
      "4b23eede-1846-482c-b96f-bfb647f058d3",
      "e1f17302-a850-419d-8cdb-a9f884a63c92"
    ], [
      "2ca4c5c1-dccb-4198-a840-309fea3b5449",
      "6d1e666e-a22c-4db4-a038-2545c2ccb6d5",
      "d4fa35ab-710f-4423-a7c2-e1ca33124df0",
      "88c65e8b-00cb-4037-a801-2549119c9a33"
    ]
  }
}

New format-config-v1.go behavior

Format structure is used as a opaque type, .Format field signifies the bootstrap format of the backend. Once the format has been identified it is now the job of the identified backend to further interpret the next structures.

type formatType string

const (
     formatFS formatType = "fs"
     formatXL            = "xl"
)

type format struct {
     Version string
     Format  BackendFormat
}

func newFormat(fmt string) (format)
func parseFormat(buf []byte) (format, error) {
     var f format
     json.Unmarshal(buf, &f)
     return f, nil
}


Current XL format.

type formatXLV1 struct{
     format
     XL struct{
        Version string
        Disk string
        JBOD []string
     }
}

New `formatXLV2` will be following

type formatXLV2 struct{
     format
     XL struct{
        Version string  `json:"version"`
        This string     `json:"this"`
        Sets [][]string `json:"sets"`
     }
}

Current FS format.

type formatFSV1 struct{
     format
     FS struct{
        Version string
     }
}