validate storage class across pools when setting config

[harshavardhana]

Description

validate storage class across pools when setting config

Motivation and Context

mc admin config set alias/ storage_class standard=EC:3

should only succeed if parity ratio is valid for all
server pools, if not we should fail proactively.

This PR also needs to bring other changes now that
we need to cater for variadic drive counts per pool.

How to test this PR?

~ mc admin config set alias/ storage_class EC:3

Will fail because it doesn't satisfy the second pool

version: '3.7'
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:       
  minio1:                    
    image: y4m4/minio:dev       
    expose:                               
      - "9000"                                                                                 
    environment:
      MINIO_ACCESS_KEY: minio                                                                  
      MINIO_SECRET_KEY: minio123
      MINIO_STORAGE_CLASS_STANDARD: "EC:2"
    command: server http://minio{1...4}/data{1...2} http://minio{5...6}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  minio2:                    
    image: y4m4/minio:dev       
    expose:                               
      - "9000"                                                                                 
    environment:
      MINIO_ACCESS_KEY: minio                                                                  
      MINIO_SECRET_KEY: minio123
      MINIO_STORAGE_CLASS_STANDARD: "EC:2"
    command: server http://minio{1...4}/data{1...2} http://minio{5...6}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  minio3:                    
    image: y4m4/minio:dev       
    expose:                               
      - "9000"                                                                                 
    environment:
      MINIO_ACCESS_KEY: minio                                                                  
      MINIO_SECRET_KEY: minio123
      MINIO_STORAGE_CLASS_STANDARD: "EC:2"
    command: server http://minio{1...4}/data{1...2} http://minio{5...6}/data{1...2}
    healthcheck:                                                                                                                                                                              
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  minio4:
    image: y4m4/minio:dev
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
      MINIO_STORAGE_CLASS_STANDARD: "EC:2"
    command: server http://minio{1...4}/data{1...2} http://minio{5...6}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  minio5:
    image: y4m4/minio:dev
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
      MINIO_STORAGE_CLASS_STANDARD: "EC:2"
    command: server http://minio{1...4}/data{1...2} http://minio{5...6}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  minio6:
    image: y4m4/minio:dev
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
      MINIO_STORAGE_CLASS_STANDARD: "EC:2"
    command: server http://minio{1...4}/data{1...2} http://minio{5...6}/data{1...2}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Documentation updated
• Unit tests added/updated

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

11320-dc8d1a8/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:32607
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 5ba284336a25:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 2 minutes and 56 seconds
(4/15) Running aws-sdk-ruby tests ... done in 22 seconds
(5/15) Running awscli tests ... FAILED in 51 seconds
{
  "name": "awscli",
  "duration": 1397,
  "function": "make_bucket\n",
  "status": "FAIL",
  "error": "An error occurred (BucketAlreadyOwnedByYou) when calling the CreateBucket operation: Your previous request to create the named bucket succeeded and you already own it."
}
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 3 minutes and 47 seconds
(7/15) Running minio-dotnet tests ... done in 1 minutes and 44 seconds
(8/15) Running minio-go tests ... done in 6 minutes and 28 seconds
(9/15) Running minio-java tests ... FAILED in 8 minutes and 58 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 171,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:15c4071b-d01e-0058-107b-f0e995000000\nTime:2021-01-22T05:03:08.9008932Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'B0YKtekqT2S69WDg4cy6JeKxdLnYVrtVHpbCPoiH2gM=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:44b510df-3fce-4352-6ba4-662b0bed9651\nx-ms-date:Fri, 22 Jan 2021 05:03:08 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:商å“�\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-2vltn8e/minio-java-test-1mvc5ok\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-2vltn8e/minio-java-test-1mvc5ok?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2021-01-22T04:31:27Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [44b510df-3fce-4352-6ba4-662b0bed9651]\n   X-Ms-Date: [Fri, 22 Jan 2021 05:03:08 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1092]\n   Content-Type: [application/xml]\n   Date: [Fri, 22 Jan 2021 05:03:08 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [15c4071b-d01e-0058-107b-f0e995000000]\n\n\n, bucketName = minio-java-test-2vltn8e, objectName = minio-java-test-1mvc5ok, resource = /minio-java-test-2vltn8e/minio-java-test-1mvc5ok, requestId = 165C75584ACD5668, hostId = 3cf1f8e1-e37c-40a5-b8fb-5b1105bd8a3b)\nrequest={method=PUT, url=http://minio-dev6.minio.io:32607/minio-java-test-2vltn8e/minio-java-test-1mvc5ok, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-dev6.minio.io:32607\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20210122T050308Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210122/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3086\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 165C75584ACD5668\nX-Xss-Protection: 1; mode=block\nDate: Fri, 22 Jan 2021 05:03:08 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(9/15) Running minio-js tests ... FAILED in 43 seconds
{
  "name": "minio-js",
  "function": "makeBucket(bucketName, region)",
  "args": "bucketName:minio-js-test-596ce2e6-df2c-4634-b30d-582952768ab4-region, region:us-east-1",
  "duration": 40423,
  "status": "FAIL",
  "error": "S3Error: The specified bucket does not exist at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:79:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (internal/streams/readable.js:684:10) at endReadableNT (internal/streams/readable.js:1327:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(9/15) Running minio-py tests ... done in 18 minutes and 30 seconds
(10/15) Running s3cmd tests ... done in 2 minutes and 22 seconds
(11/15) Running s3select tests ... done in 1 minutes and 0 seconds
(12/15) Running security tests ... done in 0 seconds

Executed 12 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

[klauspost]

LGTM

[kannappanr]

LGTM