fix: make sure to convert the error properly in HealBucket()

[harshavardhana]

Description

fix: make sure to convert the error properly in HealBucket()

Motivation and Context

server startup code expects the object layer to properly
convert error into a proper type, so that in situations when
servers are coming up and quorum is not available servers
wait on each other.

How to test this PR?

A simple local 2 node setup can easily reproduce this

#!/usr/bin/env bash

export MINIO_ACCESS_KEY="minio"
export MINIO_SECRET_KEY="minio123"
for i in {01..01}; do
    minio server --address ":90${i}" https://127.0.0.1:9001/home/harsha/disk{01..02} https://127.0.0.1:9002/home/harsha/disk{03..04} &
done

and then start

minio server --address ":9002" https://127.0.0.1:9001/home/harsha/disk{01..02} https://127.0.0.1:9002/home/harsha/disk{03..04}

You need to make sure the setup has been already formatted prior to this test and you create a fake bucket on disk01, disk02

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Documentation updated
• Unit tests added/updated

[minio-trusted]

Mint Automation

Test	Result
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-large-bucket.sh	❌ more...
mint-compress-encrypt-dist-erasure.sh	❌ more...

11610-18bb0d5/mint-compress-encrypt-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:30766
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp e2003fee61f5:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 5 seconds
(5/15) Running awscli tests ... FAILED in 33 seconds
{
  "name": "awscli",
  "duration": 2943,
  "function": "aws --endpoint-url http://minio-dev7.minio.io:30766 s3api copy-object --bucket awscli-mint-test-bucket-7520 --key datafile-1-kB-copy --copy-source awscli-mint-test-bucket-7520/datafile-1-kB\n",
  "status": "FAIL",
  "error": "Hash mismatch expected 084e1383b70fb0c51acc680fef370023, got ac57de7156d7fc25ac1a65f81fa3989b"
}
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 46 seconds
(7/15) Running minio-dotnet tests ... done in 41 seconds
(8/15) Running minio-go tests ... FAILED in 2 minutes and 21 seconds
{
  "args": {
    "destination": {
      "Bucket": "minio-go-test-u02fexk9e9zcsgov",
      "Object": "dstObject",
      "Encryption": {},
      "UserMetadata": null,
      "ReplaceMetadata": false,
      "UserTags": null,
      "ReplaceTags": false,
      "LegalHold": "",
      "Mode": "",
      "RetainUntilDate": "0001-01-01T00:00:00Z",
      "Size": 0,
      "Progress": null
    },
    "source": {
      "Bucket": "minio-go-test-u02fexk9e9zcsgov",
      "Object": "srcObject",
      "VersionID": "",
      "MatchETag": "",
      "NoMatchETag": "",
      "MatchModifiedSince": "0001-01-01T00:00:00Z",
      "MatchUnmodifiedSince": "0001-01-01T00:00:00Z",
      "MatchRange": false,
      "Start": 0,
      "End": 0,
      "Encryption": null
    }
  },
  "duration": 5331,
  "error": "We encountered an internal error, please try again.: cause(s2: corrupt input)",
  "function": "CopyObject(destination, source)",
  "message": "GetObject failed",
  "name": "minio-go: testUnencryptedToSSES3CopyObject",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... FAILED in 1 minutes and 43 seconds
{
  "name": "minio-java",
  "function": "copyObject()",
  "args": "[match etag]",
  "duration": 249,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = PreconditionFailed, message = At least one of the pre-conditions you specified did not hold, bucketName = minio-java-test-3niqeql, objectName = minio-java-test-2sv271v-copy, resource = /minio-java-test-3niqeql/minio-java-test-2sv271v-copy, requestId = 16664FFEFB45520F, hostId = 45d7400e-1362-4ee4-bfc2-73a24a3d63a9)\nrequest={method=PUT, url=http://minio-dev7.minio.io:30766/minio-java-test-3niqeql/minio-java-test-2sv271v-copy, headers=x-amz-copy-source-if-match: 71cff0a060f852067e443ad1e24ae26c-1\nx-amz-copy-source: /minio-java-test-12q5vel/minio-java-test-2sv271v\nHost: minio-dev7.minio.io:30766\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20210223T073113Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210223/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=412, headers=Accept-Ranges: bytes\nContent-Length: 418\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nETag: \"71cff0a060f852067e443ad1e24ae26c\"\nLast-Modified: Tue, 23 Feb 2021 07:31:12 GMT\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 16664FFEFB45520F\nX-Xss-Protection: 1; mode=block\nDate: Tue, 23 Feb 2021 07:31:13 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.execute(MinioClient.java:563), io.minio.MinioClient.executePut(MinioClient.java:904), io.minio.MinioClient.copyObject(MinioClient.java:1232), FunctionalTest.testCopyObjectMatchETag(FunctionalTest.java:1850), FunctionalTest.copyObject(FunctionalTest.java:2016), FunctionalTest.runObjectTests(FunctionalTest.java:3757), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(8/15) Running minio-js tests ... done in 1 minutes and 0 seconds
(9/15) Running minio-py tests ... done in 3 minutes and 8 seconds
(10/15) Running s3cmd tests ... FAILED in 5 seconds
{
  "name": "s3cmd",
  "duration": "3256",
  "function": "test_put_object_multipart",
  "status": "FAIL",
  "error": "WARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Too many failures. Giving up on '/mint/data/datafile-65-MB'\nERROR: \nUpload of '/mint/data/datafile-65-MB' part 1 failed. Use\n  /usr/local/bin/s3cmd abortmp s3://s3cmd-test-bucket-18310/s3cmd-test-object-9997 f864e4e8-f553-41e4-bdb1-6abaaae7f40c\nto abort the upload, or\n  /usr/local/bin/s3cmd --upload-id f864e4e8-f553-41e4-bdb1-6abaaae7f40c put ...\nto continue the upload.\nERROR: Upload of '/mint/data/datafile-65-MB' failed too many times (Last reason: )"
}
(10/15) Running s3select tests ... done in 9 seconds
(11/15) Running security tests ... done in 0 seconds

Executed 11 out of 15 tests successfully.

11610-18bb0d5/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:31143
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 3eac70666c24:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 33 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 1 minutes and 15 seconds
(8/15) Running minio-dotnet tests ... done in 56 seconds
(9/15) Running minio-go tests ... done in 2 minutes and 16 seconds
(10/15) Running minio-java tests ... FAILED in 11 minutes and 12 seconds
{
  "name": "minio-java",
  "function": "composeObject()",
  "args": "[two multipart sources with offset and length]",
  "duration": 300170,
  "status": "FAIL",
  "error": "java.net.SocketTimeoutException: timeout >>> [okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:143), okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:162), okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:340), okio.RealBufferedSource.indexOf(RealBufferedSource.kt:449), okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:333), okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29), okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178), okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106), okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76), okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109), okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201), okhttp3.internal.connection.RealCall.execute(RealCall.kt:154), io.minio.MinioClient.execute(MinioClient.java:647), io.minio.MinioClient.abortMultipartUpload(MinioClient.java:4071), io.minio.MinioClient.composeObject(MinioClient.java:1479), FunctionalTest.testComposeObject(FunctionalTest.java:2120), FunctionalTest.composeObjectTests(FunctionalTest.java:2195), FunctionalTest.composeObject(FunctionalTest.java:2300), FunctionalTest.runObjectTests(FunctionalTest.java:3758), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(10/15) Running minio-js tests ... done in 51 seconds
(11/15) Running minio-py tests ... done in 2 minutes and 54 seconds
(12/15) Running s3cmd tests ... done in 21 seconds
(13/15) Running s3select tests ... done in 7 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

[klauspost]

LGTM