Recreate bucket metacache if corrupted

[klauspost]

Description

If bucket metadata cannot be read, clean up existing and create a new.

Motivation and Context

If bucket metadata ended up with a read quorum error there was no automatic recovery.

Cleanup will remove any leftover data from the lost cache information.

How to test this PR?

Make the .minio.sys/buckets/BUCKET/.metacache/index.s2 impossible to read.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...
mint-dist-erasure.sh	❌ more...

10800-2bb36aa/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev4.minio.io:30838
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 3a1a9245a5fe:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 3 minutes and 15 seconds
(4/15) Running aws-sdk-ruby tests ... done in 22 seconds
(5/15) Running awscli tests ... done in 2 minutes and 46 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 18 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 36 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 52 seconds
{
  "args": {
    "bucketName": "minio-go-test-pyge1z1zbv1g31lz",
    "objectName": "test-object",
    "opts": "",
    "size": -1
  },
  "duration": 1152,
  "function": "PutObject(bucketName, objectName, reader,size,opts)",
  "message": "Unexpected size",
  "name": "minio-go: testPutObjectStreaming",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 10 minutes and 10 seconds
(10/15) Running minio-js tests ... FAILED in 47 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 80,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:682:10) at endReadableNT (_stream_readable.js:1252:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(10/15) Running minio-py tests ... done in 18 minutes and 12 seconds
(11/15) Running s3cmd tests ... done in 2 minutes and 15 seconds
(12/15) Running s3select tests ... done in 38 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

10800-2bb36aa/mint-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:32265
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp a696f5bd97c9:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 1 minutes and 15 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 51 seconds
(8/15) Running minio-dotnet tests ... done in 41 seconds
(9/15) Running minio-go tests ... done in 2 minutes and 29 seconds
(10/15) Running minio-java tests ... done in 1 minutes and 12 seconds
(11/15) Running minio-js tests ... FAILED in 10 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 16,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:682:10) at endReadableNT (_stream_readable.js:1252:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(11/15) Running minio-py tests ... done in 3 minutes and 15 seconds
(12/15) Running s3cmd tests ... done in 20 seconds
(13/15) Running s3select tests ... done in 6 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

[kannappanr]

LGTM