Support bucket versioning #9377

harshavardhana · 2020-04-17T00:45:57Z

Implement a new xl.json 2.0.0 format to support,
this moves the entire marshalling logic to POSIX
layer, top layer always consumes a common FileInfo
construct which simplifies the metadata reads.

Fixes #2111

Description

Motivation and Context

How to test this PR?

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed
Functional tests needed (If yes, add mint PR # here: )

- Implement a new xl.json 2.0.0 format to support, this moves the entire marshaling logic to POSIX layer, top layer always consumes a common FileInfo construct which simplifies the metadata reads. - Implement list object versions - Migrate to siphash from crchash for new deployments for object placements. Fixes minio#2111

kannappanr

LGTM

minio-trusted · 2020-06-13T04:39:00Z

Mint Automation

Test	Result
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-gateway-azure.sh	✔️
mint-gateway-nas.sh	✔️
mint-xl.sh	❌ more...
mint-large-bucket.sh	❌ more...
mint-dist-xl.sh	❌ more...

9377-462814c/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:32506
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 148cb68d0821:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 18 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 46 seconds
(8/15) Running minio-dotnet tests ... done in 44 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 12 seconds
{
  "args": {
    "bucketName": "minio-go-test-4k2tfyl5ztte1drz",
    "objectName": "pscp69ijswpuhcsxjvldm0jlw1pgqo",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 5693,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 2 minutes and 33 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 443,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-6vu1pf, objectName = null, resource = /minio-java-test-6vu1pf, requestId = 1617FDD730983163, hostId = 3fcf3c90-5488-4b52-8d2b-94c1a68b211c)\nrequest={method=DELETE, url=http://minio-dev6.minio.io:32506/minio-java-test-6vu1pf, headers=Host: minio-dev6.minio.io:32506\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200613T034813Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200613/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 371\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.GOGET\nVary: Origin\nX-Amz-Request-Id: 1617FDD730983163\nX-Xss-Protection: 1; mode=block\nDate: Sat, 13 Jun 2020 03:48:13 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 49 seconds
(10/15) Running minio-py tests ... done in 1 minutes and 51 seconds
(11/15) Running s3cmd tests ... done in 27 seconds
(12/15) Running s3select tests ... done in 13 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9377-462814c/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:31309
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp df445bffee1f:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 5 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 47 seconds
(4/15) Running aws-sdk-ruby tests ... done in 10 seconds
(5/15) Running awscli tests ... done in 2 minutes and 37 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 49 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 11 seconds
(9/15) Running minio-go tests ... FAILED in 2 minutes and 6 seconds
{
  "args": {
    "bucketName": "minio-go-test-2hiajjv2ydqwuau2",
    "objectName": "0jrabtlzt2lqxqueqgtynvahy9ip1z",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 4093,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 7 minutes and 7 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 1069,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-31ml57j, objectName = null, resource = /minio-java-test-31ml57j, requestId = 1617FE322D53CB29, hostId = 5f01ce2f-2057-468e-8c99-e5da96568688)\nrequest={method=DELETE, url=http://minio-dev6.minio.io:31309/minio-java-test-31ml57j, headers=Host: minio-dev6.minio.io:31309\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200613T035444Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200613/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.GOGET\nVary: Origin\nX-Amz-Request-Id: 1617FE322D53CB29\nX-Xss-Protection: 1; mode=block\nDate: Sat, 13 Jun 2020 03:54:44 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 57 seconds
(10/15) Running minio-py tests ... done in 5 minutes and 7 seconds
(11/15) Running s3cmd tests ... done in 27 seconds
(12/15) Running s3select tests ... done in 15 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9377-462814c/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:30567
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp b20ea28aa255:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 0 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 41 seconds
(4/15) Running aws-sdk-ruby tests ... done in 3 seconds
(5/15) Running awscli tests ... FAILED in 2 minutes and 5 seconds
{
  "name": "awscli",
  "duration": 62476,
  "function": "aws --endpoint-url http://minio-dev7.minio.io:30567 s3 sync --no-progress /mint/data s3://awscli-mint-test-bucket-17901/\n",
  "status": "FAIL",
  "error": "upload: ../../../data/datafile-0-b to s3://awscli-mint-test-bucket-17901/datafile-0-bupload: ../../../data/datafile-1-kB to s3://awscli-mint-test-bucket-17901/datafile-1-kBupload: ../../../data/datafile-1-b to s3://awscli-mint-test-bucket-17901/datafile-1-bupload: ../../../data/datafile-33-kB to s3://awscli-mint-test-bucket-17901/datafile-33-kBupload: ../../../data/datafile-100-kB to s3://awscli-mint-test-bucket-17901/datafile-100-kBupload: ../../../data/datafile-1.03-MB to s3://awscli-mint-test-bucket-17901/datafile-1.03-MBupload: ../../../data/datafile-1-MB to s3://awscli-mint-test-bucket-17901/datafile-1-MBupload: ../../../data/datafile-5-MB to s3://awscli-mint-test-bucket-17901/datafile-5-MBupload: ../../../data/datafile-5243880-b to s3://awscli-mint-test-bucket-17901/datafile-5243880-bupload: ../../../data/datafile-6-MB to s3://awscli-mint-test-bucket-17901/datafile-6-MBupload: ../../../data/datafile-11-MB to s3://awscli-mint-test-bucket-17901/datafile-11-MBupload: ../../../data/datafile-10-MB to s3://awscli-mint-test-bucket-17901/datafile-10-MBupload: ../../../data/datafile-65-MB to s3://awscli-mint-test-bucket-17901/datafile-65-MBupload failed: ../../../data/datafile-129-MB to s3://awscli-mint-test-bucket-17901/datafile-129-MB An error occurred (RequestTimeout) when calling the UploadPart operation (reached max retries: 4): A timeout occurred while trying to lock a resource, please reduce your request rateupload: ../../../data/datafile-10-kB to s3://awscli-mint-test-bucket-17901/datafile-10-kB"
}
(5/15) Running healthcheck tests ... done in 1 seconds
(6/15) Running mc tests ... done in 43 seconds
(7/15) Running minio-dotnet tests ... done in 30 seconds
(8/15) Running minio-go tests ... FAILED in 50 seconds
{
  "args": {
    "bucketName": "minio-go-test-b4ij6vujeyosq1nx",
    "objectName": "dxksjblf2qg0qn4yssbn1cuqft3huo",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 2581,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... FAILED in 59 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 101,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-1tsjpfl, objectName = null, resource = /minio-java-test-1tsjpfl, requestId = 1617FDB478091AFF, hostId = 443bbac9-dcda-4177-9c19-9188ef937b3d)\nrequest={method=DELETE, url=http://minio-dev7.minio.io:30567/minio-java-test-1tsjpfl, headers=Host: minio-dev7.minio.io:30567\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200613T034544Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200613/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.GOGET\nVary: Origin\nX-Amz-Request-Id: 1617FDB478091AFF\nX-Xss-Protection: 1; mode=block\nDate: Sat, 13 Jun 2020 03:45:44 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(8/15) Running minio-js tests ... done in 53 seconds
(9/15) Running minio-py tests ... done in 1 minutes and 20 seconds
(10/15) Running s3cmd tests ... done in 19 seconds
(11/15) Running s3select tests ... done in 3 seconds
(12/15) Running security tests ... done in 0 seconds

Executed 12 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

Fixes: minio#10283 regression from PR(s) minio#9259 and minio#9377

harshavardhana force-pushed the xl-v2 branch 12 times, most recently from 9602a9d to ac7eb8a Compare April 22, 2020 02:22

harshavardhana marked this pull request as ready for review April 22, 2020 02:22

harshavardhana force-pushed the xl-v2 branch 7 times, most recently from 232cc56 to 8e83285 Compare April 27, 2020 05:45

klauspost mentioned this pull request Apr 27, 2020

Store bucket created time as a metadata #9465

Merged

7 tasks

harshavardhana force-pushed the xl-v2 branch 2 times, most recently from 90e9d3f to 27663bd Compare April 27, 2020 20:37

kannappanr requested review from klauspost, aead, donatello, krishnasrinivas, vadmeste and kannappanr April 27, 2020 21:41

harshavardhana force-pushed the xl-v2 branch from 27663bd to 9065f1b Compare April 27, 2020 21:56

harshavardhana force-pushed the xl-v2 branch 11 times, most recently from 07ad04f to 6388970 Compare June 9, 2020 01:28

harshavardhana mentioned this pull request Jun 9, 2020

deprecate listDir usage for healing #9792

Merged

7 tasks

harshavardhana force-pushed the xl-v2 branch 9 times, most recently from d7e19f4 to 96b3b60 Compare June 12, 2020 22:22

harshavardhana force-pushed the xl-v2 branch from 96b3b60 to 462814c Compare June 13, 2020 02:09

kannappanr approved these changes Jun 13, 2020

View reviewed changes

harshavardhana merged commit 4915433 into minio:master Jun 13, 2020

harshavardhana deleted the xl-v2 branch June 13, 2020 03:04

poornas pushed a commit to poornas/minio that referenced this pull request Aug 18, 2020

Fix regression in web ui for retention

6485a7a

Fixes: minio#10283 regression from PR(s) minio#9259 and minio#9377

poornas pushed a commit to poornas/minio that referenced this pull request Aug 18, 2020

Fix regression in web ui for retention

8e62062

Fixes: minio#10283 regression from PR(s) minio#9259 and minio#9377

poornas mentioned this pull request Aug 18, 2020

Fix regression in web ui for retention #10285

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support bucket versioning #9377

Support bucket versioning #9377

harshavardhana commented Apr 17, 2020 •

edited

Loading

kannappanr left a comment

minio-trusted commented Jun 13, 2020

Support bucket versioning #9377

Support bucket versioning #9377

Conversation

harshavardhana commented Apr 17, 2020 • edited Loading

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

kannappanr left a comment

Choose a reason for hiding this comment

minio-trusted commented Jun 13, 2020

Mint Automation

9377-462814c/mint-dist-xl.sh.log:

9377-462814c/mint-large-bucket.sh.log:

9377-462814c/mint-xl.sh.log:

harshavardhana commented Apr 17, 2020 •

edited

Loading