Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix peer server update failure #9824

Merged
merged 1 commit into from
Jun 13, 2020
Merged

Fix peer server update failure #9824

merged 1 commit into from
Jun 13, 2020

Conversation

goodluckethan
Copy link
Contributor

Description

When updating all servers through /minio/admin/v3/update API by mc admin update,
only the endpoint server will be updated successfully.
All the other peer servers' updating failed due to the error below:

parsing time "2006-01-02T15:04:05Z07:00" as "release version": cannot parse "-01-02T15:04:05Z07:00" as "0-"

Motivation and Context

Bug fix of server update

How to test this PR?

  1. Apply this patch on your own env with minium 2 nodes.
  2. Start all servers manully:
    go run main.go server your_own_volume_sets
  3. Update all servers using mc:
    mc --debug admin update your_alias [your_own_private_mirror_site_of_minio]
  4. All servers should be restarted and updated.
    (You can check the server version by /tmp/go-buildxxxx/b001/exe/main -v,
    which can be found uding "ps -ef | grep minio")

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation needed
  • Unit tests needed
  • Functional tests needed (If yes, add mint PR # here: )

@goodluckethan
Fix peer server update failure
fcec408 
When updating all servers following the constructions of mc update,
only the endpoint server will be updated successfully.
All the other peer servers' updating failed due to the error below:
--------------------------------------------------------------------------
parsing time "2006-01-02T15:04:05Z07:00" as "<release version>": cannot parse "-01-02T15:04:05Z07:00" as "0-" 
--------------------------------------------------------------------------
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-gateway-azure.sh ✔️
mint-gateway-nas.sh ✔️
mint-xl.sh ❌ more...
mint-large-bucket.sh ❌ more...
mint-dist-xl.sh ❌ more...

9824-fcec408/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev3.minio.io:30598
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp e169be75a09b:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 42 seconds
(4/15) Running aws-sdk-ruby tests ... done in 2 seconds
(5/15) Running awscli tests ... done in 2 minutes and 14 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 34 seconds
(8/15) Running minio-dotnet tests ... done in 32 seconds
(9/15) Running minio-go tests ... FAILED in 45 seconds
{
  "args": {
    "bucketName": "minio-go-test-o4n2oemr52w1denj",
    "objectName": "14t56q9k1wcsxbv3rvuxouhf22n1s9",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 3074,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 49 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 104,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-rfdkfr, objectName = null, resource = /minio-java-test-rfdkfr, requestId = 161818AEDBF2FF16, hostId = bb70cf7f-22b3-4466-b262-fb3642ada88d)\nrequest={method=DELETE, url=http://minio-dev3.minio.io:30598/minio-java-test-rfdkfr, headers=Host: minio-dev3.minio.io:30598\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200613T120007Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200613/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 371\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-13T11-48-36Z\nVary: Origin\nX-Amz-Request-Id: 161818AEDBF2FF16\nX-Xss-Protection: 1; mode=block\nDate: Sat, 13 Jun 2020 12:00:07 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 47 seconds
(10/15) Running minio-py tests ... done in 1 minutes and 16 seconds
(11/15) Running s3cmd tests ... done in 20 seconds
(12/15) Running s3select tests ... done in 2 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9824-fcec408/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev3.minio.io:32415
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 8bd7279a7456:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 5 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 47 seconds
(4/15) Running aws-sdk-ruby tests ... done in 9 seconds
(5/15) Running awscli tests ... done in 2 minutes and 29 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 56 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 14 seconds
(9/15) Running minio-go tests ... FAILED in 2 minutes and 14 seconds
{
  "args": {
    "bucketName": "minio-go-test-59nimlz45kgrcoqm",
    "objectName": "fdcpu9o3126hvzygvml1djtgho4uj9",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 6258,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 7 minutes and 13 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 1028,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-1tbtj77, objectName = null, resource = /minio-java-test-1tbtj77, requestId = 16181933A224D7A4, hostId = 26896c2d-00a9-4263-a1fc-cc1f6b32920d)\nrequest={method=DELETE, url=http://minio-dev3.minio.io:32415/minio-java-test-1tbtj77, headers=Host: minio-dev3.minio.io:32415\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200613T120937Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200613/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-13T11-48-36Z\nVary: Origin\nX-Amz-Request-Id: 16181933A224D7A4\nX-Xss-Protection: 1; mode=block\nDate: Sat, 13 Jun 2020 12:09:37 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 56 seconds
(10/15) Running minio-py tests ... done in 5 minutes and 14 seconds
(11/15) Running s3cmd tests ... done in 26 seconds
(12/15) Running s3select tests ... done in 15 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9824-fcec408/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev3.minio.io:32443
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 2106f4fdc15a:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 21 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 42 seconds
(8/15) Running minio-dotnet tests ... done in 56 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 3 seconds
{
  "args": {
    "bucketName": "minio-go-test-9fhtg9o9nnl2h0vq",
    "objectName": "42idseqgi6gz933y0amks3dfbo1lnv",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 4372,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 2 minutes and 52 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 244,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-2585nds, objectName = null, resource = /minio-java-test-2585nds, requestId = 161818D9D592D3FE, hostId = 0c98dd16-d91d-46c4-959f-608c48e8dde4)\nrequest={method=DELETE, url=http://minio-dev3.minio.io:32443/minio-java-test-2585nds, headers=Host: minio-dev3.minio.io:32443\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200613T120312Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200613/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-13T11-48-36Z\nVary: Origin\nX-Amz-Request-Id: 161818D9D592D3FE\nX-Xss-Protection: 1; mode=block\nDate: Sat, 13 Jun 2020 12:03:12 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 49 seconds
(10/15) Running minio-py tests ... done in 1 minutes and 50 seconds
(11/15) Running s3cmd tests ... done in 25 seconds
(12/15) Running s3select tests ... done in 10 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana harshavardhana merged commit 535efd3 into minio:master Jun 13, 2020
harshavardhana pushed a commit that referenced this pull request Jun 14, 2020
@goodluckethan @harshavardhana
Fix peer server update failure (#9824)
2db5cbc 
When updating all servers following the constructions of mc update,
only the endpoint server will be updated successfully.
All the other peer servers' updating failed due to the error below:
--------------------------------------------------------------------------
parsing time "2006-01-02T15:04:05Z07:00" as "<release version>": cannot parse "-01-02T15:04:05Z07:00" as "0-" 
--------------------------------------------------------------------------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@harshavardhana harshavardhana harshavardhana approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@goodluckethan @minio-trusted @harshavardhana