Allow etcd, cache setup to exit when starting gateway mode #9842

harshavardhana · 2020-06-15T23:21:59Z

Description

Initialize etcd once per call
Fail etcd, cache setup pro-actively for gateway setups
Support deleting/updating bucket notification,
tagging, lifecycle, sse-encryption

Motivation and Context

Fixes #9841 #9788

How to test this PR?

As per the referenced issues.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed
Functional tests needed (If yes, add mint PR # here: )

- Initialize etcd once per call - Fail etcd, cache setup pro-actively for gateway setups - Support deleting/updating bucket notification, tagging, lifecycle, sse-encryption

minio-trusted · 2020-06-16T02:18:42Z

Mint Automation

Test	Result
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-gateway-azure.sh	✔️
mint-gateway-nas.sh	✔️
mint-xl.sh	❌ more...
mint-large-bucket.sh	❌ more...
mint-dist-xl.sh	❌ more...

9842-6effe01/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:31332
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 73ff2f566ab6:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 3 seconds
(5/15) Running awscli tests ... done in 2 minutes and 23 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 52 seconds
(8/15) Running minio-dotnet tests ... done in 49 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 20 seconds
{
  "args": {
    "bucketName": "minio-go-test-eq03xxm0uo94em43",
    "objectName": "3s9tayosmuqjjczeencxd3equsweqd",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 4631,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 2 minutes and 32 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 287,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-1qohvh1, objectName = null, resource = /minio-java-test-1qohvh1, requestId = 1618E22A6BC75E23, hostId = cf5f85d4-8b7b-4489-b6d3-4043d58c30b6)\nrequest={method=DELETE, url=http://minio-c3.minio.io:31332/minio-java-test-1qohvh1, headers=Host: minio-c3.minio.io:31332\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200616T013220Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200616/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-16T01-17-39Z\nVary: Origin\nX-Amz-Request-Id: 1618E22A6BC75E23\nX-Xss-Protection: 1; mode=block\nDate: Tue, 16 Jun 2020 01:32:20 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 48 seconds
(10/15) Running minio-py tests ... done in 2 minutes and 3 seconds
(11/15) Running s3cmd tests ... done in 20 seconds
(12/15) Running s3select tests ... done in 5 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9842-6effe01/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev4.minio.io:32689
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 42dd5a8e7ad0:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 4 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 47 seconds
(4/15) Running aws-sdk-ruby tests ... done in 10 seconds
(5/15) Running awscli tests ... done in 2 minutes and 36 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 58 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 12 seconds
(9/15) Running minio-go tests ... FAILED in 2 minutes and 7 seconds
{
  "args": {
    "bucketName": "minio-go-test-h5ngrmbnlf5m4b3q",
    "objectName": "o3w5fvhn6bf3rmouhpworq51oobwbv",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 4375,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 7 minutes and 6 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 1175,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-10snf8c, objectName = null, resource = /minio-java-test-10snf8c, requestId = 1618E2823E1DDB3D, hostId = 2a6451f8-f06a-4201-8be4-9eb9541d72c0)\nrequest={method=DELETE, url=http://minio-dev4.minio.io:32689/minio-java-test-10snf8c, headers=Host: minio-dev4.minio.io:32689\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200616T013837Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200616/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-16T01-17-39Z\nVary: Origin\nX-Amz-Request-Id: 1618E2823E1DDB3D\nX-Xss-Protection: 1; mode=block\nDate: Tue, 16 Jun 2020 01:38:37 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 55 seconds
(10/15) Running minio-py tests ... done in 5 minutes and 16 seconds
(11/15) Running s3cmd tests ... done in 28 seconds
(12/15) Running s3select tests ... done in 14 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9842-6effe01/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:31075
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp e9735d4f92ac:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 0 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 42 seconds
(4/15) Running aws-sdk-ruby tests ... done in 2 seconds
(5/15) Running awscli tests ... done in 1 minutes and 23 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 27 seconds
(8/15) Running minio-dotnet tests ... done in 36 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 17 seconds
{
  "args": {
    "bucketName": "minio-go-test-19facfxb1d4qszsv",
    "objectName": "pgmkegi2riw2joss0cir55gygqxznz",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 5957,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 1 minutes and 1 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 90,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-3jgj4qc, objectName = null, resource = /minio-java-test-3jgj4qc, requestId = 1618E1FCF77F396F, hostId = e0a48534-b1bd-45db-a0a4-1e705078482f)\nrequest={method=DELETE, url=http://minio-dev6.minio.io:31075/minio-java-test-3jgj4qc, headers=Host: minio-dev6.minio.io:31075\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200616T012904Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200616/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-16T01-17-39Z\nVary: Origin\nX-Amz-Request-Id: 1618E1FCF77F396F\nX-Xss-Protection: 1; mode=block\nDate: Tue, 16 Jun 2020 01:29:04 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 1 minutes and 2 seconds
(10/15) Running minio-py tests ... done in 1 minutes and 40 seconds
(11/15) Running s3cmd tests ... done in 19 seconds
(12/15) Running s3select tests ... done in 3 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

poornas

LGTM

kannappanr

LGTM

- Initialize etcd once per call - Fail etcd, cache setup pro-actively for gateway setups - Support deleting/updating bucket notification, tagging, lifecycle, sse-encryption

harshavardhana changed the title ~~Allow etcd, cache setup when starting gateway~~ Allow etcd, cache exit setup when starting gateway Jun 15, 2020

harshavardhana mentioned this pull request Jun 15, 2020

Add MINIO_ETCD_REQUIRED flag to exit minio server #9816

Closed

7 tasks

harshavardhana force-pushed the allow-etcd-retry branch 2 times, most recently from db815a9 to f0e2870 Compare June 15, 2020 23:26

harshavardhana changed the title ~~Allow etcd, cache exit setup when starting gateway~~ Allow etcd, cache setup to exit when starting gateway mode Jun 15, 2020

Allow etcd, cache setup to exit when starting gateway mode

525f45d

- Initialize etcd once per call - Fail etcd, cache setup pro-actively for gateway setups - Support deleting/updating bucket notification, tagging, lifecycle, sse-encryption

harshavardhana force-pushed the allow-etcd-retry branch from f0e2870 to 525f45d Compare June 15, 2020 23:37

harshavardhana requested review from poornas, kannappanr and nitisht June 16, 2020 01:19

poornas approved these changes Jun 16, 2020

View reviewed changes

kannappanr approved these changes Jun 16, 2020

View reviewed changes

harshavardhana merged commit cbb7a09 into minio:master Jun 16, 2020

harshavardhana deleted the allow-etcd-retry branch June 16, 2020 05:09

This was referenced Jun 16, 2020

etcd unavailable during startup disables admin apis, but etcd unavailable later does not #9788

Closed

Gateway startup leaks a Etcd connection #9841

Closed

BigUstad mentioned this pull request Jun 16, 2020

etcd unavailable start of minio or during run #9849

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow etcd, cache setup to exit when starting gateway mode #9842

Allow etcd, cache setup to exit when starting gateway mode #9842

harshavardhana commented Jun 15, 2020 •

edited

minio-trusted commented Jun 16, 2020

poornas left a comment

kannappanr left a comment

Allow etcd, cache setup to exit when starting gateway mode #9842

Allow etcd, cache setup to exit when starting gateway mode #9842

Conversation

harshavardhana commented Jun 15, 2020 • edited

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

minio-trusted commented Jun 16, 2020

Mint Automation

9842-6effe01/mint-dist-xl.sh.log:

9842-6effe01/mint-large-bucket.sh.log:

9842-6effe01/mint-xl.sh.log:

poornas left a comment

Choose a reason for hiding this comment

kannappanr left a comment

Choose a reason for hiding this comment

harshavardhana commented Jun 15, 2020 •

edited