fix: make sure to use new restClient for healthcheck #10026

harshavardhana · 2020-07-12T01:17:37Z

Description

fix: make sure to use new restClient for healthcheck

Motivation and Context

Without instantiating a new rest-client we can
have a recursive error which can lead to
health check returning always offline, this can
prematurely take the servers offline.

How to test this PR?

reproducible easily in container/k8s environments

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed

harshavardhana · 2020-07-12T02:37:34Z

@klauspost this is also perhaps the fix for the lockups that you saw.

kannappanr

LGTM

Without instantiating a new rest client we can have a recursive error which can lead to healthcheck returning always offline, this can prematurely take the servers offline.

minio-trusted · 2020-07-12T04:57:53Z

Mint Automation

Test	Result
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-gateway-nas.sh	✔️
mint-xl.sh	❌ more...
mint-large-bucket.sh	❌ more...
mint-dist-xl.sh	❌ more...
mint-gateway-azure.sh	❌ more...
mint-zoned.sh	❌ more...

10026-1298c7d/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:32233
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 59cd9ecd8527:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 3 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 5 seconds
(5/15) Running awscli tests ... done in 2 minutes and 13 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 57 seconds
(8/15) Running minio-dotnet tests ... done in 59 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 39 seconds
{
  "args": {
    "bucketName": "minio-go-test-pl0gozxn0prkaxjg",
    "objectName": "yvfeqglfukgzs1iwgpz4as9yplzuua",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 3518,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 2 minutes and 10 seconds
(10/15) Running minio-js tests ... done in 58 seconds
(11/15) Running minio-py tests ... done in 2 minutes and 54 seconds
(12/15) Running s3cmd tests ... done in 22 seconds
(13/15) Running s3select tests ... done in 8 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

10026-1298c7d/mint-zoned.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:30873
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 29e061478d9b:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 5 seconds
(5/15) Running awscli tests ... done in 1 minutes and 25 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 41 seconds
(8/15) Running minio-dotnet tests ... done in 56 seconds
(9/15) Running minio-go tests ... FAILED in 2 minutes and 2 seconds
{
  "args": {
    "bucketName": "minio-go-test-glxrv9bq3cqhsr4i",
    "objectName": "qjb16ntmpsbddtawav52ptsabqh9fj",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 4985,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 1 minutes and 50 seconds
(10/15) Running minio-js tests ... done in 1 minutes and 1 seconds
(11/15) Running minio-py tests ... done in 2 minutes and 33 seconds
(12/15) Running s3cmd tests ... done in 25 seconds
(13/15) Running s3select tests ... done in 7 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

10026-1298c7d/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:32209
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 6d8551cfc349:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 7 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 3 minutes and 30 seconds
(4/15) Running aws-sdk-ruby tests ... done in 17 seconds
(5/15) Running awscli tests ... done in 2 minutes and 8 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 2 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 34 seconds
(9/15) Running minio-go tests ... done in 5 minutes and 21 seconds
(10/15) Running minio-java tests ... done in 6 minutes and 5 seconds
(11/15) Running minio-js tests ... FAILED in 1 minutes and 58 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 83,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:673:10) at endReadableNT (_stream_readable.js:1201:12) at processTicksAndRejections (internal/process/task_queues.js:84:21)"
}
(11/15) Running minio-py tests ... done in 9 minutes and 30 seconds
(12/15) Running s3cmd tests ... done in 1 minutes and 23 seconds
(13/15) Running s3select tests ... done in 1 minutes and 4 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

10026-1298c7d/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:30088
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 447008ab1559:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 7 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 9 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 49 seconds
(8/15) Running minio-dotnet tests ... done in 47 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 21 seconds
{
  "args": {
    "bucketName": "minio-go-test-9nvtzsrcjrvs9r5e",
    "objectName": "3moqsl10g6kyhbwpp9tdowncgxcms3",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 3068,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 1 minutes and 24 seconds
(10/15) Running minio-js tests ... done in 1 minutes and 1 seconds
(11/15) Running minio-py tests ... done in 2 minutes and 18 seconds
(12/15) Running s3cmd tests ... FAILED in 10 seconds
{
  "name": "s3cmd",
  "duration": "327",
  "function": "test_sync_list_objects",
  "status": "FAIL",
  "error": "ERROR: S3 error: 409 (BucketAlreadyOwnedByYou): Your previous request to create the named bucket succeeded and you already own it."
}
(12/15) Running s3select tests ... done in 7 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

10026-1298c7d/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:32612
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp e38ffa55cf13:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 0 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 42 seconds
(4/15) Running aws-sdk-ruby tests ... done in 2 seconds
(5/15) Running awscli tests ... done in 1 minutes and 15 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 29 seconds
(8/15) Running minio-dotnet tests ... done in 34 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 27 seconds
{
  "args": {
    "bucketName": "minio-go-test-m32brosf4sxm5tpl",
    "objectName": "f3h1i2twqbwaqduqp6ywajl5pl1dpb",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 8397,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 52 seconds
(10/15) Running minio-js tests ... done in 51 seconds
(11/15) Running minio-py tests ... done in 1 minutes and 40 seconds
(12/15) Running s3cmd tests ... done in 22 seconds
(13/15) Running s3select tests ... done in 4 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

harshavardhana · 2020-07-12T05:19:32Z

Going to take it in as we need to make a release wiith this fix.

harshavardhana requested review from kannappanr and klauspost July 12, 2020 01:17

harshavardhana added the priority: high label Jul 12, 2020

harshavardhana force-pushed the azure branch 3 times, most recently from d2832ec to 228dc9d Compare July 12, 2020 02:29

harshavardhana changed the title ~~fix: treat ContainerBeingDeleted as bucket not found~~ fix: make sure to use new restClient for healthcheck Jul 12, 2020

harshavardhana force-pushed the azure branch from 228dc9d to 2635eda Compare July 12, 2020 02:35

kannappanr approved these changes Jul 12, 2020

View reviewed changes

fix: make sure to use new restClient for healthcheck

1298c7d

Without instantiating a new rest client we can have a recursive error which can lead to healthcheck returning always offline, this can prematurely take the servers offline.

harshavardhana force-pushed the azure branch from 2635eda to 1298c7d Compare July 12, 2020 04:15

harshavardhana merged commit 3b9fbf8 into minio:master Jul 12, 2020

harshavardhana deleted the azure branch July 12, 2020 05:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: make sure to use new restClient for healthcheck #10026

fix: make sure to use new restClient for healthcheck #10026

harshavardhana commented Jul 12, 2020 •

edited

harshavardhana commented Jul 12, 2020

kannappanr left a comment

minio-trusted commented Jul 12, 2020

harshavardhana commented Jul 12, 2020

fix: make sure to use new restClient for healthcheck #10026

fix: make sure to use new restClient for healthcheck #10026

Conversation

harshavardhana commented Jul 12, 2020 • edited

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

harshavardhana commented Jul 12, 2020

kannappanr left a comment

Choose a reason for hiding this comment

minio-trusted commented Jul 12, 2020

Mint Automation

10026-1298c7d/mint-large-bucket.sh.log:

10026-1298c7d/mint-zoned.sh.log:

10026-1298c7d/mint-gateway-azure.sh.log:

10026-1298c7d/mint-dist-xl.sh.log:

10026-1298c7d/mint-xl.sh.log:

harshavardhana commented Jul 12, 2020

harshavardhana commented Jul 12, 2020 •

edited