Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

heal: Fix periodic healing cleanup #10569

Merged
merged 1 commit into from
Sep 25, 2020
Merged

Conversation

vadmeste
Copy link
Member

Description

isEnded() was incorrectly calculating if the current healing sequence is
ended or not. h.currentStatus.Items could be empty if healing is very
slow and mc admin heal consumed all items.

Motivation and Context

Fix mc admin heal erroring out

How to test this PR?

  1. Create an erasure standalone with many files
  2. Apply the following diff on MinIO server code to facilitate testing:
diff --git a/cmd/admin-heal-ops.go b/cmd/admin-heal-ops.go
index a54fae9a8..7a2770e68 100644
--- a/cmd/admin-heal-ops.go
+++ b/cmd/admin-heal-ops.go
@@ -145,7 +145,7 @@ func (ahs *allHealState) periodicHealSeqsClean(ctx context.Context) {
        // it ends) from the global state after timeout has elapsed.
        for {
                select {
-               case <-time.After(time.Minute * 5):
+               case <-time.After(time.Second * 5):
                        now := UTCNow()
                        ahs.Lock()
                        for path, h := range ahs.healSeqMap {
  1. mc admin heal -r command

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation needed
  • Unit tests needed

isEnded() was incorrectly calculating if the current healing sequence is
ended or not. h.currentStatus.Items could be empty if healing is very
slow and mc admin heal consumed all items.
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-gateway-azure.sh more...

10569-aff697b/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:31923
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp cb68ee389194:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 9 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 12 seconds
(4/15) Running aws-sdk-ruby tests ... done in 20 seconds
(5/15) Running awscli tests ... done in 3 minutes and 52 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 53 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 39 seconds
(9/15) Running minio-go tests ... done in 6 minutes and 25 seconds
(10/15) Running minio-java tests ... done in 10 minutes and 25 seconds
(11/15) Running minio-js tests ... done in 2 minutes and 39 seconds
(12/15) Running minio-py tests ... done in 19 minutes and 0 seconds
(13/15) Running s3cmd tests ... done in 2 minutes and 17 seconds
(14/15) Running s3select tests ... FAILED in 5 seconds
{
  "name": "s3select:test_csv_input_quote_char",
  "function": "select_object_content(bucket_name, object_name, opts)",
  "args": {
    "bucket_name": "s3select-test-82da4682-46ce-455e-b503-83a0b21a7a67"
  },
  "duration": 4712,
  "message": "Test test_5 unexpectedly failed with: 'event-type'",
  "error": "Traceback (most recent call last):\n  File \"/mint/run/core/s3select/csv.py\", line 39, in test_sql_api\n    for d in data.stream(10*1024):\n  File \"/usr/local/lib/python3.6/dist-packages/minio/select/reader.py\", line 211, in stream\n    message = self.__extract_message()\n  File \"/usr/local/lib/python3.6/dist-packages/minio/select/reader.py\", line 161, in __extract_message\n    event_type = header_map[\"event-type\"]\nKeyError: 'event-type'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"./tests.py\", line 52, in main\n    test_csv_input_custom_quote_char(client, log_output)\n  File \"/mint/run/core/s3select/csv.py\", line 119, in test_csv_input_custom_quote_char\n    data, sql_opts, expected_output)\n  File \"/mint/run/core/s3select/csv.py\", line 45, in test_sql_api\n    'Test {} unexpectedly failed with: {}'.format(test_name, select_err))\nValueError: Test test_5 unexpectedly failed with: 'event-type'\n",
  "status": "FAIL"
}
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally
Error: No such image: minio/minio:10569-aff697b

@harshavardhana harshavardhana merged commit b302c8a into minio:master Sep 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants