Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: improve performance ListObjectParts in FS mode #10510

Merged
merged 1 commit into from Sep 18, 2020

Conversation

harshavardhana
Copy link
Member

Description

fix: improve performance ListObjectParts in FS mode

Motivation and Context

from 20s for 10000 parts to less than 1sec

Without the patch

~ time aws --endpoint-url=http://localhost:9000 --profile minio s3api \
       list-parts --bucket testbucket --key test \
       --upload-id c1cd1f50-ea9a-4824-881c-63b5de95315a

real    0m20.394s
user    0m0.589s
sys     0m0.174s

With the patch

~ time aws --endpoint-url=http://localhost:9000 --profile minio s3api \
       list-parts --bucket testbucket --key test \
       --upload-id c1cd1f50-ea9a-4824-881c-63b5de95315a

real    0m0.891s
user    0m0.624s
sys     0m0.182s

fixes #10503

How to test this PR?

As per the reproducer in #10503

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation needed
  • Unit tests needed

from 20s for 10000 parts to less than 1sec

Without the patch
```
~ time aws --endpoint-url=http://localhost:9000 --profile minio s3api \
       list-parts --bucket testbucket --key test \
       --upload-id c1cd1f50-ea9a-4824-881c-63b5de95315a

real    0m20.394s
user    0m0.589s
sys     0m0.174s
```

With the patch
```
~ time aws --endpoint-url=http://localhost:9000 --profile minio s3api \
       list-parts --bucket testbucket --key test \
       --upload-id c1cd1f50-ea9a-4824-881c-63b5de95315a

real    0m0.891s
user    0m0.624s
sys     0m0.182s
```

fixes minio#10503
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-gateway-nas.sh ✔️
mint-gateway-azure.sh more...
mint-zoned.sh more...

10510-5d469f1/mint-zoned.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:31617
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 63d076bf2bb1:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 3 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 6 seconds
(5/15) Running awscli tests ... done in 2 minutes and 10 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 1 minutes and 14 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 2 seconds
(9/15) Running minio-go tests ... done in 2 minutes and 40 seconds
(10/15) Running minio-java tests ... done in 1 minutes and 43 seconds
(11/15) Running minio-js tests ... FAILED in 15 seconds
{
  "name": "minio-js",
  "function": "fPutObject(bucketName, objectName, filePath, metaData)",
  "args": "bucketName:minio-js-test-89dab03b-62b2-4ad7-bf59-bc7e2da1c83c, objectName:datafile-65-MB, filePath:/tmp/datafile-65-MB",
  "duration": 1997,
  "status": "FAIL",
  "error": "Error: read ECONNRESET at TCP.onStreamRead (internal/stream_base_commons.js:209:20)"
}
(11/15) Running minio-py tests ... done in 3 minutes and 17 seconds
(12/15) Running s3cmd tests ... done in 25 seconds
(13/15) Running s3select tests ... done in 8 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

10510-5d469f1/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:30289
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp ebfef5b165e7:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 7 seconds
(4/15) Running aws-sdk-ruby tests ... done in 20 seconds
(5/15) Running awscli tests ... done in 3 minutes and 52 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 51 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 42 seconds
(9/15) Running minio-go tests ... done in 6 minutes and 50 seconds
(10/15) Running minio-java tests ... done in 10 minutes and 55 seconds
(11/15) Running minio-js tests ... done in 3 minutes and 46 seconds
(12/15) Running minio-py tests ... FAILED in 18 minutes and 33 seconds
{
  "name": "minio-py:test_select_object_content",
  "status": "FAIL",
  "args": {
    "bucket_name": "minio-py-test-c2903607-e9f4-45de-9838-b147ecc625b6",
    "object_name": "test.csv"
  },
  "message": "'event-type'",
  "error": "Traceback (most recent call last):\n  File \"/mint/run/core/minio-py/tests.py\", line 147, in _call_test\n    func(log_entry, *args, **kwargs)\n  File \"/mint/run/core/minio-py/tests.py\", line 317, in test_select_object_content\n    for data_bytes in data.stream(10*KB):\n  File \"/usr/local/lib/python3.6/dist-packages/minio/select/reader.py\", line 211, in stream\n    message = self.__extract_message()\n  File \"/usr/local/lib/python3.6/dist-packages/minio/select/reader.py\", line 161, in __extract_message\n    event_type = header_map[\"event-type\"]\nKeyError: 'event-type'\n",
  "duration": 1006
}
(12/15) Running s3cmd tests ... done in 2 minutes and 21 seconds
(13/15) Running s3select tests ... done in 1 minutes and 5 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally
Error: No such image: minio/minio:10510-5d469f1

Copy link
Contributor

@kannappanr kannappanr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana harshavardhana merged commit 4a36cd7 into minio:master Sep 18, 2020
@harshavardhana harshavardhana deleted the list-parts branch September 18, 2020 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FS] listing parts of multipart upload that has many parts is very slow
4 participants