Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: [fs] CompleteMultipart use trie structure for prefixMatch #10522

Merged
merged 1 commit into from Sep 21, 2020

Conversation

harshavardhana
Copy link
Member

Description

fix: [fs] CompleteMultipart use trie structure for partMatch

Motivation and Context

performance improves by around 100x or more

go test -v -run NONE -bench BenchmarkGetPartFile
goos: linux
goarch: amd64
pkg: github.com/minio/minio/cmd
BenchmarkGetPartFileWithTrie
BenchmarkGetPartFileWithTrie-4          1000000000               0.140 ns/op           0 B/op          0 allocs/op
PASS
ok      github.com/minio/minio/cmd      1.737s

fixes #10520

How to test this PR?

As per #10520 - the issue can be seen by re-using the older code with the same Benchmark

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation needed
  • Unit tests needed

performance improves by around 100x or more

```
go test -v -run NONE -bench BenchmarkGetPartFile
goos: linux
goarch: amd64
pkg: github.com/minio/minio/cmd
BenchmarkGetPartFileWithTrie
BenchmarkGetPartFileWithTrie-4          1000000000               0.140 ns/op           0 B/op          0 allocs/op
PASS
ok      github.com/minio/minio/cmd      1.737s
```

fixes minio#10520
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-gateway-azure.sh more...

10522-6111661/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev8.minio.io:32097
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp f8d42304b16b:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 9 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 16 seconds
(4/15) Running aws-sdk-ruby tests ... done in 20 seconds
(5/15) Running awscli tests ... done in 2 minutes and 50 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 42 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 38 seconds
(9/15) Running minio-go tests ... FAILED in 5 minutes and 10 seconds
{
  "args": {
    "bucketName": "minio-go-test-nj221lhqokyes9dy",
    "objectName": "9thpignj2gquli4w2i4bkqgqvoarop"
  },
  "duration": 719,
  "function": "GetObject(bucketName, objectName)",
  "message": "Incorrect read bytes v/s original buffer",
  "name": "minio-go: testGetObjectReadSeekFunctional",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 12 minutes and 43 seconds
(10/15) Running minio-js tests ... done in 2 minutes and 35 seconds
(11/15) Running minio-py tests ... done in 19 minutes and 17 seconds
(12/15) Running s3cmd tests ... done in 2 minutes and 20 seconds
(13/15) Running s3select tests ... done in 1 minutes and 5 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally
Error: No such image: minio/minio:10522-6111661

Copy link
Contributor

@kannappanr kannappanr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana harshavardhana merged commit 3831cc9 into minio:master Sep 21, 2020
@harshavardhana harshavardhana deleted the fix-fs branch September 21, 2020 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FS] CompleteMultipartUpload uses getPartFile to iterate over parts of a multipart upload, which is slow
4 participants