Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor allocation improvements #12133

Merged
merged 15 commits into from
May 7, 2021

Conversation

klauspost
Copy link
Contributor

@klauspost klauspost commented Apr 23, 2021

Description

Minor improvements to allocations and allow multiple keys to be deleted in one go.

Types of changes

  • Optimization (provides speedup with no functional changes)

Signed-off-by: Klaus Post klauspost@gmail.com

@harshavardhana harshavardhana changed the base branch from master to license-changes April 23, 2021 17:54
@harshavardhana harshavardhana force-pushed the license-changes branch 2 times, most recently from d84dba5 to cbfdf97 Compare April 23, 2021 18:59
@harshavardhana harshavardhana changed the base branch from license-changes to master April 23, 2021 19:56
cmd/xl-storage-format-utils.go Outdated Show resolved Hide resolved
@klauspost klauspost force-pushed the reduce-xlmetav2-allocs branch 2 times, most recently from 767e8a3 to 17f0892 Compare April 26, 2021 13:01
Add pool for short-lived usage of xlMetaV2 objects.

Reduces overall allocations.

Signed-off-by: Klaus Post <klauspost@gmail.com>
Signed-off-by: Klaus Post <klauspost@gmail.com>
# Conflicts:
#	cmd/storage-rest-server.go

Signed-off-by: Klaus Post <klauspost@gmail.com>
Signed-off-by: Klaus Post <klauspost@gmail.com>
This reverts commit cca6ada.

Signed-off-by: Klaus Post <klauspost@gmail.com>
Signed-off-by: Klaus Post <klauspost@gmail.com>
Signed-off-by: Klaus Post <klauspost@gmail.com>
This reverts commit 2138931.

Always use clean (baseline)

Signed-off-by: Klaus Post <klauspost@gmail.com>
Signed-off-by: Klaus Post <klauspost@gmail.com>
Signed-off-by: Klaus Post <klauspost@gmail.com>
cmd/xl-storage-format-v2.go Outdated Show resolved Hide resolved
@klauspost klauspost changed the title Add xlMetaV2Pool for short lived metadata Minor allocation improvements Apr 29, 2021
Copy link
Member

@vadmeste vadmeste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

cmd/xl-storage-format-utils.go Show resolved Hide resolved
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-compress-encrypt-dist-erasure.sh more...

12133-61add7b/mint-compress-encrypt-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:31109
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 8ec652231f0c:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... FAILED in 31 seconds
{
  "name": "awscli",
  "duration": 2937,
  "function": "aws --endpoint-url http://minio-dev7.minio.io:31109 s3api copy-object --bucket awscli-mint-test-bucket-19922 --key datafile-1-kB-copy --copy-source awscli-mint-test-bucket-19922/datafile-1-kB\n",
  "status": "FAIL",
  "error": "Hash mismatch expected 084e1383b70fb0c51acc680fef370023, got ac57de7156d7fc25ac1a65f81fa3989b"
}
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 43 seconds
(7/15) Running minio-dotnet tests ... done in 49 seconds
(8/15) Running minio-go tests ... FAILED in 2 minutes and 15 seconds
{
  "args": {},
  "duration": 824,
  "error": "At least one of the pre-conditions you specified did not hold",
  "function": "CopyObjectPart(destination, source)",
  "message": "CopyObjectPart call failed",
  "name": "minio-go: testUnencryptedToSSES3CopyObjectPart",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... FAILED in 2 minutes and 10 seconds
{
  "name": "minio-java",
  "function": "copyObject()",
  "args": "[match etag]",
  "duration": 104,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = PreconditionFailed, message = At least one of the pre-conditions you specified did not hold, bucketName = minio-java-test-35huit1, objectName = minio-java-test-23e12gc-copy, resource = /minio-java-test-35huit1/minio-java-test-23e12gc-copy, requestId = 167CC81796ADE7A4, hostId = 56c7a866-edc1-40d5-bc86-9a8ab28ba5c4)\nrequest={method=PUT, url=http://minio-dev7.minio.io:31109/minio-java-test-35huit1/minio-java-test-23e12gc-copy, headers=x-amz-copy-source-if-match: 71cff0a060f852067e443ad1e24ae26c-1\nx-amz-copy-source: /minio-java-test-3jk0v80/minio-java-test-23e12gc\nHost: minio-dev7.minio.io:31109\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20210507T121929Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210507/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=412, headers=Accept-Ranges: bytes\nContent-Length: 418\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nETag: \"71cff0a060f852067e443ad1e24ae26c\"\nLast-Modified: Fri, 07 May 2021 12:19:29 GMT\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 167CC81796ADE7A4\nX-Xss-Protection: 1; mode=block\nDate: Fri, 07 May 2021 12:19:29 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.execute(MinioClient.java:563), io.minio.MinioClient.executePut(MinioClient.java:904), io.minio.MinioClient.copyObject(MinioClient.java:1232), FunctionalTest.testCopyObjectMatchETag(FunctionalTest.java:1850), FunctionalTest.copyObject(FunctionalTest.java:2016), FunctionalTest.runObjectTests(FunctionalTest.java:3757), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(8/15) Running minio-js tests ... done in 45 seconds
(9/15) Running minio-py tests ... done in 2 minutes and 51 seconds
(10/15) Running s3cmd tests ... FAILED in 5 seconds
{
  "name": "s3cmd",
  "duration": "2600",
  "function": "test_put_object_multipart",
  "status": "FAIL",
  "error": "WARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Too many failures. Giving up on '/mint/data/datafile-65-MB'\nERROR: \nUpload of '/mint/data/datafile-65-MB' part 1 failed. Use\n  /usr/local/bin/s3cmd abortmp s3://s3cmd-test-bucket-13919/s3cmd-test-object-2775 3580e442-3886-4c27-90b8-9b8f17540ddc\nto abort the upload, or\n  /usr/local/bin/s3cmd --upload-id 3580e442-3886-4c27-90b8-9b8f17540ddc put ...\nto continue the upload.\nERROR: Upload of '/mint/data/datafile-65-MB' failed too many times (Last reason: )"
}
(10/15) Running s3select tests ... done in 7 seconds
(11/15) Running security tests ... done in 0 seconds

Executed 11 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana harshavardhana merged commit 254698f into minio:master May 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants