Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update S2 compression #11753

Merged
merged 1 commit into from
Mar 10, 2021
Merged

Conversation

klauspost
Copy link
Contributor

Description

Relevant updates:

Upgrade cpuid to match simdjson.

How to test this PR?

Existing tests should do checks.

Types of changes

  • Optimization (provides speedup with no functional changes)

Relevant updates:

* Less allocations on decode: klauspost/compress#322
* Fixed rare out-of-bounds write on amd64.
* ARM64 decompression assembly. Around 2x output speed. klauspost/compress#324
* Speed up decompression on non-assembly platforms. klauspost/compress#328

Upgrade cpuid to match simdjson.
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-large-bucket.sh more...
mint-compress-encrypt-dist-erasure.sh more...

11753-6c8f004/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev4.minio.io:30694
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp ec350ccbd450:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 18 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 1 minutes and 9 seconds
(8/15) Running minio-dotnet tests ... done in 52 seconds
(9/15) Running minio-go tests ... done in 2 minutes and 10 seconds
(10/15) Running minio-java tests ... done in 1 minutes and 21 seconds
(11/15) Running minio-js tests ... FAILED in 7 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 11,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:79:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (internal/streams/readable.js:684:10) at endReadableNT (internal/streams/readable.js:1327:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(11/15) Running minio-py tests ... done in 3 minutes and 17 seconds
(12/15) Running s3cmd tests ... done in 18 seconds
(13/15) Running s3select tests ... done in 6 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

11753-6c8f004/mint-compress-encrypt-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:31957
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp d629dae4dc78:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... FAILED in 33 seconds
{
  "name": "awscli",
  "duration": 2916,
  "function": "aws --endpoint-url http://minio-c3.minio.io:31957 s3api copy-object --bucket awscli-mint-test-bucket-12672 --key datafile-1-kB-copy --copy-source awscli-mint-test-bucket-12672/datafile-1-kB\n",
  "status": "FAIL",
  "error": "Hash mismatch expected 084e1383b70fb0c51acc680fef370023, got ac57de7156d7fc25ac1a65f81fa3989b"
}
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 48 seconds
(7/15) Running minio-dotnet tests ... done in 45 seconds
(8/15) Running minio-go tests ... FAILED in 2 minutes and 28 seconds
{
  "args": {},
  "duration": 546,
  "error": "At least one of the pre-conditions you specified did not hold",
  "function": "CopyObjectPart(destination, source)",
  "message": "CopyObjectPart call failed",
  "name": "minio-go: testUnencryptedToSSES3CopyObjectPart",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... FAILED in 1 minutes and 55 seconds
{
  "name": "minio-java",
  "function": "copyObject()",
  "args": "[match etag]",
  "duration": 238,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = PreconditionFailed, message = At least one of the pre-conditions you specified did not hold, bucketName = minio-java-test-3m1qch8, objectName = minio-java-test-1tokune-copy, resource = /minio-java-test-3m1qch8/minio-java-test-1tokune-copy, requestId = 166AF086CBA60DE6, hostId = 7c7ad6e0-5b92-44ea-8920-1ea9f8c00c48)\nrequest={method=PUT, url=http://minio-c3.minio.io:31957/minio-java-test-3m1qch8/minio-java-test-1tokune-copy, headers=x-amz-copy-source-if-match: 71cff0a060f852067e443ad1e24ae26c-1\nx-amz-copy-source: /minio-java-test-g1pbsp/minio-java-test-1tokune\nHost: minio-c3.minio.io:31957\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20210310T091758Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210310/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=412, headers=Accept-Ranges: bytes\nContent-Length: 418\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nETag: \"71cff0a060f852067e443ad1e24ae26c\"\nLast-Modified: Wed, 10 Mar 2021 09:17:58 GMT\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 166AF086CBA60DE6\nX-Xss-Protection: 1; mode=block\nDate: Wed, 10 Mar 2021 09:17:58 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.execute(MinioClient.java:563), io.minio.MinioClient.executePut(MinioClient.java:904), io.minio.MinioClient.copyObject(MinioClient.java:1232), FunctionalTest.testCopyObjectMatchETag(FunctionalTest.java:1850), FunctionalTest.copyObject(FunctionalTest.java:2016), FunctionalTest.runObjectTests(FunctionalTest.java:3757), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(8/15) Running minio-js tests ... done in 48 seconds
(9/15) Running minio-py tests ... done in 3 minutes and 24 seconds
(10/15) Running s3cmd tests ... FAILED in 6 seconds
{
  "name": "s3cmd",
  "duration": "2790",
  "function": "test_put_object_multipart",
  "status": "FAIL",
  "error": "WARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Too many failures. Giving up on '/mint/data/datafile-65-MB'\nERROR: \nUpload of '/mint/data/datafile-65-MB' part 1 failed. Use\n  /usr/local/bin/s3cmd abortmp s3://s3cmd-test-bucket-20799/s3cmd-test-object-25679 17bda53d-b1df-4182-a9c0-71d546af4851\nto abort the upload, or\n  /usr/local/bin/s3cmd --upload-id 17bda53d-b1df-4182-a9c0-71d546af4851 put ...\nto continue the upload.\nERROR: Upload of '/mint/data/datafile-65-MB' failed too many times (Last reason: )"
}
(10/15) Running s3select tests ... done in 6 seconds
(11/15) Running security tests ... done in 0 seconds

Executed 11 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana harshavardhana merged commit 952b0f1 into minio:master Mar 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants