Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't copy interesting ids, clean up logging #11102

Merged
merged 2 commits into from
Dec 14, 2020

Conversation

klauspost
Copy link
Contributor

@klauspost klauspost commented Dec 14, 2020

Description

When searching the caches don't copy the ids, instead inline the loop.

Benchmark_bucketMetacache_findCache-32    	   19200	     63490 ns/op	    8303 B/op	       5 allocs/op
Benchmark_bucketMetacache_findCache-32    	   20338	     58609 ns/op	     111 B/op	       4 allocs/op

Add a reasonable, but still simplistic benchmark.

Motivation and Context

Generate less garbage.
Make logging look nicer.

Replaces #11058

How to test this PR?

Types of changes

  • Optimization

When searching the caches don't copy the ids, instead inline the loop.

```
Benchmark_bucketMetacache_findCache-32    	   19200	     63490 ns/op	    8303 B/op	       5 allocs/op
Benchmark_bucketMetacache_findCache-32    	   20338	     58609 ns/op	     111 B/op	       4 allocs/op
```

Add a reasonable, but still simplistic benchmark.
@klauspost klauspost changed the title Don't copy interesting ids Don't copy interesting ids, clean up logging Dec 14, 2020
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-gateway-azure.sh more...

11102-fde0d90/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:31400
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 8cfabd430a4b:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 9 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 2 minutes and 37 seconds
(4/15) Running aws-sdk-ruby tests ... done in 21 seconds
(5/15) Running awscli tests ... done in 2 minutes and 55 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 3 minutes and 31 seconds
(8/15) Running minio-dotnet tests ... done in 2 minutes and 45 seconds
(9/15) Running minio-go tests ... done in 8 minutes and 36 seconds
(10/15) Running minio-java tests ... FAILED in 9 minutes and 12 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 180,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:bbb8480b-f01e-0162-2e17-d2ec63000000\nTime:2020-12-14T12:47:05.8992187Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'hSX8HdVyRIOoTYHmNJ7GYdee/ugJwxd/VqG38pvGzCo=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:496ba7d6-f6c3-47ad-5c40-b5b459f1fd87\nx-ms-date:Mon, 14 Dec 2020 12:47:05 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:商å“�\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-28oeggo/minio-java-test-3f0tijo\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-28oeggo/minio-java-test-3f0tijo?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2020-12-14T12:10:51Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [496ba7d6-f6c3-47ad-5c40-b5b459f1fd87]\n   X-Ms-Date: [Mon, 14 Dec 2020 12:47:05 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1092]\n   Content-Type: [application/xml]\n   Date: [Mon, 14 Dec 2020 12:47:05 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [bbb8480b-f01e-0162-2e17-d2ec63000000]\n\n\n, bucketName = minio-java-test-28oeggo, objectName = minio-java-test-3f0tijo, resource = /minio-java-test-28oeggo/minio-java-test-3f0tijo, requestId = 1650960777721368, hostId = d6f8f0c1-f117-4bb7-a1c7-b3d1dd56d3da)\nrequest={method=PUT, url=http://minio-c2.minio.io:31400/minio-java-test-28oeggo/minio-java-test-3f0tijo, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-c2.minio.io:31400\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20201214T124705Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20201214/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3086\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 1650960777721368\nX-Xss-Protection: 1; mode=block\nDate: Mon, 14 Dec 2020 12:47:05 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(10/15) Running minio-js tests ... done in 2 minutes and 43 seconds
(11/15) Running minio-py tests ... done in 18 minutes and 43 seconds
(12/15) Running s3cmd tests ... done in 2 minutes and 21 seconds
(13/15) Running s3select tests ... FAILED in 23 seconds
{
  "name": "s3select:test_csv_output_quote_char",
  "function": "select_object_content(bucket_name, object_name, request)",
  "args": {
    "bucket_name": "s3select-test-74587a12-ceda-42ff-b16b-d04ea9bd772f"
  },
  "duration": 6408,
  "message": "Test test_7 unexpectedly failed with: 'event-type'",
  "error": "Traceback (most recent call last):\n  File \"/mint/run/core/s3select/csv.py\", line 43, in test_sql_api\n    for d in data.stream(10*1024):\n  File \"/usr/local/lib/python3.6/dist-packages/minio/select/reader.py\", line 211, in stream\n    message = self.__extract_message()\n  File \"/usr/local/lib/python3.6/dist-packages/minio/select/reader.py\", line 161, in __extract_message\n    event_type = header_map[\"event-type\"]\nKeyError: 'event-type'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"./tests.py\", line 57, in main\n    test_csv_output_custom_quote_char(client, log_output)\n  File \"/mint/run/core/s3select/csv.py\", line 176, in test_csv_output_custom_quote_char\n    input_data, sql_opts, expected_output)\n  File \"/mint/run/core/s3select/csv.py\", line 49, in test_sql_api\n    'Test {} unexpectedly failed with: {}'.format(test_name, select_err))\nValueError: Test test_7 unexpectedly failed with: 'event-type'\n",
  "status": "FAIL"
}
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

Copy link
Contributor

@poornas poornas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana harshavardhana merged commit f6fb27e into minio:master Dec 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants