metacache: Always close block writer

[klauspost]

Description

In some cases a writer could be left behind unclosed, leaking compression blocks.

Always close and set compression concurrency to 2 which should be fine to keep up.

How to test this PR?

After serving list operations, servers would have leaked goroutines:

# 0x139a384 github.com/klauspost/compress/s2.(*Writer).write.func1+0x284 github.com/klauspost/compress@v1.10.3/s2/encode.go:586

Types of changes

• Bug fix (non-breaking change which fixes an issue)

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

10973-50267bf/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev8.minio.io:32295
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp a1643fbfbd00:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 10 seconds
(4/15) Running aws-sdk-ruby tests ... done in 20 seconds
(5/15) Running awscli tests ... done in 3 minutes and 57 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 5 minutes and 13 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 41 seconds
(9/15) Running minio-go tests ... done in 6 minutes and 32 seconds
(10/15) Running minio-java tests ... FAILED in 49 seconds
{
  "name": "minio-java",
  "function": "downloadObject()",
  "args": "[single upload with multiple path segments]",
  "duration": 640,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = InternalError, message = We encountered an internal error, please try again., bucketName = minio-java-test-1gcgqv2, objectName = path/to/minio-java-test-rj23v9, resource = /minio-java-test-1gcgqv2/path/to/minio-java-test-rj23v9, requestId = 164AB9A8A1474785, hostId = 42b453e7-104e-4237-8691-ee61876591c3)\nrequest={method=GET, url=http://minio-dev8.minio.io:32295/minio-java-test-1gcgqv2/path/to/minio-java-test-rj23v9, headers=Host: minio-dev8.minio.io:32295\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/7.1.4\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20201125T103231Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20201125/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=500, headers=Accept-Ranges: bytes\nContent-Length: 407\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nETag: \"03da054f179eed855c27d7d6b2041e2a\"\nLast-Modified: Wed, 25 Nov 2020 10:32:31 GMT\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 164AB9A8A1474785\nX-Cache: HIT\nX-Cache-Lookup: HIT\nX-Xss-Protection: 1; mode=block\nDate: Wed, 25 Nov 2020 10:32:31 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1135), io.minio.MinioClient.execute(MinioClient.java:935), io.minio.MinioClient.executeGet(MinioClient.java:1194), io.minio.MinioClient.getObject(MinioClient.java:1697), io.minio.MinioClient.downloadObject(MinioClient.java:1861), FunctionalTest.testDownloadObject(FunctionalTest.java:1302), FunctionalTest.downloadObject_test(FunctionalTest.java:1336), FunctionalTest.runTests(FunctionalTest.java:3794), FunctionalTest.main(FunctionalTest.java:4058)]"
}
(10/15) Running minio-js tests ... done in 2 minutes and 48 seconds
(11/15) Running minio-py tests ... done in 19 minutes and 6 seconds
(12/15) Running s3cmd tests ... done in 2 minutes and 15 seconds
(13/15) Running s3select tests ... FAILED in 3 seconds
{
  "name": "s3select:test_csv_input_quote_char",
  "function": "select_object_content(bucket_name, object_name, opts)",
  "args": {
    "bucket_name": "s3select-test-12ade01d-d863-4f11-890b-0f5ad857e869"
  },
  "duration": 2471,
  "message": "Test test_2 unexpectedly failed with: 'event-type'",
  "error": "Traceback (most recent call last):\n  File \"/mint/run/core/s3select/csv.py\", line 39, in test_sql_api\n    for d in data.stream(10*1024):\n  File \"/usr/local/lib/python3.6/dist-packages/minio/select/reader.py\", line 211, in stream\n    message = self.__extract_message()\n  File \"/usr/local/lib/python3.6/dist-packages/minio/select/reader.py\", line 161, in __extract_message\n    event_type = header_map[\"event-type\"]\nKeyError: 'event-type'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"./tests.py\", line 52, in main\n    test_csv_input_custom_quote_char(client, log_output)\n  File \"/mint/run/core/s3select/csv.py\", line 119, in test_csv_input_custom_quote_char\n    data, sql_opts, expected_output)\n  File \"/mint/run/core/s3select/csv.py\", line 45, in test_sql_api\n    'Test {} unexpectedly failed with: {}'.format(test_name, select_err))\nValueError: Test test_2 unexpectedly failed with: 'event-type'\n",
  "status": "FAIL"
}
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

[vadmeste]

LGTM