Allow optionally to disable range caching. #9908

poornas · 2020-06-24T03:19:42Z

The default behavior is to cache each range requested
to cache drive. Add a environment variable
MINIO_RANGE_CACHE - when set to off, it disables
range caching and instead downloads entire object
in the background.

Fixes #9870

Description

Motivation and Context

How to test this PR?

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed
Functional tests needed (If yes, add mint PR # here: )

docs/disk-caching/DESIGN.md

cmd/disk-cache.go

vadmeste

mc admin config get myminio/ cache does not seem to show the new parameter

cmd/disk-cache.go

harshavardhana · 2020-06-26T23:52:15Z

mc admin config get myminio/ cache does not seem to show the new parameter

This is because there is no Help entry for this new option // cc @poornas can you fix that?

harshavardhana

Help! needs to be improved

cmd/disk-cache.go

cmd/config/cache/help.go

cmd/config/errors.go

docs/disk-caching/DESIGN.md

The default behavior is to cache each range requested to cache drive. Add a environment variable `MINIO_RANGE_CACHE` - when set to off, it disables range caching and instead downloads entire object in the background. Fixes minio#9870

minio-trusted · 2020-06-29T02:28:42Z

Mint Automation

Test	Result
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-gateway-nas.sh	✔️
mint-xl.sh	❌ more...
mint-large-bucket.sh	❌ more...
mint-dist-xl.sh	❌ more...
mint-gateway-azure.sh	❌ more...

9908-fb919a6/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:31979
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp dd42ca6a5cab:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 4 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 3 seconds
(5/15) Running awscli tests ... done in 1 minutes and 20 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 26 seconds
(8/15) Running minio-dotnet tests ... done in 37 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 6 seconds
{
  "args": {
    "bucketName": "minio-go-test-633dzuubuky1nqq5",
    "objectName": "ade4ii991tu6r0vbetgk0lkcxwt3nx",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 3422,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 2 minutes and 46 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 145,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-2f299kv, objectName = null, resource = /minio-java-test-2f299kv, requestId = 161CDFF21E418C33, hostId = 05f87571-076d-47e6-81f1-4c502fb802d9)\nrequest={method=DELETE, url=http://minio-c3.minio.io:31979/minio-java-test-2f299kv, headers=Host: minio-c3.minio.io:31979\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200629T013639Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200629/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-29T01-23-37Z\nVary: Origin\nX-Amz-Request-Id: 161CDFF21E418C33\nX-Xss-Protection: 1; mode=block\nDate: Mon, 29 Jun 2020 01:36:39 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 47 seconds
(10/15) Running minio-py tests ... done in 4 minutes and 27 seconds
(11/15) Running s3cmd tests ... done in 22 seconds
(12/15) Running s3select tests ... done in 8 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9908-fb919a6/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev4.minio.io:30166
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp d2dc469f45f8:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 32 seconds
(2/15) Running aws-sdk-java tests ... done in 4 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 12 seconds
(4/15) Running aws-sdk-ruby tests ... done in 1 minutes and 2 seconds
(5/15) Running awscli tests ... done in 4 minutes and 21 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 1 minutes and 13 seconds
(8/15) Running minio-dotnet tests ... done in 4 minutes and 23 seconds
(9/15) Running minio-go tests ... FAILED in 5 minutes and 8 seconds
{
  "args": {
    "bucketName": "minio-go-test-n9fcincdw5gnvaya",
    "objectName": "fhvg5b14evrtag5pzftc5ekkk5k31w",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 8506,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 10 minutes and 2 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 4071,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-2bt55n6, objectName = null, resource = /minio-java-test-2bt55n6, requestId = 161CE114EF1AB04B, hostId = 4f131f5d-5b7d-4267-b257-ec4daa642570)\nrequest={method=DELETE, url=http://minio-dev4.minio.io:30166/minio-java-test-2bt55n6, headers=Host: minio-dev4.minio.io:30166\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200629T015728Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200629/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-29T01-23-37Z\nVary: Origin\nX-Amz-Request-Id: 161CE114EF1AB04B\nX-Xss-Protection: 1; mode=block\nDate: Mon, 29 Jun 2020 01:57:28 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 1 minutes and 17 seconds
(10/15) Running minio-py tests ... done in 13 minutes and 44 seconds
(11/15) Running s3cmd tests ... done in 40 seconds
(12/15) Running s3select tests ... done in 58 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9908-fb919a6/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:31262
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 00f1aa050552:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 7 seconds
(2/15) Running aws-sdk-java tests ... done in 3 seconds
(3/15) Running aws-sdk-php tests ... done in 2 minutes and 22 seconds
(4/15) Running aws-sdk-ruby tests ... done in 16 seconds
(5/15) Running awscli tests ... done in 2 minutes and 46 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 5 minutes and 10 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 34 seconds
(9/15) Running minio-go tests ... done in 7 minutes and 12 seconds
(10/15) Running minio-java tests ... done in 24 minutes and 59 seconds
(11/15) Running minio-js tests ... FAILED in 1 minutes and 31 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 84,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:673:10) at endReadableNT (_stream_readable.js:1201:12) at processTicksAndRejections (internal/process/task_queues.js:84:21)"
}
(11/15) Running minio-py tests ... done in 9 minutes and 54 seconds
(12/15) Running s3cmd tests ... done in 1 minutes and 33 seconds
(13/15) Running s3select tests ... done in 1 minutes and 4 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

9908-fb919a6/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:31965
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 442c4664648f:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 10 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 6 seconds
(5/15) Running awscli tests ... done in 2 minutes and 38 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 1 minutes and 8 seconds
(8/15) Running minio-dotnet tests ... done in 3 minutes and 6 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 57 seconds
{
  "args": {
    "bucketName": "minio-go-test-oyo0hac5uvqvd5h2",
    "objectName": "2yx4xzmilq5hrordevajz2nhlejvtc",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 4387,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 3 minutes and 38 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 371,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-3pm7mne, objectName = null, resource = /minio-java-test-3pm7mne, requestId = 161CE04AEBEFEF7B, hostId = f3cfaf9c-d041-4da1-9adf-e0fd4e21bd0b)\nrequest={method=DELETE, url=http://minio-c3.minio.io:31965/minio-java-test-3pm7mne, headers=Host: minio-c3.minio.io:31965\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200629T014300Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200629/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-29T01-23-37Z\nVary: Origin\nX-Amz-Request-Id: 161CE04AEBEFEF7B\nX-Xss-Protection: 1; mode=block\nDate: Mon, 29 Jun 2020 01:43:00 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 51 seconds
(10/15) Running minio-py tests ... done in 2 minutes and 26 seconds
(11/15) Running s3cmd tests ... done in 21 seconds
(12/15) Running s3select tests ... done in 6 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

vadmeste

LGTM & tested

The default behavior is to cache each range requested to cache drive. Add an environment variable `MINIO_RANGE_CACHE` - when set to off, it disables range caching and instead downloads entire object in the background. Fixes #9870

harshavardhana requested changes Jun 24, 2020

View reviewed changes

docs/disk-caching/DESIGN.md Outdated Show resolved Hide resolved

harshavardhana requested changes Jun 24, 2020

View reviewed changes

cmd/disk-cache.go Show resolved Hide resolved

poornas force-pushed the disableRange branch from 6a76044 to 0d69cd5 Compare June 24, 2020 05:04

harshavardhana approved these changes Jun 24, 2020

View reviewed changes

kannappanr requested a review from vadmeste June 24, 2020 14:49

vadmeste reviewed Jun 26, 2020

View reviewed changes

cmd/disk-cache.go Show resolved Hide resolved

harshavardhana requested changes Jun 27, 2020

View reviewed changes

poornas force-pushed the disableRange branch from 3264756 to fe610cb Compare June 27, 2020 15:13

harshavardhana requested changes Jun 28, 2020

View reviewed changes

cmd/disk-cache.go Outdated Show resolved Hide resolved

cmd/config/cache/help.go Outdated Show resolved Hide resolved

cmd/config/errors.go Outdated Show resolved Hide resolved

docs/disk-caching/DESIGN.md Outdated Show resolved Hide resolved

Allow optionally to disable range caching.

fb919a6

The default behavior is to cache each range requested to cache drive. Add a environment variable `MINIO_RANGE_CACHE` - when set to off, it disables range caching and instead downloads entire object in the background. Fixes minio#9870

poornas force-pushed the disableRange branch from fe610cb to fb919a6 Compare June 29, 2020 01:23

harshavardhana approved these changes Jun 29, 2020

View reviewed changes

vadmeste approved these changes Jun 29, 2020

View reviewed changes

harshavardhana merged commit 55a3b07 into minio:master Jun 29, 2020

poornas deleted the disableRange branch June 13, 2022 22:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow optionally to disable range caching. #9908

Allow optionally to disable range caching. #9908

poornas commented Jun 24, 2020

vadmeste left a comment

harshavardhana commented Jun 26, 2020

harshavardhana left a comment

minio-trusted commented Jun 29, 2020

vadmeste left a comment

Allow optionally to disable range caching. #9908

Allow optionally to disable range caching. #9908

Conversation

poornas commented Jun 24, 2020

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

vadmeste left a comment

Choose a reason for hiding this comment

harshavardhana commented Jun 26, 2020

harshavardhana left a comment

Choose a reason for hiding this comment

minio-trusted commented Jun 29, 2020

Mint Automation

9908-fb919a6/mint-xl.sh.log:

9908-fb919a6/mint-large-bucket.sh.log:

9908-fb919a6/mint-gateway-azure.sh.log:

9908-fb919a6/mint-dist-xl.sh.log:

vadmeste left a comment

Choose a reason for hiding this comment