Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

preserve context per request for local locks #9828

Merged
merged 1 commit into from
Jun 14, 2020
Merged

preserve context per request for local locks #9828

merged 1 commit into from
Jun 14, 2020

Conversation

harshavardhana
Copy link
Member

@harshavardhana harshavardhana commented Jun 14, 2020 •
edited

Description

In the current implementation we were re-using
the context from previously granted lockers, this would
lead to lock timeouts for existing valid read or write locks,
leading to prematurely timeout the locks.

This bug affects only local lockers in FS
or standalone erasure-coded mode. This issue
is rather historical as well and was present
in lsync for some time but we were lucky to
not see it.

Similar changes are done in dsync as well
to keep the code more familiar

Motivation and Context

Fixes #9827

How to test this PR?

As per instructions in #9827

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation needed
  • Unit tests needed
  • Functional tests needed (If yes, add mint PR # here: )

@harshavardhana
preserve context per request for local locks
eb3e305 
Current bug was we were re-using the context
from previously granted lockers, this would
lead to lock timeouts for existing valid
read or write locks, leading to premature
timeout of locks.

This bug affects only local lockers in FS
or standalone erasure coded mode. This issue
is rather historical as well and was present
in lsync for some time but we were lucky to
not see it.

Similar changes are done in dsync as well
to keep the code more familiar

Fixes minio#9827
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-gateway-azure.sh ✔️
mint-gateway-nas.sh ✔️
mint-xl.sh ❌ more...
mint-large-bucket.sh ❌ more...
mint-dist-xl.sh ❌ more...

9828-eb3e305/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:31680
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 23dbe8306ad1:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 0 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 42 seconds
(4/15) Running aws-sdk-ruby tests ... done in 2 seconds
(5/15) Running awscli tests ... done in 1 minutes and 20 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 29 seconds
(8/15) Running minio-dotnet tests ... done in 32 seconds
(9/15) Running minio-go tests ... FAILED in 58 seconds
{
  "args": {
    "bucketName": "minio-go-test-sgxahtb1uasxt6g6",
    "objectName": "gg4rxc5osgcro6rsx9axdlfxtn1h1a",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 7488,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 52 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 139,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-h1jq3l, objectName = null, resource = /minio-java-test-h1jq3l, requestId = 16184833A29BD4E6, hostId = e5c5d085-b628-4a1c-a614-314423f0348c)\nrequest={method=DELETE, url=http://minio-dev7.minio.io:31680/minio-java-test-h1jq3l, headers=Host: minio-dev7.minio.io:31680\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200614T023054Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200614/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 371\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-14T02-20-05Z\nVary: Origin\nX-Amz-Request-Id: 16184833A29BD4E6\nX-Xss-Protection: 1; mode=block\nDate: Sun, 14 Jun 2020 02:30:54 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 50 seconds
(10/15) Running minio-py tests ... done in 1 minutes and 15 seconds
(11/15) Running s3cmd tests ... done in 19 seconds
(12/15) Running s3select tests ... done in 3 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9828-eb3e305/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:30771
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 1d0d11281769:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 5 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 47 seconds
(4/15) Running aws-sdk-ruby tests ... done in 9 seconds
(5/15) Running awscli tests ... done in 1 minutes and 39 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 1 minutes and 25 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 11 seconds
(9/15) Running minio-go tests ... FAILED in 2 minutes and 25 seconds
{
  "args": {
    "bucketName": "minio-go-test-bwn9q1l23w5mlxe0",
    "objectName": "iz1ta5opx093izynrew1yvt4zea13z",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 6287,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 7 minutes and 11 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 1021,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-3k5l6pc, objectName = null, resource = /minio-java-test-3k5l6pc, requestId = 161848BEF3CD004B, hostId = 45731b6f-a910-4dc2-bca7-77fb4b5eeef6)\nrequest={method=DELETE, url=http://minio-dev6.minio.io:30771/minio-java-test-3k5l6pc, headers=Host: minio-dev6.minio.io:30771\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200614T024053Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200614/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-14T02-20-05Z\nVary: Origin\nX-Amz-Request-Id: 161848BEF3CD004B\nX-Xss-Protection: 1; mode=block\nDate: Sun, 14 Jun 2020 02:40:53 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 56 seconds
(10/15) Running minio-py tests ... done in 5 minutes and 7 seconds
(11/15) Running s3cmd tests ... done in 27 seconds
(12/15) Running s3select tests ... done in 14 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9828-eb3e305/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:31510
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp a6685244ef7a:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 1 minutes and 25 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 36 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 6 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 16 seconds
{
  "args": {
    "bucketName": "minio-go-test-5elwjxnbdl1gxb49",
    "objectName": "hn4qt292og3adwk5sw03ng4ytocd2k",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 4301,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 3 minutes and 3 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 248,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-22bh4a6, objectName = null, resource = /minio-java-test-22bh4a6, requestId = 16184861FF1EA4A5, hostId = 2f0055fb-1db7-429d-ad97-71547e4d0b36)\nrequest={method=DELETE, url=http://minio-dev6.minio.io:31510/minio-java-test-22bh4a6, headers=Host: minio-dev6.minio.io:31510\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200614T023414Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200614/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-14T02-20-05Z\nVary: Origin\nX-Amz-Request-Id: 16184861FF1EA4A5\nX-Xss-Protection: 1; mode=block\nDate: Sun, 14 Jun 2020 02:34:14 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 48 seconds
(10/15) Running minio-py tests ... done in 1 minutes and 47 seconds
(11/15) Running s3cmd tests ... done in 28 seconds
(12/15) Running s3select tests ... done in 6 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

klauspost
klauspost approved these changes Jun 14, 2020
View reviewed changes
Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@harshavardhana harshavardhana merged commit d55f433 into minio:master Jun 14, 2020
@harshavardhana harshavardhana deleted the fix-lsync branch June 14, 2020 14:43
harshavardhana added a commit that referenced this pull request Jun 14, 2020
@harshavardhana
preserve context per request for local locks (#9828)
068ad33 
In the Current bug we were re-using the context
from previously granted lockers, this would
lead to lock timeouts for existing valid
read or write locks, leading to premature
timeout of locks.

This bug affects only local lockers in FS
or standalone erasure coded mode. This issue
is rather historical as well and was present
in lsync for some time but we were lucky to
not see it.

Similar changes are done in dsync as well
to keep the code more familiar

Fixes #9827
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@klauspost klauspost klauspost approved these changes

@kannappanr kannappanr kannappanr approved these changes

@krishnasrinivas krishnasrinivas Awaiting requested review from krishnasrinivas

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Copy fails for files larger than 2GB
4 participants
@harshavardhana @minio-trusted @klauspost @kannappanr