Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix possible crashes on deleteMarker replication #11308

Merged
merged 1 commit into from
Jan 20, 2021
Merged

fix possible crashes on deleteMarker replication #11308

merged 1 commit into from
Jan 20, 2021

Conversation

harshavardhana
Copy link
Member

Description

fix possible crashes on deleteMarker replication

Motivation and Context

Delete marker can have metaSys set to nil, that
can lead to crashes after the delete marker has
been healed.

Additionally also fix isObjectDangling check
for transitioned objects, that do not have parts
should be treated similar to Delete marker.

How to test this PR?

The crash is harder to reproduce as it requires
taking down servers and delete content -
letting it heal etc.

This PR also fixes a situation where transitioned
the object might be falsely considered as dangling.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation updated
  • Unit tests added/updated

@harshavardhana
fix possible crashes on deleteMarker replication
3c58a4d 
Delete marker can have `metaSys` set to nil, that
can lead to crashes after the delete marker has
been healed.

Additionally also fix isObjectDangling check
for transitioned objects, that do not have parts
should be treated similar to Delete marker.
poornas
poornas approved these changes Jan 20, 2021
View reviewed changes
Copy link
Contributor

@poornas poornas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-gateway-nas.sh ✔️
mint-gateway-azure.sh ❌ more...
mint-zoned.sh ❌ more...

11308-3c58a4d/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:31265
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 29f5272b3ee0:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 9 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... FAILED in 2 minutes and 47 seconds
{
  "name": "aws-sdk-php",
  "function": "getBucketPolicy ( array $params = [] )",
  "args": {
    "Bucket": "aws-sdk-php-6007a5f1e520c"
  },
  "duration": "40870",
  "status": "FAIL",
  "error": "Error executing \"CreateBucket\" on \"http://minio-c3.minio.io:31265/aws-sdk-php-6007a5f1e520c\"; AWS HTTP error: Client error: `PUT http://minio-c3.minio.io:31265/aws-sdk-php-6007a5f1e520c` resulted in a `404 Not Found` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Mes (truncated...)\n NoSuchBucket (client): The specified bucket does not exist - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>aws-sdk-php-6007a5f1e520c</BucketName><Resource>/aws-sdk-php-6007a5f1e520c</Resource><RequestId>165BD3C53AAF4A39</RequestId><HostId>988451de-b0ca-4c07-8d17-c70b9a1815d9</HostId></Error>"
}
(3/15) Running aws-sdk-ruby tests ... done in 21 seconds
(4/15) Running awscli tests ... done in 2 minutes and 51 seconds
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 3 minutes and 59 seconds
(7/15) Running minio-dotnet tests ... done in 1 minutes and 47 seconds
(8/15) Running minio-go tests ... done in 6 minutes and 53 seconds
(9/15) Running minio-java tests ... FAILED in 9 minutes and 19 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 180,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:f48029cd-701e-00dd-7fe1-eebe40000000\nTime:2021-01-20T04:07:25.9231846Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'RGsmDS6SH0jo8SucLN37YqAK5MCIiloDaFnP/cBTbu4=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:8d5f35a0-3068-4eb0-43fd-f6d2115e9af7\nx-ms-date:Wed, 20 Jan 2021 04:07:25 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:商å“�\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-1nt51kg/minio-java-test-2j5mda4\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-1nt51kg/minio-java-test-2j5mda4?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2021-01-20T03:33:00Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [8d5f35a0-3068-4eb0-43fd-f6d2115e9af7]\n   X-Ms-Date: [Wed, 20 Jan 2021 04:07:25 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1092]\n   Content-Type: [application/xml]\n   Date: [Wed, 20 Jan 2021 04:07:25 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [f48029cd-701e-00dd-7fe1-eebe40000000]\n\n\n, bucketName = minio-java-test-1nt51kg, objectName = minio-java-test-2j5mda4, resource = /minio-java-test-1nt51kg/minio-java-test-2j5mda4, requestId = 165BD524D09358BC, hostId = 988451de-b0ca-4c07-8d17-c70b9a1815d9)\nrequest={method=PUT, url=http://minio-c3.minio.io:31265/minio-java-test-1nt51kg/minio-java-test-2j5mda4, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-c3.minio.io:31265\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20210120T040725Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210120/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3086\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 165BD524D09358BC\nX-Xss-Protection: 1; mode=block\nDate: Wed, 20 Jan 2021 04:07:25 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(9/15) Running minio-js tests ... done in 2 minutes and 49 seconds
(10/15) Running minio-py tests ... done in 19 minutes and 2 seconds
(11/15) Running s3cmd tests ... done in 2 minutes and 23 seconds
(12/15) Running s3select tests ... done in 1 minutes and 1 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

11308-3c58a4d/mint-zoned.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:30901
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp b3ee8ef6a80d:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 1 minutes and 22 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 47 seconds
(8/15) Running minio-dotnet tests ... done in 57 seconds
(9/15) Running minio-go tests ... done in 2 minutes and 21 seconds
(10/15) Running minio-java tests ... done in 1 minutes and 27 seconds
(11/15) Running minio-js tests ... done in 56 seconds
(12/15) Running minio-py tests ... done in 3 minutes and 19 seconds
(13/15) Running s3cmd tests ... FAILED in 21 seconds
{
  "name": "s3cmd",
  "duration": "10012",
  "function": "test_sync_list_objects",
  "status": "FAIL",
  "error": "WARNING: Bucket is not empty. Removing all the objects from it first. This may take some time...\nERROR: S3 error: 404 (Not Found)"
}
(13/15) Running s3select tests ... done in 10 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

klauspost
klauspost approved these changes Jan 20, 2021
View reviewed changes
Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana harshavardhana merged commit d1a8f0b into minio:master Jan 20, 2021
@harshavardhana harshavardhana deleted the fix-possible branch January 20, 2021 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@poornas poornas poornas approved these changes

@klauspost klauspost klauspost approved these changes

@vadmeste vadmeste Awaiting requested review from vadmeste

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@harshavardhana @minio-trusted @poornas @klauspost