Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: legacy objects with 10MiB blockSize should use right buffers #12459

Merged
merged 1 commit into from Jun 7, 2021

Conversation

harshavardhana
Copy link
Member

Description

fix: legacy objects with 10MiB blockSize should use right buffers

Motivation and Context

healing code was using incorrect buffers to heal older
objects with 10MiB erasure blockSize, incorrect calculation
of such buffers can lead to incorrect premature closure of
io.Pipe() during healing.

fixes #12410

How to test this PR?

Just upgrade from an older 2020 release and run mc admin heal -r on those objects.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation updated
  • Unit tests added/updated

Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

healing code was using incorrect buffers to heal older
objects with 10MiB erasure blockSize, incorrect calculation
of such buffers can lead to incorrect premature closure of
io.Pipe() during healing.

fixes minio#12410
Copy link
Contributor

@kannappanr kannappanr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-fs.sh more...
mint-compress-encrypt-dist-erasure.sh more...

12459-f28d746/mint-compress-encrypt-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev8.minio.io:30442
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp ced48a39c55b:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 11 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 1 minutes and 13 seconds
(8/15) Running minio-dotnet tests ... done in 45 seconds
(9/15) Running minio-go tests ... done in 2 minutes and 5 seconds
(10/15) Running minio-java tests ... FAILED in 1 minutes and 22 seconds
{
  "name": "minio-java",
  "function": "composeObject()",
  "args": "[single source with offset]",
  "duration": 70,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = InvalidArgument, message = Range specified is not valid for source object, bucketName = minio-java-test-2imb1dt, objectName = minio-java-test-37cai5r, resource = /minio-java-test-2imb1dt/minio-java-test-37cai5r, requestId = 16865855ED7E6BDF, hostId = d4329304-3111-44fb-a9f9-dd33dec14780)\nrequest={method=PUT, url=http://minio-dev8.minio.io:30442/minio-java-test-2imb1dt/minio-java-test-37cai5r?uploadId=0c9e95c6-e209-4a19-abbc-61eb7c88abee&partNumber=1, headers=x-amz-copy-source: /minio-java-test-2imb1dt/minio-java-test-36tlrop\nx-amz-copy-source-range: bytes=2048-1048575\nx-amz-copy-source-if-match: cb92d17a904ccec2e6e23b8bb66245fb\nHost: minio-dev8.minio.io:30442\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20210607T160401Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210607/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-copy-source-range;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=400, headers=Accept-Ranges: bytes\nContent-Length: 390\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 16865855ED7E6BDF\nX-Xss-Protection: 1; mode=block\nDate: Mon, 07 Jun 2021 16:04:01 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.uploadPartCopy(MinioClient.java:4804), io.minio.MinioClient.composeObject(MinioClient.java:1431), FunctionalTest.testComposeObject(FunctionalTest.java:2120), FunctionalTest.composeObjectTests(FunctionalTest.java:2145), FunctionalTest.composeObject(FunctionalTest.java:2300), FunctionalTest.runObjectTests(FunctionalTest.java:3758), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(10/15) Running minio-js tests ... done in 47 seconds
(11/15) Running minio-py tests ... done in 2 minutes and 49 seconds
(12/15) Running s3cmd tests ... done in 18 seconds
(13/15) Running s3select tests ... done in 6 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

12459-f28d746/mint-fs.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev4.minio.io:31185
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 01bf09d12465:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 0 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 41 seconds
(4/15) Running aws-sdk-ruby tests ... done in 1 seconds
(5/15) Running awscli tests ... done in 1 minutes and 8 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 25 seconds
(8/15) Running minio-dotnet tests ... done in 27 seconds
(9/15) Running minio-go tests ... done in 1 minutes and 29 seconds
(10/15) Running minio-java tests ... done in 21 seconds
(11/15) Running minio-js tests ... FAILED in 14 seconds
{
  "name": "minio-js",
  "function": "fPutObject(bucketName, objectName, filePath, metaData)",
  "args": "bucketName:minio-js-test-ad7f84d2-eeaf-4171-b9e4-f4fe792c39c5, objectName:datafile-65-MB, filePath:/tmp/datafile-65-MB",
  "duration": 1609,
  "status": "FAIL",
  "error": "Error: write ECONNRESET at afterWriteDispatched (internal/stream_base_commons.js:156:25) at writeGeneric (internal/stream_base_commons.js:147:3) at Socket._writeGeneric (net.js:785:11) at Socket._write (net.js:797:8) at doWrite (internal/streams/writable.js:377:12) at clearBuffer (internal/streams/writable.js:529:7) at Socket.Writable.uncork (internal/streams/writable.js:317:7) at connectionCorkNT (_http_outgoing.js:733:8) at processTicksAndRejections (internal/process/task_queues.js:79:21)"
}
(11/15) Running minio-py tests ... done in 1 minutes and 30 seconds
(12/15) Running s3cmd tests ... done in 22 seconds
(13/15) Running s3select tests ... done in 3 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana harshavardhana merged commit 542fe4e into minio:master Jun 7, 2021
@harshavardhana harshavardhana deleted the fix-healing branch June 7, 2021 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error: io: read/write on closed pipe (*errors.errorString) in healing
4 participants