Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix incorrect http range in SelectObjectContentHandler #11585

Merged
merged 1 commit into from
Feb 20, 2021

Conversation

mailsmail
Copy link
Contributor

Description

This commit fix the HTTP range calculated in SelectObjectContentHandler.

Motivation and Context

Without this commit, there will be no functionality issue, select object content works just fine. The problem is that HTTP header Range is off by 1 byte. So, for example, when selecting parquet object, the reader need to request an object with start offset 0 and length 1024 Bytes. The expected range should be bytes=0-1023, but get bytes=0-1024 instead. Which means it is requested 1025 Bytes.

How to test this PR?

Run MinIO as S3 gateway and then select object content with parquet format. Then, inspect variable length here:

End: offset + length,

and here:

fn, off, length, err := minio.NewGetObjectReader(rs, objInfo, opts)

The value must be the same.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation updated
  • Unit tests added/updated

@mailsmail mailsmail marked this pull request as ready for review February 19, 2021 20:53
cmd/object-handlers.go Outdated Show resolved Hide resolved
@mailsmail mailsmail force-pushed the fix-http-range branch 2 times, most recently from 05c1765 to 3b25aa8 Compare February 19, 2021 22:25
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-compress-encrypt-dist-erasure.sh more...

11585-8ef10d7/mint-compress-encrypt-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:32514
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 002915220d91:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... FAILED in 33 seconds
{
  "name": "awscli",
  "duration": 2791,
  "function": "aws --endpoint-url http://minio-dev6.minio.io:32514 s3api copy-object --bucket awscli-mint-test-bucket-20019 --key datafile-1-kB-copy --copy-source awscli-mint-test-bucket-20019/datafile-1-kB\n",
  "status": "FAIL",
  "error": "Hash mismatch expected 084e1383b70fb0c51acc680fef370023, got ac57de7156d7fc25ac1a65f81fa3989b"
}
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 48 seconds
(7/15) Running minio-dotnet tests ... done in 52 seconds
(8/15) Running minio-go tests ... FAILED in 2 minutes and 24 seconds
{
  "args": {
    "destination": {
      "Bucket": "minio-go-test-wzqrtq5dn3ow4y34",
      "Object": "dstObject",
      "Encryption": {},
      "UserMetadata": null,
      "ReplaceMetadata": false,
      "UserTags": null,
      "ReplaceTags": false,
      "LegalHold": "",
      "Mode": "",
      "RetainUntilDate": "0001-01-01T00:00:00Z",
      "Size": 0,
      "Progress": null
    },
    "source": {
      "Bucket": "minio-go-test-wzqrtq5dn3ow4y34",
      "Object": "srcObject",
      "VersionID": "",
      "MatchETag": "",
      "NoMatchETag": "",
      "MatchModifiedSince": "0001-01-01T00:00:00Z",
      "MatchUnmodifiedSince": "0001-01-01T00:00:00Z",
      "MatchRange": false,
      "Start": 0,
      "End": 0,
      "Encryption": null
    }
  },
  "duration": 6540,
  "error": "We encountered an internal error, please try again.: cause(s2: corrupt input)",
  "function": "CopyObject(destination, source)",
  "message": "GetObject failed",
  "name": "minio-go: testUnencryptedToSSES3CopyObject",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... FAILED in 1 minutes and 43 seconds
{
  "name": "minio-java",
  "function": "copyObject()",
  "args": "[match etag]",
  "duration": 200,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = PreconditionFailed, message = At least one of the pre-conditions you specified did not hold, bucketName = minio-java-test-37dcntq, objectName = minio-java-test-21j2k39-copy, resource = /minio-java-test-37dcntq/minio-java-test-21j2k39-copy, requestId = 16654A38BEFE1EDF, hostId = bdef6f49-d9da-4546-973d-80e8a33ce5ff)\nrequest={method=PUT, url=http://minio-dev6.minio.io:32514/minio-java-test-37dcntq/minio-java-test-21j2k39-copy, headers=x-amz-copy-source-if-match: 71cff0a060f852067e443ad1e24ae26c-1\nx-amz-copy-source: /minio-java-test-1nkg1ff/minio-java-test-21j2k39\nHost: minio-dev6.minio.io:32514\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20210219T233409Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210219/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=412, headers=Accept-Ranges: bytes\nContent-Length: 418\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nETag: \"71cff0a060f852067e443ad1e24ae26c\"\nLast-Modified: Fri, 19 Feb 2021 23:34:08 GMT\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 16654A38BEFE1EDF\nX-Xss-Protection: 1; mode=block\nDate: Fri, 19 Feb 2021 23:34:09 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.execute(MinioClient.java:563), io.minio.MinioClient.executePut(MinioClient.java:904), io.minio.MinioClient.copyObject(MinioClient.java:1232), FunctionalTest.testCopyObjectMatchETag(FunctionalTest.java:1850), FunctionalTest.copyObject(FunctionalTest.java:2016), FunctionalTest.runObjectTests(FunctionalTest.java:3757), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(8/15) Running minio-js tests ... done in 50 seconds
(9/15) Running minio-py tests ... done in 2 minutes and 56 seconds
(10/15) Running s3cmd tests ... FAILED in 5 seconds
{
  "name": "s3cmd",
  "duration": "3003",
  "function": "test_put_object_multipart",
  "status": "FAIL",
  "error": "WARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Too many failures. Giving up on '/mint/data/datafile-65-MB'\nERROR: \nUpload of '/mint/data/datafile-65-MB' part 1 failed. Use\n  /usr/local/bin/s3cmd abortmp s3://s3cmd-test-bucket-14398/s3cmd-test-object-31673 5a68696b-08b0-4a49-8fcb-e8bbd2985fa7\nto abort the upload, or\n  /usr/local/bin/s3cmd --upload-id 5a68696b-08b0-4a49-8fcb-e8bbd2985fa7 put ...\nto continue the upload.\nERROR: Upload of '/mint/data/datafile-65-MB' failed too many times (Last reason: )"
}
(10/15) Running s3select tests ... done in 9 seconds
(11/15) Running security tests ... done in 0 seconds

Executed 11 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana harshavardhana merged commit 1732849 into minio:master Feb 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants