Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add CopyObject optimization when source and destination are same #10170

Merged
merged 1 commit into from Aug 3, 2020

Conversation

harshavardhana
Copy link
Member

Description

add CopyObject optimization when source and destination are the same

Motivation and Context

when source and destination are the same and versioning is enabled
on the destination bucket - we do not need to re-create the entire
an object once again to optimize for space utilization.

Cases this PR is not supporting

  • any pre-existing legacy object will not
    be preserved in this manner, meaning a new
    dataDir will be created.

  • key-rotation and storage class changes
    of course will never re-use the dataDir

How to test this PR?

1. there is a fresh bucket with versioning enabled without any objects
2. upload an object passwd - version-1
3. delete passwd - deleteMarker version-2
4. do copy-object of version-1 to create a new version-3 but the contents are same as version-1
   - i.e now new dataDir is created

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation needed
  • Unit tests needed

when source and destination are same and versioning is enabled
on the destination bucket - we do not need to re-create the entire
object once again to optimize on space utilization.

Cases this PR is not supporting

- any pre-existing legacy object will not
  be preserved in this manner, meaning a new
  dataDir will be created.

- key-rotation and storage class changes
  of course will never re-use the dataDir
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-xl.sh ✔️
mint-large-bucket.sh ✔️
mint-dist-xl.sh ✔️
mint-gateway-s3.sh ✔️
mint-gateway-azure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-fs.sh more...

10170-327635b/mint-fs.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:31955
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 804084310b45:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 0 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 41 seconds
(4/15) Running aws-sdk-ruby tests ... done in 2 seconds
(5/15) Running awscli tests ... done in 1 minutes and 3 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 25 seconds
(8/15) Running minio-dotnet tests ... done in 27 seconds
(9/15) Running minio-go tests ... done in 24 seconds
(10/15) Running minio-java tests ... done in 21 seconds
(11/15) Running minio-js tests ... FAILED in 12 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 6,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:673:10) at endReadableNT (_stream_readable.js:1201:12) at processTicksAndRejections (internal/process/task_queues.js:84:21)"
}
(11/15) Running minio-py tests ... done in 56 seconds
(12/15) Running s3cmd tests ... done in 14 seconds
(13/15) Running s3select tests ... done in 2 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana
Copy link
Member Author

ping @krishnasrinivas @poornas

Copy link
Contributor

@poornas poornas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana harshavardhana merged commit 5ce82b4 into minio:master Aug 3, 2020
@harshavardhana harshavardhana deleted the fix-copy-object branch August 3, 2020 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants