Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow adding metadata to objects #843

Merged
merged 13 commits into from
Jan 11, 2024
Merged

Allow adding metadata to objects #843

merged 13 commits into from
Jan 11, 2024

Conversation

peterjan
Copy link
Member

@peterjan peterjan commented Dec 20, 2023

This PR adds user metadata to objects. Some S3 clients send metadata in the form of X-Amz-Meta- headers and then rely on the API to return them when making HEAD and/or GET requests. We had an issue report of a user that was experiencing continuous syncing of the same files.

I originally extended the ObjectMetadata with this user defined metadata, but eventually found that was not necessary and that cleaned things up a bunch. Multipart uploads are somewhat annoying, we have to add a separate table just for those and it's attached on CreateMultipartUpload.

I would not include it in 1.0.0 simply because it touches quite a lot of things. I am running it without issues on my personal node now though.

Fixes #833

@peterjan peterjan mentioned this pull request Dec 20, 2023
@peterjan peterjan self-assigned this Dec 20, 2023
@peterjan peterjan marked this pull request as ready for review December 20, 2023 15:32
@peterjan
Copy link
Member Author

peterjan commented Jan 9, 2024

still need to convert migrations to raw sql migrations

api/object.go Outdated Show resolved Hide resolved
api/object.go Show resolved Hide resolved
api/object.go Outdated Show resolved Hide resolved
api/object.go Outdated Show resolved Hide resolved
api/object.go Outdated Show resolved Hide resolved
internal/testing/s3_test.go Show resolved Hide resolved
internal/node/node.go Show resolved Hide resolved
s3/backend.go Outdated Show resolved Hide resolved
s3/backend.go Outdated Show resolved Hide resolved
stores/multipart.go Outdated Show resolved Hide resolved
@peterjan
Copy link
Member Author

verified updated code on my node and rclone sync still shows 2024/01/10 18:25:32 INFO : There was nothing to transfer because it now properly annotates objects with mod times, so still works after changes 👍

@peterjan peterjan changed the title Object Metadata Allow adding metadata to objects Jan 10, 2024
@peterjan peterjan merged commit a4b7e89 into dev Jan 11, 2024
6 checks passed
@peterjan peterjan deleted the pj/metadata branch January 11, 2024 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

s3: rclone sync is re-uploading already existing files
2 participants