v0.16.0

Highlights

Multipart upload state has moved out of buckets. Parts of an in-progress upload used to sit under __s2mp__/ inside the target bucket, where no S3 call could find them and nothing removed them. They now live under <Root>/.multipart/<uploadId>/, expire after S2_SERVER_MULTIPART_MAX_AGE (24 hours by default), are swept at startup and hourly, and can be inspected with ListMultipartUploads and ListParts. An upload is bound to the bucket it was started in, so deleting and recreating a bucket invalidates its uploads instead of completing them into the new bucket.

Changes

Features

• Store in-progress multipart uploads under <Root>/.multipart/ instead of inside the bucket, and keep .-prefixed root entries out of the bucket namespace (#223, #225)
• Expire multipart uploads after S2_SERVER_MULTIPART_MAX_AGE and sweep their parts at startup and every hour (#228)
• Bind a multipart upload to its bucket's creation time, so a recreated bucket does not accept the old upload IDs (#230)
• Add ListMultipartUploads and ListParts, with the s3:ListBucketMultipartUploads and s3:ListMultipartUploadParts policy actions (#231)

Bug Fixes

• Compute the completed object's ETag before the final write, so CompleteMultipartUpload no longer leaves an object without one when the second write fails (#225)
• Delete only the named bucket, not every bucket whose name shares its prefix (#226)
• Keep an existing bucket's .keep on a repeated CreateBucket, so CreationDate no longer resets on ensure-exists calls (#230)
• Reject CompleteMultipartUpload with an empty parts list (#220)

Documentation

• Show the multipart directory in the README's storage layout (#232)
• Correct where the README's storage layout places metadata sidecars (#233)

Dependencies

• Bump github.com/mojatter/wfs to v0.7.1 (#227)
• Bump the go-dependencies group across 3 directories with 6 updates (#222)

Upgrading

• Submodule consumers: go get github.com/mojatter/s2/<module>@v0.16.0
• cmd/s2-server/v0.16.0 is published separately, shortly after
• Multipart uploads in progress at upgrade time answer NoSuchUpload; start them again. Their parts stay under __s2mp__/ in each bucket. The S3 API still hides that prefix in this release (the filter is removed in v0.17), so remove it one of these ways:
  • Web Console: open the bucket and delete the __s2mp__ folder.
  • osfs: rm -rf "$S2_SERVER_ROOT"/*/__s2mp__ "$S2_SERVER_ROOT"/*/.meta/__s2mp__ (the second path holds their metadata).
  • Cloud backends: delete <bucket>/__s2mp__/ under the configured root with the provider's own tool, e.g. aws s3 rm --recursive s3://<root>/<bucket>/__s2mp__/.
• Uploads older than S2_SERVER_MULTIPART_MAX_AGE (default 24 hours) are refused and their parts removed. Set a negative value to keep uploads forever.
• ListBuckets and the Web Console skip top-level entries whose name starts with .; .multipart is one.
• Policies: s3:ListBucket no longer covers GET /{bucket}?uploads. Grant s3:ListBucketMultipartUploads for ListMultipartUploads and s3:ListMultipartUploadParts for ListParts; neither can be given to the anonymous user.
• server module: Buckets.CreatedAt now returns (time.Time, error) — a zero time when the bucket has no .keep marker, and an error instead of the current time when the marker cannot be read.
• Storage implementers: DeleteRecursive with a prefix ending in / must remove only that directory, not names that merely share the prefix (#226). s2test.TestStorageDelete checks this; run it against your backend.

Full Changelog

v0.15.0...v0.16.0