S3 features

S3 features

Upcoming. This page describes the s3 branch. It is not part of a release, and details may change.

Part of S3 implementation. This page covers what the gateway does with checksums, conditional and ranged requests, versions, Object Lock, lifecycle rules, CORS, tags and names, with the limits it applies.

Checksums

The gateway knows five checksum algorithms, each with its header x-amz-checksum-<name>:

Algorithm	Header
CRC32	x-amz-checksum-crc32
CRC32C	x-amz-checksum-crc32c
CRC64NVME	x-amz-checksum-crc64nvme
SHA1	x-amz-checksum-sha1
SHA256	x-amz-checksum-sha256

Crc64Nvme is a reflected table CRC with the polynomial 0x9A6C9329AC4BC9B5 (0xAD93D23594C93659 bit-reversed), an initial value of all ones and a final inversion. Its check value over 123456789 is ae8b14860a799888.

A write may carry a single checksum, as a header or as a trailer of a chunked body, and a body that does not match it is refused with BadDigest before it is kept; see S3 server and authentication. A read returns the stored checksums only with x-amz-checksum-mode: ENABLED and without a range. The ETag is always the MD5 of the body, or for a multipart object the MD5 over the MD5s of its parts with - and their count.

Preconditions

A read checks its conditions in this order, comparing times at whole seconds:

1. If-Match, or else If-Unmodified-Since: 412 PreconditionFailed when it fails;
2. If-None-Match, or else If-Modified-Since: 304 NotModified when it holds.

A write takes two conditions:

Header	Fails
If-None-Match: *	With 412 while a current object exists under the key
If-Match: <etag>	With 404 when there is no current object, and 412 when its ETag differs

Write conditions are checked before the body is stored, and again as the guard of the index write, so a write that raced past the first check still fails.

Ranges

GetObject serves one byte range of the forms a-b, a- and -n. A Range header with several ranges, or one that is not a byte range, is ignored. An unsatisfiable range, such as one that starts past the end of the object, is 416 InvalidRange with Content-Range: bytes */<size>.

Versioning

A bucket's versioning is UNVERSIONED, ENABLED or SUSPENDED. Once enabled it never goes back to unversioned, and a bucket with Object Lock stays enabled.

• Version ids. A write to an enabled bucket gives the new version its write sequence as version id, in 16 hex digits. Any other bucket drops its null version and stores the new one as the version null.
• Delete markers. A delete without a version id stores a delete marker on a versioned or suspended bucket, and drops the null version of an unversioned bucket. A delete with a version id removes that version, unless Object Lock keeps it.
• Version listings. ListObjectVersions (?versions) lists the versions and delete markers of each key, newest first, in pages of at most 1000 entries and common prefixes together; for the keys behind this order, see S3 namespace index.

Object Lock

A bucket with Object Lock is versioned and stays locked. Its configuration may set a default retention of 1 to 36,500 days or 1 to 100 years, which applies to a new version unless the request names its own. A write that locks a version needs Content-MD5 or a checksum.

Mode	Holds
COMPLIANCE	Until its retain-until date, for everyone
GOVERNANCE	Until its retain-until date, unless the request carries x-amz-bypass-governance-retention: true

A retention that still holds can only be replaced by one of the same mode with a date no earlier (Retention.yieldsTo). A legal hold is set or cleared on its own, and blocks the removal of a version as well, whatever its retention.

The checks run as guards of the index write that removes a version, so no request can remove a locked version between the check and the write. The lifecycle sweeper honours them too: Object Lock outranks every lifecycle rule.

Lifecycle rules

Lifecycle.rules validates a configuration before it is kept:

• at most 1000 rules, with unique ids of at most 255 characters;
• exactly one of days, a date or expired delete markers in an expiration;
• NewerNoncurrentVersions of 1 to 100;
• dates at midnight UTC.

The document is stored as sent.

The sweeper. LifecycleSweeper runs a round every --s3lifecycleinterval (default PT1H, PT0S turns the rounds off) on a daemon thread. A round lists the versions of each bucket 1000 at a time, and gathers all versions of one key before it changes any, so no page resumes after a version that the round itself removed. For each key it:

1. removes a delete marker that is the only version left, or expires a current version by storing a delete marker in front of it;
2. removes the noncurrent versions that a rule expires, each counted as noncurrent from the time the newer version after it was written.

It then aborts the multipart uploads older than their rule, matched by prefix alone. A failed round is logged and left to the next.

Guards. Every change is guarded by what the listing showed. A delete marker only lands while the listed version is still the newest, and a version is only removed while it is unchanged and not locked. A request that changed the key first therefore wins.

Counting days. Rule.due counts days from the next midnight UTC, as S3 counts them. An object written at any time on a day, under a rule of one day, expires at the second midnight after it was written.

CORS

A bucket holds at most 100 CORS rules. A preflight OPTIONS request skips authentication and is answered from the rules of the bucket alone; CorsFilter adds the CORS headers to the other responses. A bucket that does not exist has no rules, so a preflight reveals no bucket names.

Tagging

	Objects	Buckets
Tags at most	10	50
Key length	128 characters	128 characters
Value length	256 characters	256 characters

Tag keys starting with aws: are reserved.

Names

• Buckets. Bucket names follow the S3 rules: 3 to 63 characters of [a-z0-9.-], not in the form of an IP address, and none of the reserved prefixes and suffixes.
• Keys. An object key holds at most 1024 bytes of UTF-8.

XML

Request bodies in XML are parsed by S3Xml, which refuses document types and external entities. An XML body is capped at 1 MiB. Bucket configurations are kept as the XML the client sent.

In the source

• s3/src/main/java/org/unigrid/hedgehog/s3/checksum/Crc64Nvme.java: the CRC64/NVME checksum, with ChecksumAlgorithm.java
• s3/src/main/java/org/unigrid/hedgehog/s3/request/Preconditions.java: conditional requests, with ByteRange.java
• s3/src/main/java/org/unigrid/hedgehog/s3/resource/Versions.java: reading and deleting versions
• s3/src/main/java/org/unigrid/hedgehog/s3/resource/ObjectLocks.java: retention and legal holds, with s3/src/main/java/org/unigrid/hedgehog/s3/index/Retention.java
• s3/src/main/java/org/unigrid/hedgehog/s3/lifecycle/LifecycleSweeper.java: the lifecycle rounds, with Lifecycle.java and Rule.java
• s3/src/main/java/org/unigrid/hedgehog/s3/resource/Cors.java: the CORS rules, with CorsFilter.java
• s3/src/main/java/org/unigrid/hedgehog/s3/resource/Tags.java: tag limits
• s3/src/main/java/org/unigrid/hedgehog/s3/request/BucketNames.java: bucket names, with ObjectKeys.java
• s3/src/main/java/org/unigrid/hedgehog/s3/entity/S3Xml.java: XML parsing