S3 server and authentication

S3 server and authentication

Upcoming. This page describes the s3 branch. It is not part of a release, and details may change.

Part of S3 implementation. This page covers the HTTP server of the hedgehog-s3 module, how a request reaches a resource and what it answers, and how requests are authenticated with Signature Version 4, presigned URLs and browser form uploads.

The server

The protocol is a Jersey 3.0.18 application with three resources: ServiceResource at /, BucketResource at /{bucket} and ObjectResource at /{bucket}/{key:.+}. Jersey's public Netty factory owns its event loops and hides the container that has to be shut down, so S3Server builds Jersey's package-private NettyHttpContainer by reflection and runs its own ServerBootstrap on 4 event loop threads (COMMUNICATION_THREADS). WADL is off, so no route is described, not even to a signed request.

S3ChannelInitializer builds the pipeline of each connection:

Handler	Job
SslHandler	TLS, left out with --s3plain
HttpServerCodec	HTTP parsing, with a request line of up to 16 KiB, headers of up to 16 KiB and chunks of 64 KiB
HttpServerKeepAliveHandler	Keeps the connection open between requests
S3RequestHandler	A fresh one per connection; hands each request to Jersey

Jersey's own initializer keeps Netty's 4 KiB request line, which a long key with a presigned query exceeds. With TLS, S3Server builds a Netty SelfSignedCertificate on every start and never stores it.

S3RequestHandler hands each request to Jersey with an S3RequestBody as entity stream and an S3ResponseWriter, and runs it on Jersey's executor, off the event loop. An unparsable request gets 400 and the connection closes. 100 Continue is sent from the event loop as the request is handed to Jersey, before SignatureFilter has run.

A connection serves one request at a time. Once the last content of a request has arrived, the handler holds the read gate and parks anything pipelined behind it in the deque held until the response commits, so responses never interleave. It then replays what it parked.

Flow control

Jersey's own handler queues whole bodies and lets its writer block the event loop. The gateway replaces both, so a fast client is held back by its own TCP window, not by server memory.

The read gate. S3ReadGate keeps a set of holders and turns Netty's autoRead off while the set is not empty. Three things hold it:

• the request body, while its queue is full;
• the request handler, from the last content of a request until the response commits;
• a body that the response discards without draining, so the connection reads nothing more before it closes.

The request body. S3RequestBody is an InputStream over a queue of Netty buffers. It holds the gate once bodyHighWater waits unread, and releases it on a read that leaves at most bodyLowWater. The low mark lies far enough below the high one that reading is not toggled on every packet. A read that waits longer than stallTimeout fails and closes the channel. Every buffer is released once it is read, discarded or failed.

The response. S3ResponseWriter writes from the resource thread in pieces of CHUNK. Before each piece, while the channel is not writable, it waits for the previous write to complete, at most the stall timeout, and fails on a failed write or a closed channel. Waiting on the previous write needs no event to wake the writer, and the event loop never waits for a resource. A body of declared length is cut together with its connection when it runs past that length or ends short of it, so a client never takes a wrong body for a whole one. A suspension with a timeout is refused.

An unread body. When the response commits, an unread rest of the request body is drained to keep the connection only if at most MAX_DRAIN of it remains. Past that the response carries Connection: close, as a client could otherwise make the server read an endless body.

Value	Default	Effect
bodyHighWater	1 MiB	Unread body at which reading from the socket stops
bodyLowWater	256 KiB	Unread body at which reading resumes
stallTimeout	1 minute	Longest wait for a body to arrive or for room to write
CHUNK	64 KiB	Size of a written piece of a response
MAX_DRAIN	1 MiB	Largest unread rest of a body drained to keep the connection

Filters and routing

Two filters run before any route is chosen. SignatureFilter is @PreMatching at priority AUTHENTICATION, so nothing about the routes is learnt without a valid signature. VirtualHostFilter follows at AUTHENTICATION + 1. When --s3domain is set, it rewrites a request to bucket.domain/key to the path style domain/bucket/key, so the resources only see path-style requests. It runs after authentication because clients sign the path they sent. Without --s3domain, only path style is served.

Two response filters follow the resource. RequestIdFilter sets x-amz-request-id (8 random bytes in hex), x-amz-id-2 (48 random bytes in Base64) and Date, from the same clock that signatures are checked against. CorsFilter adds the CORS headers.

S3 names most operations by a query parameter, the subresource. Subresource recognises 13 of them and answers NotImplemented for 22 more, among them acl, policy, encryption and website. Each method names the subresources it serves; any other is 501, since it would otherwise be taken for the plain operation.

Operations

The selecting subresource or header is in parentheses. Bucket configurations use the subresources tagging, versioning, object-lock, lifecycle and cors.

Method	/ and bucket /b	Object /b/k
GET	ListBuckets (/); ListObjects; ListObjectsV2 (list-type=2); ListObjectVersions (versions); ListMultipartUploads (uploads); GetBucketLocation (location); GetBucketTagging, GetBucketVersioning, GetObjectLockConfiguration, GetBucketLifecycleConfiguration, GetBucketCors	GetObject; GetObjectTagging (tagging); ListParts (uploadId); GetObjectRetention (retention); GetObjectLegalHold (legal-hold)
HEAD	HeadBucket	HeadObject
PUT	CreateBucket; PutBucketTagging, PutBucketVersioning, PutObjectLockConfiguration, PutBucketLifecycleConfiguration, PutBucketCors	PutObject; CopyObject (x-amz-copy-source); UploadPart, UploadPartCopy (uploadId); PutObjectTagging, PutObjectRetention, PutObjectLegalHold
POST	DeleteObjects (delete); PostObject (browser form)	CreateMultipartUpload (uploads); CompleteMultipartUpload (uploadId)
DELETE	DeleteBucket; DeleteBucketTagging, DeleteBucketLifecycle, DeleteBucketCors	DeleteObject; DeleteObjectTagging; AbortMultipartUpload (uploadId)
OPTIONS	CORS preflight	CORS preflight

Any other subresource on a method, the 22 names of Subresource.UNSUPPORTED, server-side encryption headers, storage classes other than STANDARD, MFA delete and Signature Version 4A answer NotImplemented.

• DeleteObjects takes up to 1000 keys, with Quiet, and reports an error per key.
• DeleteBucket succeeds only when no version and no upload remains in the bucket.
• ListObjectsV2 takes continuation-token, start-after and fetch-owner.
• GetObject takes the response-* overrides.
• HeadBucket reports the location of a bucket, or us-east-1.

Errors

S3ExceptionMapper walks the cause chain of an exception. The first S3Exception wins. Otherwise the first cause that matches decides, in this order:

1. BlobUnavailableException: 503 ServiceUnavailable with Retry-After: 5;
2. malformed XML;
3. no such bucket;
4. bucket not empty;
5. no such upload;
6. a failed condition: 412;
7. NotAllowedException;
8. NotFoundException: NotImplemented, since every S3 path is routed;
9. ClientErrorException: InvalidRequest.

Anything else is logged and answered as InternalError. HEAD and 304 responses carry no <Error> body. S3ErrorCode has 49 constants, named exactly as the S3 codes that clients match on:

Status	Codes
304	NotModified
400	AccessControlListNotSupported, AuthorizationHeaderMalformed, AuthorizationQueryParametersError, BadDigest, BadRequest, EntityTooLarge, EntityTooSmall, IllegalVersioningConfigurationException, IncompleteBody, IncorrectNumberOfFilesInPostRequest, InvalidArgument, InvalidBucketName, InvalidDigest, InvalidPart, InvalidPartOrder, InvalidPolicyDocument, InvalidRequest, InvalidTag, KeyTooLongError, MalformedPOSTRequest, MalformedTrailerError, MalformedXML, MaxPostPreDataLengthExceededError, XAmzContentSHA256Mismatch
403	AccessDenied, AccessForbidden, InvalidAccessKeyId, RequestTimeTooSkewed, SignatureDoesNotMatch
404	NoSuchBucket, NoSuchCORSConfiguration, NoSuchKey, NoSuchLifecycleConfiguration, NoSuchObjectLockConfiguration, NoSuchTagSet, NoSuchUpload, NoSuchVersion, ObjectLockConfigurationNotFoundError
405	MethodNotAllowed
409	BucketAlreadyOwnedByYou, BucketNotEmpty, InvalidBucketState
411	MissingContentLength
412	PreconditionFailed
416	InvalidRange
500	InternalError
501	NotImplemented
503	ServiceUnavailable

Authentication

SignatureFilter lets exactly two kinds of request through unsigned:

• A CORS preflight on a bucket: an OPTIONS request with Origin and Access-Control-Request-Method, which the CORS rules of the bucket alone answer. An OPTIONS request without these headers is signed like any other.
• A browser form upload: a multipart/form-data POST to the root of a bucket, which carries a signed policy instead (see Browser form uploads).

Every other request needs an Authorization header or a presigned query. Both at once is InvalidArgument, neither is AccessDenied. The filter looks up the secret of the access key (InvalidAccessKeyId when there is none), derives the signing key, checks the time and the signed headers, rebuilds the canonical request and compares the signatures in constant time.

Signature Version 4

SigV4 follows Signature Version 4 as S3 applies it. The canonical request is built from:

• the method;
• the path, decoded and encoded once, slashes kept, without normalisation;
• the query parameters, re-encoded and sorted by name and value, without X-Amz-Signature when presigned;
• the signed headers, each value with its whitespace collapsed and several values joined by commas;
• the list of signed header names and the payload hash.

The string to sign is the algorithm, the timestamp, the scope and the SHA-256 of the canonical request. The scope is <date>/<region>/s3/aws4_request, and any non-blank region is accepted. The signing key is a chain of four HMACs over the parts of the scope: the first, keyed by AWS4 and the secret, takes the date, and each next one, keyed by the result before it, takes the region, s3 and aws4_request in turn. Each signature is the HMAC of a string to sign under this key.

host must be signed. An unsigned x-amz-* header is AccessDenied, as anyone on the way could have added it.

Time. The time of a request comes from x-amz-date, else from an RFC 1123 Date, and its day must equal the date of the scope.

Form	Accepted when
Authorization header	Within MAX_SKEW (15 minutes) of the server clock, either way; else RequestTimeTooSkewed
Presigned query	Not more than 15 minutes in the future, and not past its expiry; else RequestTimeTooSkewed or AccessDenied

A presigned request has no bound into the past, but expires after X-Amz-Expires, which takes 1 to 604800 seconds (MAX_EXPIRY, 7 days).

Payloads

x-amz-content-sha256 selects how the body is checked. A presigned request without it is unsigned.

x-amz-content-sha256	Body
64 hex digits	Plain, its SHA-256 checked at the end
UNSIGNED-PAYLOAD	Plain, not covered by the signature
STREAMING-AWS4-HMAC-SHA256-PAYLOAD	aws-chunked, every chunk signed
STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER	Signed chunks and a signed trailer
STREAMING-UNSIGNED-PAYLOAD-TRAILER	Unsigned chunks and a trailer
STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD	NotImplemented (Signature Version 4A)

Chunked bodies. The chunked modes need x-amz-decoded-content-length. The filter rewrites Content-Length to it and strips aws-chunked from Content-Encoding. AwsChunkedInputStream reads chunks, each a line <hex size>[;chunk-signature=<sig>], the data and CRLF, then a zero chunk and the trailers.

Limit	Value
Chunk line	4096 bytes
Chunk size	15 hex digits
Trailers	8, in 16 KiB together
Data of all chunks	Never past the decoded length

Trailers must be announced in x-amz-trailer, and x-amz-trailer-signature is allowed for signed chunks. A chunk is verified as its last byte is read, so a reader never sees the end of a refused chunk. ChunkSigner signs each chunk with the signing key over a string that holds the signature before it and the SHA-256 of the chunk data; the first chunk follows the signature of the request, and the trailer's string holds the SHA-256 of its canonical lines. The signatures form a chain in which no chunk can be left out, added or moved.

Checks at the end. Payload runs its checks inside the read that reaches the end of the body, so a storing resource fails before it keeps a refused body:

Check	Error
Declared length	IncompleteBody
Signed SHA-256	XAmzContentSHA256Mismatch
Content-MD5, and a single x-amz-checksum-* header or trailer	BadDigest

The ETag of a stored body is always its MD5.

Presigned URLs

Authorization.fromQuery reads the signature parameters of a presigned URL, which is then checked the same way as a signed header. The presigned query must fit in the 16 KiB request line.

Browser form uploads

A browser uploads a file with a multipart/form-data POST to the bucket, authenticated by a POST policy the owner of a key signed in advance. The streaming parser FormData reads the form, with a boundary of at most 70 bytes and part headers of at most 16 KiB. FormUpload takes at most 64 fields of 20 KiB each before the file field.

PostPolicy.verify checks the HMAC of the Base64 policy before it parses the JSON, then its expiration and its conditions:

• an exact match, or eq;
• starts-with;
• content-length-range, capped at 5 GiB.

Every field but policy, x-amz-signature, file and x-ignore-* must be covered by a condition. A policy is bound to exactly one bucket: an exact condition on the bucket is mandatory, and starts-with on the bucket is refused, since a policy handed out for one bucket would otherwise upload to every other.

Access keys

S3Credentials keeps one line <id> <secret> per key in <data>/s3/credentials, which the hedgehog s3 key commands manage; see S3 endpoint.

• A secret is 30 SecureRandom bytes in Base64.
• The directory and the file are created rwx------ and rw-------, and never restricted afterwards.
• The file is written atomically, through a temporary file moved over the old one.
• The owner of every bucket is the constant hedgehog.

In the source

• s3/src/main/java/org/unigrid/hedgehog/s3/server/S3Server.java: the bootstrap and the Jersey container, with S3ChannelInitializer.java and S3ServerConfig.java
• s3/src/main/java/org/unigrid/hedgehog/s3/server/S3RequestHandler.java: one request at a time per connection
• s3/src/main/java/org/unigrid/hedgehog/s3/server/S3ReadGate.java: turning reads off and on, with S3RequestBody.java
• s3/src/main/java/org/unigrid/hedgehog/s3/server/S3ResponseWriter.java: writing a response in pieces
• s3/src/main/java/org/unigrid/hedgehog/s3/http/VirtualHostFilter.java: virtual-host style, with RequestIdFilter.java and Subresource.java
• s3/src/main/java/org/unigrid/hedgehog/s3/error/S3ExceptionMapper.java: exceptions to S3 errors, with S3ErrorCode.java
• s3/src/main/java/org/unigrid/hedgehog/s3/auth/SignatureFilter.java: the authentication of every request
• s3/src/main/java/org/unigrid/hedgehog/s3/auth/SigV4.java: the canonical request and the signing key, with Authorization.java
• s3/src/main/java/org/unigrid/hedgehog/s3/auth/AwsChunkedInputStream.java: chunked bodies, with ChunkSigner.java
• s3/src/main/java/org/unigrid/hedgehog/s3/auth/PostPolicy.java: the policy of a form upload
• application/src/main/java/org/unigrid/hedgehog/server/s3/S3Credentials.java: the access key file