S3 endpoint

S3 endpoint

Upcoming. This page describes the s3 branch. It is not part of a release, and details may change.

A Hedgehog daemon serves network storage over the Amazon S3 API, so the AWS SDKs, the aws command line, rclone, restic and other S3 clients store, list and read objects without knowing of gridnodes. Every object is encrypted, erasure coded and spread over the gridnodes like any other file in network storage. The list of buckets, objects, versions and uploads, the index, is kept in network storage as well. The node itself keeps only a journal, a root file, its access keys and a cache.

The endpoint

The endpoint has its own port, 52885, bound to localhost by default, next to the peer-to-peer port 52883 and the REST port 52884. It is enabled by default, and --no-s3 starts the daemon without the endpoint, its index and its lifecycle rounds.

• TLS. The endpoint speaks HTTPS with a self-signed certificate that is made at every start and never stored, so a client has to be told not to verify it. --s3plain serves plain HTTP instead, for local tools that cannot skip the check. Plain HTTP carries objects unencrypted between client and node.
• Addressing. Path style, /<bucket>/<key>, always works. With --s3domain s3.example.com the node also takes virtual-host style, as in bucket.s3.example.com/<key>.
• Signing. Every request is signed with Signature Version 4 and an access key. Any region name is accepted, and a bucket created without a location reports us-east-1. A signed request is refused when its time is more than 15 minutes from the node's clock, and a presigned URL lives at most 7 days.
• Unsigned requests. Only a CORS preflight on a bucket and a browser form upload, which carries a signed policy instead, are taken without a signature.
• No network storage. While the node cannot store or read, because it knows too few gridnodes or storage is switched off, a request is answered with 503 ServiceUnavailable and Retry-After: 5.

Options

These options go after daemon, for example hedgehog daemon --s3plain.

Option	Default	Meaning
--no-s3	endpoint on	Start without the endpoint, the index and the lifecycle rounds; --s3 turns it back on
--s3host	localhost	Address the endpoint binds to
--s3port	52885	Port of the endpoint
--s3plain	off	Serve plain HTTP instead of TLS with a self-signed certificate
--s3domain	unset	Domain whose subdomains name buckets; path style only when unset
--s3cachedir	s3/cache in the data directory	Directory of the local cache
--s3cachesize	1073741824 (1 GiB)	Bytes the cache may hold
--s3cachehalflife	P1D	Time after which a read counts half as much toward keeping an object cached, an ISO-8601 duration
--s3lifecycleinterval	PT1H	Time between the rounds that apply lifecycle rules, an ISO-8601 duration; PT0S turns them off
--s3root	none	Blob id of an index root to mount in place of this node's own, see Recovery

The index directory, s3/index in the data directory, has no option.

Access keys

hedgehog s3 key create
hedgehog s3 key list
hedgehog s3 key delete <access key id>

create makes a key and prints its access key id and secret. The secret is printed only this once. An id is HK followed by 18 characters of A-Z and 0-9. list prints the ids without their secrets, and delete removes a key, after which nothing signed with it is accepted. The commands work on the file s3/credentials in the data directory whether or not the daemon runs, and the daemon reads that file for every request, so a change applies at once. They exit with 0, with 1 for an unknown key and with 2 when the file cannot be read or written.

The file holds the secrets as plain text, since a signature check needs the raw secret. Its directory and the file are created readable by their owner only. There is one account: every valid key has full access to every bucket, and there are no ACLs or policies.

Using clients

The examples use plain HTTP, a daemon started with --s3plain. Against the default TLS, write https:// and add the flag shown that skips the certificate check. Each client takes the access key id and secret from hedgehog s3 key create.

aws, with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set:

aws --endpoint-url http://127.0.0.1:52885 s3 mb s3://photos
aws --endpoint-url https://127.0.0.1:52885 --no-verify-ssl s3 ls

rclone, with a remote in its configuration file:

[hedgehog]
type = s3
provider = Other
endpoint = http://127.0.0.1:52885
force_path_style = true
access_key_id = <access key id>
secret_access_key = <secret access key>

rclone lsd hedgehog: then lists the buckets. Over TLS, add --no-check-certificate.

restic, with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set:

restic -r s3:http://127.0.0.1:52885/backups init

Over TLS, add --insecure-tls.

An AWS SDK needs the endpoint and path-style addressing, unless the node is reached under --s3domain.

What is supported

Area	Operations
Buckets	Create, delete, head, list, location; delete succeeds only when no version and no upload remains
Objects	Put, get, head, delete, copy, with metadata, response-* overrides, one byte range and conditional requests
Listings	ListObjects, ListObjectsV2 and ListObjectVersions with prefix and delimiter, at most 1000 entries a page
Batch delete	DeleteObjects, up to 1000 keys
Multipart uploads	Create, upload a part or copy one, list parts and uploads, complete, abort
Browser uploads	Presigned URLs and POST form uploads with a signed policy
Checksums	CRC32, CRC32C, CRC64NVME, SHA1 and SHA256, and Content-MD5
Bucket settings	Versioning, Object Lock with retention and legal hold, lifecycle rules, CORS, tagging
Object settings	Tagging, retention, legal hold

Not supported, answered with 501 NotImplemented: bucket policies, ACLs, server-side encryption headers and settings, website hosting, storage classes other than STANDARD, MFA delete, Signature Version 4A and the other bucket subresources of S3 not listed above. Encryption settings are left out because network storage already encrypts every object.

Limit	Value
Object stored by one PUT	5 GiB
Multipart upload	10,000 parts, each at least 5 MiB except the last
Key	1024 bytes of UTF-8
Bucket name	3 to 63 characters of a-z, 0-9, . and -, as in S3
Tags	10 per object, 50 per bucket
Lifecycle rules, CORS rules	1000 and 100 per bucket
XML request body	1 MiB

How each feature behaves in detail is on S3 features.

The cache

Objects written or read through the endpoint are kept, decrypted, in a local memory-mapped cache, and so are the blocks of the index. Network storage reads a blob whole, so the first read of an object, even of a range, fetches all of it, and later ranges come from the cache. An object larger than --s3cachesize is not cached. When the cache is full, the entries with the fewest recent reads per byte go first, and a read loses half its weight every --s3cachehalflife. After a restart the cache is rebuilt from the files it holds.

Files

All under the data directory, see Running Hedgehog:

File or folder	Holds
s3/index/journal	Index writes that are not yet in network storage, forced to disk before a write counts as done; journal.frozen beside it while a flush runs
s3/index/root	The blob id of the index's current root in network storage, as plain text
s3/cache/	The cache, decrypted, readable by its owner only; --s3cachedir moves it
s3/credentials	The access keys, one <id> <secret> line each, readable by its owner only

Recovery

Objects and the index live in network storage, so a lost node loses neither. The node writes its index writes to network storage once 16 MiB of them wait or 30 seconds after the oldest, so a lost node takes at most the last 30 seconds of writes with it.

• Keep the root id. The id in s3/index/root opens the whole index, and with it every object. Keep a copy away from the node, and keep it as secret as an access key: it gives read access to everything.
• Mount it elsewhere. hedgehog daemon --s3root <id> mounts that index in place of the node's own, such as that of a lost node. The node refuses while its own index holds writes that are not yet in network storage. One node writes an index at a time. To keep the id out of the shell history, put --s3root <id> in a file and pass the file as @<file>.
• Starting. A node that has just started cannot reach network storage until it knows enough gridnodes. The daemon then keeps running without the endpoint, warns that the S3 endpoint waits for network storage (shown with -vv), and tries again every 10 seconds; the port is bound once the index opens. Any other failure stops the daemon at start.
• Stopping. A stop writes the waiting index writes to network storage. If that fails, they stay in the journal and go out after the next start.

More

Page	For
S3 implementation	Developers: the module, its interfaces towards the node and the classes in the daemon
S3 server and authentication	The HTTP server, flow control, routing, errors and Signature Version 4
S3 namespace index	The index of buckets, versions and uploads: journal, runs, manifests and crash safety
S3 blob layer and cache	How objects map to blobs in network storage, and the local cache
S3 features	Checksums, preconditions, Object Lock, lifecycle rules, CORS, tags and names