threecchi

"Three files in a directory,
masquerading as S3."

Threecchi is a minimal S3-compatible storage server for testing, home lab settings, and very small deployments.

A single ~15 MB executable (or container image) that you can drop right into your test suite, serving files over S3 that you already have on disk.

It can also be used an S3 compatible backend when the primary driver for using an S3 compatible backend is the protocol, but not necessarily the guarantees made by cloud based storage solutions.

Features

Threecchi aims to cover a reasonably complete range of S3 features insofar as they can be implemented without significant external infrastructure, while being relatively easy to use:

• Instant S3. Point threecchi at an existing location, and it will serve it within the limitations of the S3 protocol. Top level directories turn into buckets, content turns into objects. Threecchi maintains state and metadata in an sqlite3 database within the location, so everything stays in one place.

• Regular expression based access control. The access granted by an API key is defined by a regular expression matching buckets or objects within buckets. For testing and development purposes, authentication can also be disabled entirely (anonymous mode).

• Broad S3 surface. Threecchi is small, but covers the features that clients commonly need: multipart, copy, batch delete, object tagging, presigned URLs, browser-based POST uploads, path style and virtual host based addressing, checksums, and conditionals including atomic write semantics suitable for locking.

• Tested against real clients. Test suites exercise common clients including the AWS CLI, boto3, and rclone.

Quick start

Run the container image and point it at a directory to serve. Top-level directories under it become buckets and their contents become objects, so it can also serve files you already have.

Choose the directory to serve:

DATA=/path/to/data

Add an API key with full access (pick your own ID and secret):

podman run --rm -v ${DATA}:/data \
    codeberg.org/snokatt/threecchi:latest \
    key add -id local -secret localsecret -read '.*' -write '.*'

Start the server:

podman run --rm -p 9000:9000 -v ${DATA}:/data \
    codeberg.org/snokatt/threecchi:latest

Verify it works with curl, which signs requests with --aws-sigv4:

Set key ID and secret, and create a file:

export AKID=local SECRET=localsecret
echo "hello threecchi" > hello.txt

Create a bucket:

curl --aws-sigv4 "aws:amz:us-east-1:s3" --user "$AKID:$SECRET" \
    -X PUT http://localhost:9000/my-bucket

Upload the file:

curl --aws-sigv4 "aws:amz:us-east-1:s3" --user "$AKID:$SECRET" \
    -T hello.txt http://localhost:9000/my-bucket/hello.txt

Download it again:

curl --aws-sigv4 "aws:amz:us-east-1:s3" --user "$AKID:$SECRET" \
    http://localhost:9000/my-bucket/hello.txt

Upload an object with public-read and it can be fetched without credentials:

curl --aws-sigv4 "aws:amz:us-east-1:s3" --user "$AKID:$SECRET" \
    -H "x-amz-acl: public-read" -T hello.txt http://localhost:9000/my-bucket/public.txt
curl http://localhost:9000/my-bucket/public.txt

Building

Build and run from source on the host:

The directory to serve:

DATA=/path/to/data

Build: make build

Create an API key (user your own ID and secret):

./threecchi key add -data ${DATA} -id local -secret localsecret -read '.*' -write '.*'

Run the server:

./threecchi -data ${DATA}

Build a container image locally:

podman build -t threecchi .

Use the image as described above in the Quick start section.

Flags

General operation:

• -listen HOST:PORT The IP address and port to listen on (default :9000).

• -data /PATH/TO/DATA The directory to serve (default /data).

• -version Print the version and exit.

Feature flags:

• -min-part-size BYTES The minimum non-final multipart part size (default 5 MiB).

• -anonymous Disable authentication and accept every request.

• -fake-sse Accept and echo x-amz-server-side-encryption without encrypting (instead of refusing requests to encrypt at rest).

TLS:

• -tls Serve HTTPS. With no -tls-cert, generates a self-signed CA and a short-lived leaf under .threecchi/tls and rotates the leaf as it ages.

• -tls-cert FILE / -tls-key FILE Serve this PEM certificate and key as-is, instead of generating one.

• -tls-host NAME An extra SAN (DNS name or IP literal) for the generated certificate. Repeatable.

• -tls-validity DUR The generated leaf lifetime (default 1080h, ~45 days).

The generated certificate covers localhost, the configured hostname (and its *.hostname wildcard), and the host's interface IPs. The leaf renews itself before it expires; point clients at the printed CA path to trust it. For real public certificates, terminate TLS at a proxy (e.g. Caddy) instead.

Configuration

API keys and config live in their own database (.threecchi/state.db), kept apart from the rebuildable object-metadata cache so they survive a cache reset. The management subcommands take the same -data directory as the server.

Recognized config values:

• region The region name reported by certain API calls (default: us-east-1). When set, it is also enforced: a request whose SigV4 credential scope names a different region is rejected with AuthorizationHeaderMalformed, matching real S3. While unset, any client region is accepted.

• hostname The host name (for virtual host style addressing).

Reading/writing Config values:

threecchi config set  -data /data region eu-west-1
threecchi config get  -data /data region
threecchi config list -data /data

API keys with read/write scopes (regular expressions matched against the "bucket/key" resource path; -read and -write are repeatable):

threecchi key add -data /data -read 'photos(/.*)?' -write 'photos/uploads/.*'
threecchi key add -data /data -id MYKEY -secret SECRET -read '.*'
threecchi key list -data /data
threecchi key rm   -data /data ACCESS_KEY_ID

Credentials are generated when -id/-secret are omitted and printed once.

For listing bucket contents, a read scope must match the name of the bucket without the trailing slash.

For listing all buckets, a read scope must match the empty string.

Development

Integration tests:

make test

Integration tests with coverage:

make coverage

Client tests:

make clienttest      # run all client tests
make clienttest-aws  # run a specific client test 

Static tests (go vet, staticcheck):

make statictest