storage

Batteries-included toolkit for building applications on top of object storage in Go.

Motivation

Object storage is one of the most powerful building blocks available to application developers — infinitely scalable, durable, and cheap. There are already Go packages that abstract away provider differences behind a common interface, and that part is relatively straightforward. What's harder is everything else you need to actually build on top of object storage: presigned URLs, caching layers, bucket notifications, secret storage, observability, and composable middleware — the kind of infrastructure that every serious application ends up reimplementing from scratch.

The storage package ships all of that in one cohesive toolkit. At its core is a single Bucket interface covering S3, Cloudflare R2, Google Cloud Storage, the local file system, HTTP, and in-memory storage — pick a URI, import a driver, and go. But the real value is what's built on top: composable adapters for caching, prefixing, instrumentation, and read-only access; first-class presigned URL support across backends; bucket change notifications; and a secret management layer. Streaming operations return iter.Seq2 iterators that plug straight into range loops and the standard library, keeping everything idiomatic.

Whether you are building a data pipeline, a media service, or a CLI tool that needs to talk to the cloud, storage is designed to let you focus on your application instead of the plumbing underneath it.

Usage

storage.LoadBucket

Load a bucket by URI. The scheme selects the backend — import the backend package for side effects to register it.

import (
    "github.com/firetiger-oss/storage"
    _ "github.com/firetiger-oss/storage/s3"  // register s3:// scheme
    _ "github.com/firetiger-oss/storage/gs"  // register gs:// scheme
    _ "github.com/firetiger-oss/storage/file" // register file:// scheme
)

bucket, err := storage.LoadBucket(ctx, "s3://my-bucket")

storage.GetObject / storage.PutObject

Top-level convenience functions operate directly on object URIs without loading a bucket first.

// Write an object
_, err := storage.PutObject(ctx, "s3://my-bucket/path/to/file.txt",
    strings.NewReader("Hello, World!"),
    storage.ContentType("text/plain"),
)

// Read it back
reader, info, err := storage.GetObject(ctx, "s3://my-bucket/path/to/file.txt")
defer reader.Close()

storage.ListObjects

List objects under a prefix. Results stream as an iterator.

for object, err := range storage.ListObjects(ctx, "s3://my-bucket/logs/") {
    if err != nil {
        return err
    }
    fmt.Printf("%s (%d bytes)\n", object.Key, object.Size)
}

storage.AdaptBucket

Wrap a bucket with adapters to add caching, prefixing, instrumentation, or read-only protection.

bucket = storage.AdaptBucket(bucket,
    storage.WithPrefix("data/"),
    storage.NewCache(),
    storage.WithInstrumentation(),
)

readOnly := storage.ReadOnlyBucket(bucket)

Backends

Backend	URI	Import
Amazon S3	s3://bucket/prefix	_ "github.com/firetiger-oss/storage/s3"
Cloudflare R2	r2://bucket/prefix	_ "github.com/firetiger-oss/storage/r2"
Google Cloud Storage	gs://bucket/prefix	_ "github.com/firetiger-oss/storage/gs"
Local file system	file:///path	_ "github.com/firetiger-oss/storage/file"
In-memory	:memory:	_ "github.com/firetiger-oss/storage/memory"
HTTP (S3-compatible)	http://host/path	_ "github.com/firetiger-oss/storage/http"

Contributing

Contributions are welcome! To get started:

1. Ensure you have Go 1.25+ installed
2. Run go test ./... to verify tests pass

Please report bugs and feature requests via GitHub Issues.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.