Add S3-Compatible Encrypted Blob Storage Backend

[TheSilkky]

Add S3-Compatible Encrypted Blob Storage Backend

Priority

P2

Type

feature

Labels

Suggested GitHub labels:

• backlog
• go
• testing
• docs
• deployment
• security

Branch scope

• Current branch: develop
• Current HEAD: 539bb8d0a2cab1b9dc5e18ff394ae9959c8d7350
• Target branch, if known: develop
• Reviewed branch/ref, if applicable: develop
• Reviewed commit SHA, if applicable: 539bb8d0a2cab1b9dc5e18ff394ae9959c8d7350
• Target release/version, if applicable: post-#88 production-cluster follow-up
• Scope classification: follow-up-after-merge
• Scope note: This draft was generated from the branch above. Revalidate against the target branch before creating or closing public GitHub issues if the branch has moved or has not yet merged.

Summary

Add optional S3-compatible object storage for committed encrypted chunks while preserving local filesystem blob storage as the default.

Context

docs/production-cluster-scope.md plans S3-compatible object storage as the optional production blob backend. The object store must hold opaque encrypted bytes only, use server-controlled object keys, preserve no-overwrite behavior for committed chunks, and provide cleanup guidance for abandoned staging objects. The current internal/storage implementation uses local temp files and hard links to provide immutable final paths.

Proposed change

Implement an optional S3-compatible blob backend after backend selection configuration and a blob-store interface exist. The backend should preserve the current upload contract: uploaded ciphertext is hashed before commit, committed objects are immutable, stored identifiers are server-controlled, and bundle generation streams encrypted bytes without exposing storage internals.

Acceptance criteria

• Local filesystem blob storage remains the default and keeps current behavior.
• S3-compatible storage is selected only through explicit configuration.
• Final object keys are server-controlled and do not accept client-provided stored paths.
• Final object writes use conditional no-overwrite behavior or a documented equivalent.
• Staging object behavior and cleanup expectations are documented.
• Bundle downloads expose controlled ZIP entry names and no object-store URLs or filesystem paths.
• Tests cover no-overwrite behavior, unsafe-key rejection, missing object failure behavior, and local filesystem regression behavior.
• Docs make clear that object storage does not add backend decryption or public-production readiness.

Tests / validation

• gofmt -w ./cmd ./internal ./migrations, if Go files change
• go test ./..., if code changes
• go vet ./..., if code changes or CI/testing changes
• simulator smoke test, if relevant
• docs updated, if relevant
• revalidate on target branch before public issue creation, if branch-scoped

Out of scope

• Do not add provider-specific deployment files, Docker Compose, Kubernetes, Terraform, account management, public /v1 exposure, backend decryption, key escrow, or server-side playable media export.
• Do not require S3-compatible storage for local development.
• Do not implement PostgreSQL or Valkey/Redis coordination in this task unless explicitly split into separate accepted issues.

Notes

• Supported by docs/production-cluster-scope.md, internal/storage, internal/httpapi/upload.go, internal/httpapi/bundles.go, docs/security-model.md, and docs/threat-model.md.
• Related PR: #88.
• Suggested prerequisite drafts: 001-add-backend-selection-configuration-scaffold.md and 003-introduce-blob-store-backend-interface.md.
• Label note: repository labels do not include storage, s3, or object-storage; this draft uses go, testing, docs, deployment, and security.

[TheSilkky]

Closed by PR #109.