Skip to content

Scale S3 authority indexes and auxiliary state #213

Description

@ian-pascoe

Problem

The composable shared-storage review found three normal-path amplification points in the S3 authority:

  • semantic commits rewrite the deployment-wide live-receipt manifest (up to 10,000 receipts);
  • session touches read, sort, hash, and rewrite the complete auxiliary object (up to 50,000 sessions and 10,000 events);
  • static/default credential paths construct and destroy an S3 client per object request instead of reusing connection pools.

At documented cardinalities, mutation/authentication cost grows with deployment-wide history and unrelated replicas contend on the same ETags.

Acceptance criteria

  • Receipt lookup/commit cost is bounded independently of total retained receipts (sharded bounded pages or immutable receipt objects plus maintenance discovery).
  • Sessions use independently conditional records; security events use bounded immutable pages behind a small head/watermark.
  • Static/default SDK clients are owned and reused; dynamic credential/client-factory paths keep bounded request ownership.
  • Export/restore preserves canonical receipts, sessions, events, and watermarks.
  • Multi-replica load tests cover contention and cleanup without LIST-based correctness assumptions.

Evidence

Review anchors: packages/core/src/storage/s3-authority.ts around receipt publication (~2010), auxiliary serialization (~1443), and default client lifecycle (~2268).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions