RFC: tiered storage

[numinnex]

This is the design discussion for the RFC tiered_storage.
The RFC text is #4404 (rendered).

Final comment period: not started

About this discussion

This discussion is where the design is debated. The pull request keeps editorial review: wording, structure and completeness. Revisions land as new commits on the pull request, never squashed or rebased, and each one is announced in this thread with its commit hash and permalinks to the changed lines. Implementation questions wait for the tracking issue. A maintainer ends the debate by posting a summary and a motion for final comment period with a disposition (merge or close).

Summary

Move sealed segment files from a partition's local directory to an object store (S3 and compatible stores first, GCS and Azure Blob behind the same abstraction) and serve them back through a local chunk cache, without changing what clients see. The partition primary uploads a segment once it seals and commits a tiered_up_to_offset floor to the partition's control log, a replicated log separate from the data log, where a small state machine applies it on every replica. Each replica may then evict its local copy under a per-topic local retention policy. A tiered segment stays in the partition's segment list, backed by a small local marker, so offset and timestamp resolution never touch the network. Total retention is evaluated on the primary over local and remote data together and replicated as the existing TruncatePartition watermark. Object store access goes through an Iggy-owned abstraction with one compio-native HTTP transport and a small signer per provider, so no third-party storage crate and no tokio runtime enter the server.

Proof of concept

A single-node proof of concept of the on-shard approach on tiered_storage_poc branch.

• Covers: upload at seal, eviction, and polls served from the store, with the fs provider and S3 (MinIO).
• Leaves out: the floor and the control log, replication, the chain file, state transfer, store identity, the retention split and TieredReadFailed.
• Setup: MinIO held in RAM on separate cores, one partition, 8 GiB produced at 150 MB/s, three runs per configuration.
• Result: uploading 1 GiB segments on the partition's own shard (about 800 MiB/s) did not move produce p50 or p99, or how late the shard's event loop run
  s. Produce p999 rose from a 38 ms median to 76-103 ms, though the ranges overlap. Capping uploads at 100 MiB/s brought it back to 33 ms.
• Reads from the store: 828 MB/s against 3.2 GB/s from local disk.
• Limits: one machine, a best-case store and no network, so this is a first data point, not a verdict.

Unresolved questions

Listed in the RFC's Unresolved questions section, which revisions update as questions are settled. The ones still open at merge move to the tracking issue.

Revisions

Prior discussion

• Tiered Storage for Iggy #3312 Tiered Storage for Iggy
• Design proposal: S3 as optional primary storage backend (future work, server-ng-targeted) #3232 Design proposal: S3 as optional primary storage backend

[numinnex]

I've expanded the RFC with answer to one of the semi-open questions w.r.t partition state machine holding the watermark for uploaded message offset in this commit

[numinnex]

I've resolved one of the out of scope questions w.r.t cloud topics, we need to do it right now when it's free to do, commit that introduces those changes