docs: write architecture document

[frrist]

What this is

The design doc set for the Ingot S3-over-Forge redesign, bundled as one PR so the architecture
and the analysis that backs it are reviewed together. No code changes — these four docs are the
shared understanding we need before building.

docs/
├── aggregation-gate.md            ← start here (5 min): why this is gated on Piri
├── architecture.md                ← the target design (the main artifact)
├── rfc-pdp-minimum-piece-size.md  ← the PDP gas/economics + Piri policy that the size knobs rest on
└── pdp-cost-calculator.html       ← interactive model behind the RFC (open in a browser)

Start here

Suggested reading order

1. aggregation-gate.md — the one-page framing. Thesis: none of the S3 design runs on Forge until Piri stops size-pooling unrelated objects into ~128 MiB pieces. Read this first; it tells you why the other two docs exist and what the two asks of the rest of the stack are.
2. architecture.md — the target architecture, layer by layer (S3 → catalog → data → Forge/chain → cross-cutting), then a system contract, topology, gaps, and a Postgres schema appendix. This is the artifact most of your review time should go to.
3. rfc-pdp-minimum-piece-size.md — open to the Decision summary (top). That page is the whole argument; §1–§10 are the supporting analysis and the Appendices (behind the "Evidence" banner) are the raw gas audit — skim unless you're checking the numbers.
4. pdp-cost-calculator.html — play with piece size / churn / base fee / sender count to pressure-test the economics yourself.

Where your input is most wanted

The docs call these out explicitly; these are the decisions we want ratified, not just read:

• Aggregation model (aggregation-gate.md, architecture.md §6, rfc §7): the proposal is no cross-object aggregation — store each object as its own piece(s) — with the size floor (min_aggregate_size, ~8 MiB) demoted to the fallback if the contracts can't take that piece volume. The open question for FilOZ / FOC: can the PDP contracts/proving take object=piece at PiB scale? (See the top-level note "Should Piri do cross-object aggregation at all?")
• Versioned key encoding (architecture.md §3 callout): composite-key vs per-key version index; and versionId as a direct locator (scan-free, but leaks version ordering) vs opaque. We picked the direct locator — weigh the trade-off.
• The two asks (aggregation-gate.md): (1) rethink Piri's aggregation — target no cross-object aggregation (object=piece), with a generational background aggregator as the relief valve and the size floor as the fallback; (2) prove small pieces are operationally viable in the PDP contracts at PiB scale. Do you agree this is the gate, and the sequencing?

Correctness — what was verified against source

Every gas/contract claim in the RFC was checked against the actual contracts (by Claude!)
(PDPVerifier.sol and FilecoinWarmStorageService.sol, re-verified against FWSS v1.3.0 / PDPVerifier v3.4.0 after an upstream contract update) and the calibnet
gas-benchmark fixtures. The load-bearing (dammit claude) findings held: no proof-slashing exists (the new 0.1 FIL cleanup deposit is refundable, not a penalty), add cost is size-independent (~120,700 gas), every on-chain hot path is bounded (no O(live pieces) loop), and the calibnet anchors matched exactly. Corrections that came out of the audit and are already baked in:

• ⚠️ The addPieces batch cap was removed upstream (FWSS v1.3.0 / PDPVerifier v3.4.0). EXTRA_DATA_MAX_SIZE / MAX_ADD_PIECES_EXTRA_DATA_SIZE are gone; batch size is now bound by the FVM PiecesAdded event-size + per-tx gas. So the registration-throughput numbers (pieces/tx, txns/PiB, sender fleet) are an open measurement, not fixed — the earlier ~13-pieces/tx / ~11M-txn / ~12-sender figures are withdrawn pending re-measurement. (Good news for "many small pieces": the contract wall that would have throttled them is gone.)
• Storage price is 2.5 USDFC/TiB/mo (not 0.9), now an immutable constant (lib/PriceListUSDFC.sol); v1.3.0 also adds a per-dataset USDFC fee layer. Doesn't move the margin conclusion.
• Citation fixes (nextPieceId now :800, standalone-submit jobqueue.go:205, challenge-window 60 mainnet / 20 calibnet / 10 devnet, etc.).

Scope / non-goals

• Docs only — no implementation in this PR. The architecture explicitly supersedes the bootstrap MVP currently in the tree.
• Catalog GC, the allocate-by-size security model, and the batch allocate/accept wire format are named as known gaps, not solved here (architecture.md §11).
• Some RFC code references point at sibling repos (piri, filecoin-services/FilOzone) by design, since the gas model is derived from them.

[bajtos]

This is a well-written, detailed proposal. Love it!

I am afraid I ran out of energy & concentration by the time I got to the middle of docs/architecture.md. I'll continue the review of section 7 and onwards tomorrow.

[bajtos]

Wow, that's a lot of transactions!

• Do we have an estimate of how much we will pay in gas fees to onboard PiB of 8 MiB pieces?
• How quickly can we submit these 11M transactions on the chain?

[frrist]

Good questions, and the answer splits ~cleanly: gas dollars are (probably) not the problem; transaction throughput is.

• Gas cost: registration (addPieces) is ~120,700 EVM gas/piece (size-independent), ~8× in FEVM. For a whole PiB at 8 MiB (as an example) that's on the order of a fraction of a FIL to low tens of dollars at a quiet base fee — negligible against storage revenue. And registration is deferrable: Piri gas-gates it (defer-and-retry, no deadline, no slashing for being late), so we ride cheap-gas windows and never pay spike prices for it. So I'm not worried about the fee. But there is a tradeoff for operators here, since not registering during times of high gas fees means not being paied for data stored.
• Throughput: this is the real wall. And this all depends on how many roots we can add in a single transaction, and how many of those we can send per epoch. The calculator in here attempts to model this.

So "how fast" is a function of how many addPieces transactions we can send, and how many pieces we can stuff in each transaction.

To be upfront: this throughput magnitude is exactly the assumption I'm asking FilOZ to stress-test in the top-level note ("Should Piri do cross-object aggregation at all?"). Good news since I first wrote this: FilOZ already removed the addPieces batch cap in v1.3.0, so the registration wall is softer than the early ~13-pieces/tx draft assumed — but the measured pieces/tx (and whether to shard across multiple data sets) is still the open question that decides whether we can drop aggregation entirely or have to keep a floor.

[bajtos]

I am wondering how you chose 8 MiB as the new floor, rather than 1, 2, 4, 16, or 32 MiB?

[frrist]

Honestly, 8 MiB is a principled starting point, not a derived optimum. It's the AWS SDK default: both boto3 and the aws CLI default multipart_threshold and multipart_chunksize to 8 MiB, so any object large enough to matter arrives already split into ~8 MiB parts by the client. Setting the floor there makes "one client part ≈ one on-chain piece" land for the dominant upload path, while smaller whole objects fall below the floor and get aggregated.

The trade across the range: a smaller floor (1/2/4) puts more objects above the floor (better delete granularity, each its own piece) but multiplies piece count and therefore registration transactions; a larger floor (16/32) means fewer pieces but more sub-floor aggregation and bigger repack units when a member is deleted. 8 sits at the natural client boundary.

That said — I'm now questioning whether we want a floor at all. The cleaner model is one object = its own piece(s) with no cross-object aggregation, which makes "8 vs 1/2/4/16/32" moot. I've written that up as a top-level note ("Should Piri do cross-object aggregation at all?"); short version: 8 MiB only matters if FilOZ tells us we must aggregate, in which case the floor is a function of the measured object-size distribution we don't have yet (the instrumentation/Aurora thread). Absent that constraint, there's no floor to pick.

[bajtos]

Honestly, 8 MiB is a principled starting point, not a derived optimum. It's the AWS SDK default: both boto3 and the aws CLI default multipart_threshold and multipart_chunksize to 8 MiB, so any object large enough to matter arrives already split into ~8 MiB parts by the client. Setting the floor there makes "one client part ≈ one on-chain piece" land for the dominant upload path, while smaller whole objects fall below the floor and get aggregated.

Thank you for the explanation; now the choice makes sense to me!

[bajtos]

Do we have any concerns about the one node-wide PDP data set as we grow the amount of data stored?

If a node is storing 100TiB of data, spread across 1M objects >= 8MiB:

• The data set has 1M roots, does the on-chain data structure support this?

The PDP proofs are probabilistic. I asked Claude to compare one 100TiB-sized proof set vs 100x 1TiB-sized proof sets, and it says there is meaningful difference.

Summary:

The probabilistic guarantee is "you can't cheat much for long without detection," not "every byte is checked every period." But "eventually" scales with how small the missing fraction is — and a single 100 TiB set makes "eventually" very long for small losses.

More details:

• One proof set, 100 TiB, 5 challenges/period: If the provider drops 1% of your data (1 TiB), per period: 1 − 0.99^5 ≈ 4.9% chance of catching it. They'd expect to get away with it for ~20 periods on average. Drop 0.1% (100 GiB) and per-period detection falls to ~0.5%.
• 100 proof sets, 1 TiB each, 5 challenges each = 500 challenges/period total:
  Now you have 500 independent challenges across your data every period. If the provider drops 1% spread across everything: 1 − 0.99^500 ≈ 99.3% caught per period. Even 0.1% missing gives 1 − 0.999^500 ≈ 39% per period.
• There's a localization problem beyond raw probability. With 1M roots in one set and only 5 challenges, an adversary who deletes one entire root (say one of your 1M pieces, ~100 MiB) is missing 1/1,000,000 of the leaves. Per-period detection ≈ 5/1,000,000 = 0.0005%. That root could be gone for thousands of years of proving periods before a challenge happens to land on it. The data is effectively unprotected at fine granularity.

[frrist]

Two halves to this, and I want to answer them differently.

On the security/probabilistic half (the 5-challenges-over-1M-roots dilution argument): I'd rather not pull that thread here. This doc treats the PDP/PDPVerifier contract security model as a vetted assumption, and how hard it is to cheat over a proving window is covered by the contract security analysis, not this doc.

On the operational scaling half, though — your instinct is right and it's very much live, so I've pulled it into a top-level note ("Should Piri do cross-object aggregation at all?"). Structurally, I believe ~1M roots is fine: every on-chain hot path is bounded (O(batch) / O(5) / O(log cumulative) / O(removals≤2000)). The real open question you're circling is how many proof sets a node should maintain. Today it's one data set per node for operational simplicity — but each set is a proof the node must produce every period, so sharding into N sets is N proofs/day and shifts the node's SLA and gas usaged since more proofs per day. The flip side is that a cap on pieces-per-set + sharding could help delete throughput, localization, and per-tenant rails. I don't have the answer — it's exactly the kind of thing we need FilOZ to weigh in on, and it's tied to whether we can drop cross-object aggregation entirely. Let's track it in that note.

[bajtos]

On the security/probabilistic half (the 5-challenges-over-1M-roots dilution argument): I'd rather not pull that thread here. This doc treats the PDP/PDPVerifier contract security model as a vetted assumption, and how hard it is to cheat over a proving window is covered by the contract security analysis, not this doc.

I agree to not question the PDPVerifier contract's security model.

What I am questioning: Did FilOz consider the security implications of very large ProofSets (like 100TiB+)? If they did, what is their recommendation for the maximum proofset size that still keeps PDPVerifier security promises?

I think it's important to include this point in our discussions with FilOz.

[alanshaw]

Can this please be a PR against the RFC repo?

I haven't yet reviewed rfc-pdp-minimum-piece-size.md but I have review fatigue so I'm going to stop there for today and publish the comments I have.

[alanshaw]

I'd like to not never repack. I'm unsure how we square this off, i.e. if we allow storage nodes to prove over pieces that no customer is paying for, it's basically on us to pick up the bill. Furthermore how do we distinguish a uncollected garbage piece from a completely random piece the storage node inserted to earn more $$$.

[frrist]

Agreed — "never" was too strong, and I think the cleanest way to "not never repack" is to not cross-object aggregate in the first place. I've written this up as a top-level note ("Should Piri do cross-object aggregation at all?"). If each object is its own piece(s), there's nothing to repack on delete (you retire the object's own piece), and the concerns somewhaat dissolve:

• Unpaid pieces: there's no shared aggregate accumulating dead bytes from other objects, so the proven set maps 1:1 to objects we're paying for.
• Garbage vs. fabricated — and these split on trust, which I want to be careful about: the garbage half (dead bytes from our own deletes) is honest-operator GC — non-adversarial — so our own bookkeeping tells us which pieces are no longer referenced and we schedule them for retirement on the normal delete path. The fabricated-piece half is adversarial and harder. If we, or a client ran Ingot we could reconcile on-chain roots against blob_refs. But that does not work for if the SP runs Ingot since:
  1. blob_refs is keyed by the blob's sha256, but the on-chain root is a CommP, and there's no way to verify a CommP corresponds to a given sha256 — that binding is a UCAN attestation (trust), not a computation
  2. the SP operator is the one running Ingot, so reconciling against their reverse index is letting the adversary audit themselves.

The trust-correct anchor has to be FilOne-side: Sprue issues every accept, so FilOne holds a central, SP-independent record of what it authorized (space, digest, size), and FilOne is the FilecoinPay payer, so the real backstop is reconciling the on-chain dataset size/accounting against the authorized total (which needs no per-piece CommP↔sha256 identity) and halting the rail on divergence. That covers inflate-to-earn; it doesn't by itself cover byte-substitution, but that's always been the case for this deployment configuration requiring mnual download and verification by a trusted party.

[bajtos]

From what I remember, the TX adding a new root to a PDP ProofSet must contain extraData signed by the Filecoin Pay payeer (FilOne). This extraData also contains the CommP of the added root. In other words, the SP cannot add arbitrary roots to PDP behind our back; every such addition must be signed by FilOne's wallet.

Also, since PDP is using CommPv2, we know what size we are signing over when FilOne is authorising a PDP addRoot transaction.

The point I want to make: I think we don't need to compare the total PDP proofset size against the authorised total, because we should be able to verify the size on a per-object basis.

I don't fully understand how Forge interacts with PDP, so I may be wrong.

[alanshaw]

I'm starting to wonder if we even bother with a minimum size...

While there's probably some aspect of churn related to size, my suspicion is that time is the dominant factor for churn. In other words, the longer a piece lives for, the less likely it is to be deleted.

What if Piri just put every piece on the chain as is, but had a background process to aggregate old pieces together? It allows new pieces to churn easily with minimal cost, it only aggregates pieces that are unlikely to be removed (and thus reduces re-aggregation work/cost), and it keeps the total number of proving roots low.

We can also implement this in 2 steps. Step 1 is just put every piece on chain as is. Ship it, then...Step 2 is add the generational aggregator background process.

[frrist]

This is the comment thats reshaping my thinking, and I've written the full version up as a top-level note ("Should Piri do cross-object aggregation at all?") — let me anchor the discussion there, but the short version: I think you're right, and I'd go further than "maybe no minimum" to no cross-object aggregation at all. Your "time dominates churn" prior is backed by the gas RFC (on-chain cost favors bigger pieces at every churn level, §3.2), so the only reason to aggregate is throughput/root-count, not cost — and if the contracts can take the piece volume, the cleanest answer is one object = its own piece(s), no co-tenancy, deletion always O(1).

In that world the two steps map cleanly: Step 1 is object=piece (no floor) — register everything as-is; Step 2, the generational background aggregator, becomes the optional relief valve for root-count/throughput, aggregating only old, cold pieces unlikely to be deleted (so it rarely triggers a repack).

Two caveats I'd flag rather than gloss:

1. "every piece as-is" maximizes piece count, which makes Miro's ~11M-txn / send wall worse for a heavy tail of tiny objects — so whether we can do
   this at all depends on the contract-practicality question we need to raise with FilOZ in the top-level note (and on raising the extraData cap / sharding across data sets).
2. Generational aggregation reduces the live-root count, but because nextPieceId is monotonic it doesn't undo the proving-depth ratchet, and registration txns are paid per-object up front regardless — so it's relief, not a free lunch.

Net: this is my preferred direction and I'd ship it iff FilOZ confirms the piece volume is practical; if they say it isn't, that's exactly when the size floor comes back as the fallback.

[bajtos]

I'd go further than "maybe no minimum" to no cross-object aggregation at all.

Sounds great to me! 💯

Generational aggregation reduces the live-root count, but because nextPieceId is monotonic it doesn't undo the proving-depth ratchet, and registration txns are paid per-object up front regardless — so it's relief, not a free lunch.

I am wondering if we can work around the monotonic nextPieceId by creating a new proofset for each generation aggregation run?

1. Flag the current proofset as "gc_pending".
2. Create a new proofset that will be used for new uploads from now on.
3. Wait until all TXs adding pieces to the gc_pending proofset are settled
4. Perform generational aggregation for pieces in the gc_pending proofset
   1. Create a new proofset to hold the aggregated roots
   2. Iterate over all roots in the processed proofset
      • Large root - move to the new proofset as-is
      • Aggregate small roots into bigger ones
   3. Schedule the processed proofset for deletion

Anyhow, I agree generational aggregation is post-MVP (post-M2), so we don't need to worry about it right now.

[alanshaw]

Just to clarify, my mental model of this currently is that a body digest is the hash of the entire object (single part) OR the hash of a root UnixFS node that links to all shards of the object (multi part).

Location claims are created by Piri nodes for each shard, telling us where the shard(s) for an object are.

A (sharded DAG) index is created by Ingot for the object - the index lists the shard digests that comprise the blob. The new nodes property (used only for multipart uploads) carries the root UnixFS node and it tells us the order the shards need to be served for retrieval. See RFC for details.

An object read queries the indexer for the body digest and gets back location claims and an index, allowing retrieval to take place.

What we could do, to remove the indexer would be:

1. Assume all blobs are stored at a specific Piri node URL (configurable probably).
2. Encode the (ordered) shard hashes in the object manifest.

This does however make the MST and blobs undiscoverable without Ingot. This actively prevents (or at least makes it hard) to credible exit.

[alanshaw]

Ah I see, the plan is to store these in the object manifest anyway? What do we get from this? We have to go to the indexer anyway to get the locations...

Consider calling this "shards" to align with what they are referred to in Forge.

[alanshaw]

Of course, we still need to index the MST blocks...

[alanshaw]

Um, so this is basically the reason the indexer exists. Items written to the cache are given ample time for IPNI to catch up (7 days) so this shouldn't be necessary.

That said a belt and braces approach might be a good idea. We have a library for getting notifications that an IPNI chain has "done a sync" you might want to take a look at: https://github.com/storacha/go-libstoracha/blob/main/ipnipublisher/notifier/notifier.go

[alanshaw]

Suggested change

	mints a fresh per-space location claim.
	issues a fresh per-space location commitment.

[alanshaw]

I guess so. I just don't know why you wouldn't use an index? You have to go to the indexer anyway and we should probably try to put as little information in the object manifest as possible as we don't have a good garbage collection story for the calalog.

If we ever want to access that object outside of the catalog then it is impossible - you can't ask the indexer about that object CID. 🤷‍♂️ IDK maybe not important, just seems weird to break compatibility here.

How about as a compromise we don't create indexes when the object is single part, but if there is > 1 shard then we do?

[frrist]

Should Piri do cross-object aggregation at all?

A theme runs through a lot of these comments, so I want to pull it out here and link the individual threads back to it.

The assumption I want to challenge. Piri has aggregated blobs into larger on-chain pieces since it was first written, and I think we've all just assumed that was necessary — for gas/throughput reasons — without ever actually stress-testing it. Reading this review, it struck me that

almost every hard problem in this doc traces back to one thing: cross-object aggregation.

Deletion complexity, compaction, "never repack," proving over pieces nobody's paying for, telling uncollected garbage from a fabricated piece, the GDPR compaction window, the size floor itself — they exist only because we pool blobs from different objects into a shared piece. Remove that and the whole cluster collapses.

The proposal (the simple world). Drop cross-object aggregation. Store each S3 object as its own content-addressed blob(s): one on-chain piece if it fits under a ceiling, or a handful of object-owned pieces if it's larger (split at the proving-code ceiling; per-part for multipart, which keeps us compatible with the encryption RFC's per-part envelopes). No floor, no co-tenancy — a piece belongs to exactly one object. Then:

• Deletion is ~O(1) per object, always — delete the object's piece(s), no survivors to re-hash, no compaction, no SLA window to chase.
• The garbage-vs-fabricated-piece question splits cleanly: garbage (our own dead bytes from deletes) is honest-operator GC; fabricated pieces are adversarial and can't be caught by Ingot's blob_refs (sha256↔CommP isn't verifiable — it's a UCAN attestation and the SP runs Ingot, so it can't audit itself), so that one anchors on FilOne-side accounting (Sprue's central accept log) + payment leverage. Still an open problem, but a tractable, correctly-anchored one.
• Per-tenant attribution gets cleaner, and the entire floor / "Regime B" compaction machinery in this doc just disappears.

The catch, and the real point: this hinges on a contract-practicality question we don't have the answer to. Object=piece maximizes piece count, so it only works if the PDP contracts and proving can take that volume in practice. We've never actually asked, though we meant to at the FilDev summit ( 🤦 ) The open questions for FilOZ / FOC — as the experts:

1. Is there a practical ceiling on the number of pieces in a single proof set? Where does it bite first — registration throughput (the addPieces batch is no longer contract-capped as of FWSS v1.3.0 — it's now bound by the FVM PiecesAdded event-size + per-tx gas, so what's the measured pieces/tx?), the monotonic nextPieceId proving-depth ratchet, the MAX_ENQUEUED_REMOVALS=2000/period delete cap, or the view-getter scans?
2. How many proof sets should a node maintain? Today one Piri node = one data set, for operational simplicity — but every data set is a proof the node has to produce each period, so N sets = N proofs/day and it shifts the node's SLA, proving burden, and most importantly cost via extra gas. Is a cap on pieces-per-set + sharding across multiple sets the right move (it would help delete throughput)? If so, what's the right number?
3. What's the registration-throughput ceiling now that you've removed the extraData cap? As of FWSS v1.3.0 / PDPVerifier v3.4.0 the addPieces byte cap (EXTRA_DATA_MAX_SIZE / MAX_ADD_PIECES_EXTRA_DATA_SIZE) is gone — the limit is now the FVM PiecesAdded event-size + per-tx gas. A measured pieces/tx (and whether that event-size limit is worth lifting at the protocol level) is what we need to size the sender fleet for "many small pieces."

The decision rule. I want to build around the assumption that object=piece is viable, because if it is, this gets dramatically simpler. But I'm treating it as an assumption to stress-test, not a settled fact:

• If FilOZ confirms the contracts can take it → we drop cross-object aggregation; the floor and compaction in this doc go away; Alan's generational background-aggregator becomes an optional relief valve for root-count/throughput (it only ever aggregates old, cold pieces unlikely to be deleted, so it rarely triggers a repack); and Miro's multi-dataset sharding is how we scale it.
• If FilOZ says there's a hard practical limit → we keep the size floor exactly as this doc outlines, as the documented fallback.

So the floor stays in the doc — but as the fallback, not the default. The default I want to pursue is "no cross-object aggregation," pending FilOZ telling us whether the assumption holds. This is also why I'd treat the aggregation strategy (and, relatedly, the indexer scope) as open in both this doc and the appliance plan until we have that answer.

How this relates to the S3 sharding RFC. I think these are two halves of one model, not competing proposals. The RFCs flat-file sharding strategy is the object→shard half: for each PutObject/UploadPart it shards data at 256 MB into ordered, content-addressed blobs (I think we can tune this limit as needed). This thesis is the shard→piece half: each of those shards, being its own object-owned blob, becomes its own on-chain PDP piece. The join is a single number — the RFCs 256 MB shard threshold, this thesis's blob ceiling, and Piri's MaxMemtreeSize = 256<<20 (pkg/pdp/proof/proof.go:114) are the same constant for the same reason (the commP memtree is built in RAM, for now, we will be improving this via code from Curio implementation). So under no-aggregation, shard == blob == piece, 1:1, and per-object deletion is per-shard == per-piece == O(1). The thesis is therefore the Shard RFC plus one line: each shard is its own piece. And because retrieval addresses bytes by digest, the aggregation decision is invisible to its read path entirely — the only thing the two halves must negotiate is the multi-shard ordering record (see the §8 / arch:467 thread).

[Peeja]

question: Can you say more? I didn't follow this line at all.

[frrist]

Yeah, sorry. This is a very subtle reference to the fact that Piris RemoveRoot method is incomplete.

[Peeja]

question: I didn't follow this either. I know what the nextPieceId is, but I'm not following why it's mentioned here, and I'm having trouble parsing "must not drift proving cost or the chain-reconciliation view-getters into trouble at high churn". Must not drift it into trouble?

[frrist]

Whoops, yeah I need to clear this up. What I am trying to indicate is that some methods on the PDPVerifier, and iirc a view contract over it will iterate from N->nextPieceId (where N = 0, and sometimes > 0). And since nextPieceId only ever increases, even with deletes, as the dataset churns iteration will become prohibitivly expensive over time.

[Peeja]

question: I'm not sure what that means. What's a "sender fleet"?

[frrist]

another whoops 😅 this leaked in - its a separate idea in Piri I have been playing with that fell out of this whole exploration, will provide a better definition and motivation for it in doc.

Basically its the idea of multiple blockchain accounts Piri submits transactions from in parallel, so registrations aren't all stuck single-file behind one account's queue. i.e. an account's messages must apply in strictly increasing nonce order, one stuck transaction stalls every higher nonce behind it — an underpriced tx during a gas spike, a revert, a dropped message, a bad gas estimate. At millions of registrations under variable base fee, that single-file fragility is the problem. I might be overthinking this here, but seems like a real issue that could popup at high throughput.

[Peeja]

question: What are our knobs to adjust in response to signals? Does instrumenting after launch still give us room to respond to observation?

[frrist]

Currently we have no knobs, reaction would be via an operator re-configuring their node - say for a lower floor (if we even go that route), different aggregation threshold, re-pack strats, etc. Or via an update we ship, preference for the former.

[Peeja]

Gotcha. Let's still note what those "knobs" are, even if they require work from the operator or writing new code. In other words, what can we safely change after we've started, contrasted with decisions we have to make up front that will be baked into, say, data that's hard to migrate.

[Peeja]

suggestion: Let's reuse a noun here instead of introducing "body", to clarify what we're talking about. This is the blob digest, right?

[frrist]

👍 yes the blob digest.

[Peeja]

thought: Two things:

• The stream will also encrypt, yielding a FEE envelope of the blob on disk and a region-wrapped CEK to store in the DB. Doesn't necessarily need to be explicit here, but I want to make sure that's in your head too.
• It sounds like this is writing the whole thing out to disk before moving on, but we can upload each part blob as it's complete on disk. I had been thinking of this as bounding the space Ingot needs on disk, because we can throw out blobs after they're sent to Piri, but it turns out we want to hold onto them anyway for read-after-write and as a cache, so…that's less important. 😛

[Peeja]

question: Are these sizes in plaintext, or in FEE envelopes?

[Peeja]

question: …🦵❓

[Peeja]

question: The entire manifest is held in RAM, right? How pathological does a bucket have to be for that to be a problem?

[Peeja]

question: In other words, for credible exit, multipart manifests need to exist in the space somewhere and not just in a Postgres DB?

[hannahhoward]

This is broadly excellent.

My only hard change is I'd like to have a pretty serious conversation about my proposed alternate form for doing versioning, which I think more closely mirrors the domain model for how S3 versioning is supposed to work.

Beyond that, is just amazing well thought out.

I do want to call attention to the fact that some of the most thinking goes into questions we may not need to answer immediately:

• The first version of deletion is just crypto-shredding the regional encryption key
• The absolute top level goal in R0 is standup an end to end Forge provider in FilOne. No versioning, no real deletion, no various other things.

So my main direction is is focus on "how can I deliver this incrementally to meet R0 (which is just staging and can be completely wiped), R1 (which needs to be data format complete, but need not be feature complete), R2 (where we really want everything working)

[hannahhoward]

I want to second this and make a concreate proposal, after honestly rabbit holing an entire day of this design and how S3 versioning is supposed to work.

Before I get to what I propose, let me cover what I learned about S3 Version:

In particular: in S3 version id is different from its ordinal order:

• seq — the per-bucket monotonic ordinal (buckets.next_version_seq). Its only job is ordering (recency). Internal; never leaves the server.
• version_id — a string, the S3 client handle (x-amz-version-id). Its job is identity. It's opaque, and for suspended/unversioned writes it is the literal string "null".

Unfortunately this distinction is not just an information leaking concern.

For numbered versions they can coincide (version_id can just be seq rendered, information concerns not withstanding). They diverge for the null version: its version_id is frozen at "null" by S3, but it still has a real ordering (a seq).

Blame S3 for a wildly complicated concept.

invertedVersionId collapses the two — it's derived from the ordinal but handed back as the client version_id — and the null version is where that breaks: its id has to be "null", not an encoded ordinal, and there is only ever one null per key, replaced in place (verified against real S3 - I encourage you to make a bucket and mess around with turning on versioning, suspending and re-enabling multiple times etc). Keeping seq and version_id as separate fields makes that case fall out instead of needing a sentinel.

Ok now to the concrete proposal built on @Peeja 's idea:

1. Top MST stays keyed by the plain object key
2. Each leaf is:
   node = {seq, version_id, CatalogCID}
   leaf = { current: node, prev: MST<seq -> node >, null_seq?: seq }

current gives the fastest current path — GET / ListObjects read one leaf, no descent. prev is a per-key sub-tree of older versions keyed by seq; because it's scoped to one buckey key, the prev key is just the ordinal — no fancy error prone string math to produce an object-key + version composite; "order an integer" is all that's left. null_seq is to track if there is a null version id in the prev tree -- because of S3's very unique way of having exactly one "null" in the version history.

Write rule — depending on versioning state, push the displaced current into prev or discard it; and since S3 allows only one null per key, creating a new null also evicts any existing one:

retain = (versioning == Enabled) || (displaced.version_id != "null")
if retain: 
   prev = put(prev, displaced.seq, displaced)
   if displaced.version_id == "null": null_seq = displaced.seq
else: 
   gc_release(displaced)
if new.version_id == "null":
   if null_seq: prev = remove(prev, null_seq)
   null_seq = undefined
current = new

Mode	Displaced	retain
Enabled	numbered or null	push (S3 keeps a re-enabled null as noncurrent "null")
Suspended	numbered	push
Suspended	null	discard (replace the null in place)
Unversioned	null	discard

prev is written only when we retain history — always for Enabled, never for Unversioned, only at the numbered→null boundary for Suspended. The lingering null is often noncurrent (it lands in prev when an Enabled write displaces it), so the leaf tracks its seq (null_seq);

Optional (to speed up lists): lift is_delete + render fields (etag/size/last_modified) into the leaf and prev entries, so ListObjects/ListObjectVersions render straight from the tree without fetching manifests, still verifiable. Left out above to keep the base shape minimal.

Cost: Enabled writes do a prev insert + top resplice each time — more blocks per write than a flat splice, but scoped to keys you're actually versioning. One thing to confirm: rebuild assumes logstore keeps per-op records / op-roots that point at created manifests, not just the latest forge_root_cid.

[hannahhoward]

MinIO or "storage backend" -- I do think there's a world where some people just wanna throw ZFS at this.

[bajtos]

I finished reading the version from yesterday and have a few more comments. Next, I'll read the discussion and the updated version.

[pyropy]

What happens if the objects are less than 8 MiB in size? Should they get padded or are they going to be aggregated with unrelated objects into a shared piece like they are aggregated today?

[pyropy]

question: Is Sprue aware if Ingot node also doubles as Piri node? Will it prioritize such node for blob slot allocation over the other nodes since there's no moving bytes across the wire?

[pyropy]

Is Ingot going to check UCAN delegations for local reads?

[bajtos]

The proposal (the simple world). Drop cross-object aggregation. Store each S3 object as its own content-addressed blob.
The catch, and the real point: this hinges on a contract-practicality question we don't have the answer to. Object=piece maximizes piece count, so it only works if the PDP contracts and proving can take that volume in practice.

FWIW, Filecoin Onchain Cloud uses exactly this design (or at least used to use it back in Q1 2026): one file = one PDP piece. So PDP & FWSS was designed with the assumption that there will be one piece per file, with files as small as ~32 bytes.

Of course, the question is whether FOC has ambitions to handle the storage scale FilOne is aiming for, or whether their design works for FOC only because they operate at a much smaller scale.

[frrist]

Where this PR is going — a stake in the ground, not a contract

Thank you all — this is some of the most thorough review I've had, and the discussion has genuinely reshaped the design (the cross-object aggregation thread especially). It's also gotten big enough that the real forks are getting lost in the volume, so this comment is to set the frame for how we land it and what's actually still open.

The ask: I want to merge this as a living design baseline — the agreed place to build the next increment from — not as a finalized design that merging signs everyone up to. Several of the biggest questions here are explicitly unresolved (and some can only be answered by FilOZ/FOC, not by us). Landing it means "this is the shared starting point and the open questions are written down," not "every decision is final." It should be a stake in the ground near where it needs to be; we'll move it as we learn.

How to read the doc going forward

I'm tagging every major area with a decision status so it's clear what merging does and doesn't commit us to:

• Decided — settled; build on it.
• Provisional (default) — we're building on this assumption; it's reversible and the fallback is named.
• Open — explicitly not settled; the comment notes who owns the answer and what unblocks it.

Status of the major areas

Area	Status	Who owns / what unblocks
Cross-object aggregation → object = piece	Provisional (default)	Open-pending FilOZ/FOC (3 questions below) + a scaled proof-of-capability. We build assuming object=piece; the size floor is the documented fallback, not the default.
Size floor / Regime-B compaction	Fallback only	Activates only if FilOZ says the contracts can't take the piece volume. Stays specified for that reason.
Versioning data model (§3)	Open — design call	@hannahhoward + @Peeja — nested-MST vs composite-key, seq vs version_id. hannah's one hard change-request; a thread can't settle it.
Indexer: keep (target) vs drop (R0/R1)	Open — design call	@hannahhoward + @alanshaw — reconcile with the appliance plan.
Milestone scoping (R0 / R1 / R2)	To be added to the doc	Me — map deletion=cryptoshred@R0, versioning@R2, instrumentation-tuning@R1, etc. (hannah's incremental-delivery ask).
Deletion / GDPR compliance story	Open — needs a subsection	Coordinate with RFC #9 owners — cryptoshred + the FEE two-copy recoverability problem + the 24h window.
Read cache, instrumentation timing, Ingot-in-Piri packaging	Provisional	Minor/reversible; tuning is R1-gated.
S3 layer, catalog/data plane split, retrieval addressing, write/multipart/latch	Decided (stable)	The core of the doc; this is what merging actually establishes.
Deduplication	Decided	Low value once encryption is on (fresh per-object CEK); machinery kept for re-PUT of identical objects.
Postgres schema (Appendix C)	Non-binding pseudocode	Illustrative only; names/types/constraints get designed at implementation. Schema-detail comments captured for then, not blocking now.

Path forward

1. Unblock the critical path (now, in parallel). Send FilOZ/FOC the three aggregation questions; start the scaled register→prove→delete proof-of-capability and finish Piri's schedulePieceDeletions signature (wired but unsigned today). This is the cheapest way to find out if object=piece is dead on arrival — everything downstream waits on it.
2. Two design calls. (a) Versioning model with hannah + Peeja → update §3. (b) Indexer + R0/R1/R2 scoping with hannah + alan → map the doc to milestones.
3. Write the deletion/GDPR-compliance subsection, coordinated with RFC Architecture migration (Phases 1–6): object-aligned blobs, sync durability, dedup/reference-counted delete, S3 correctness + multipart #9.
4. Apply the cleanup threads (schema naming, ETag rationale, "shards" naming, NUL-encoding edge case, clarity edits) as non-binding revisions.
5. Land as the baseline; remaining opens become tracked issues.

The three questions for FilOZ / FOC (the gate)

Object=piece maximizes piece count, so it only works if the contracts can take the volume — which we've never actually asked. (@bajtos notes FOC already does one-file = one-piece, which is our leading hypothesis.)

1. Is there a practical ceiling on pieces per proof set, and where does it bite first (registration throughput, the monotonic nextPieceId proving-depth ratchet, the MAX_ENQUEUED_REMOVALS=2000/period delete cap, or view-getter scans)?
2. How many proof sets per node should we run — is a pieces-per-set cap + sharding across sets the right scaling move, and at what number?
3. What's the registration-throughput ceiling now that FWSS v1.3.0 removed the extraData cap (limit is now the FVM PiecesAdded event-size + per-tx gas) — measured pieces/tx?

Routing your comments

If I don't reply to a thread directly, it's in one of the buckets below — accepted as suggested, captured for implementation, or answered here — and I'll resolve it as I apply the change. That's a deliberate ack, not a miss; say so if you want any of them reopened for real discussion.

• Design forks (versioning, indexer/scope) → being taken to the two calls above; please hold those threads for there.
• Clarifications (the O(log) terms, ETag, "blob digest" naming, data-layer ownership, credible-exit definition) → folding into the next revision; thank you.
• Cleanups / schema details → applying as non-binding; the schema appendix stays pseudocode until we design it for real.
• @alanshaw on RFC repo: I'd like to keep these in ingot for now as a living baseline (they move fast and are ingot-specific), and promote the aggregation decision to the RFC repo once FilOZ answers and it's a cross-team decision rather than an open question. Will do that as a follow-up.
• @hannahhoward on CHANGES_REQUESTED: both your items are on the path — versioning gets its own call, and I'll add the R0/R1/R2 incremental-delivery mapping to the doc. Flag if you'd rather those land before merge or as immediate follow-ups.