PDP: Support retrieval of aggregate root CID?

[rvagg]

This is initially two questions, and then a follow-up task if we need to proceed to doing something about it.

What?

PDP pieces are limited to 254MiB, if you want to add a larger piece then you do it in smaller chunks, each of which has their own PieceCID (CommP) and then when you call "AddRoot" you do it with your aggregate PieceCID—this is the equivalent to what we call CommD for PoRep, but for PDP it could be any size where you combine one or more sub-pieces (code to calculate this in Go from a set of sub-pieces can be found here). You have to calculate this on the client size and on the server side and they have to match. For your large PDP piece, this final CommP/CommD thing is the identifier for your whole piece, even though Curio has it stored as many sub-pieces. When Curio communicates with the chain, it only sends this aggregate one, the notion of "subroots" is Curio-local, the chain doesn't care, and ultimately the user probably shouldn't even care in most cases, it's just an implementation detail.

But, the catch is, if the user only cares about this aggregate CID, then should retrieval work with it or should we force the client to do the reverse of the CommD process and retrieve the sub-pieces? Do they need to then store and record those sub-pieces? They can't get them from the chain or any chain indexing service, but they could (potentially) ask a Curio server for them (and verify that the aggregate CommP calculation === their CommD). Then they can get https://.../piece/baga... for each subroot and then their final piece is the aggregation of all of these. In that process you also get a form of incremental verification, because you get to hash CommP for each subpiece along the way and fail "early" if it doesn't match.

Or, should we just support full aggregate CommP retrieval for potentially very very large pieces that are not incrementally verifiable but are identified by what the user has in their possession and what's on chain?

Should FilCDN be expected to support aggregate CommP retrieval? Or should it only support subroot retrieval; and if so, how does it know what the subroots are, does it need to query a Curio server when it finds a withCDN root just to find out if it's comprised of many smaller pieces? Should an SDK be capable of doing the translation for you such that a download() operation is able to look-up the subroots before streaming and then fetch pieces one-by-one and do incremental verification along the way but present you with a singular stream of bytes?

1. Do we support this already?

I don't believe we do by cruising through the code. And we have reports of failures trying to retrieve large pieces from the /piece/ endpoint:

{"level":"warn","ts":"2025-06-19T10:54:26.162Z","logger":"cached-reader","caller":"cachedreader/cachedreader.go:255","msg":"failed to get piece reader from sector","piececid":"baga6ea4seaqnx4gnoeuqjyu7ctmhqtow4nnzukdfuyw3wr5bm73o5vlvbl5mgny","err":"piece cid baga6ea4seaqnx4gnoeuqjyu7ctmhqtow4nnzukdfuyw3wr5bm73o5vlvbl5mgny: no deals found"}
{"level":"error","ts":"2025-06-19T10:54:26.163Z","logger":"cached-reader","caller":"cachedreader/cachedreader.go:262","msg":"failed to get piece reader from piece park","piececid":"baga6ea4seaqnx4gnoeuqjyu7ctmhqtow4nnzukdfuyw3wr5bm73o5vlvbl5mgny","err":"failed to find piece in parked_pieces for piece cid baga6ea4seaqnx4gnoeuqjyu7ctmhqtow4nnzukdfuyw3wr5bm73o5vlvbl5mgny"}
{"level":"error","ts":"2025-06-19T10:54:26.163Z","logger":"retrievals","caller":"retrieval/piecehandler.go:52","msg":"server error getting content for piece CID baga6ea4seaqnx4gnoeuqjyu7ctmhqtow4nnzukdfuyw3wr5bm73o5vlvbl5mgny: failed to get piece reader from sector or piece park: failed to find piece in parked_pieces for piece cid baga6ea4seaqnx4gnoeuqjyu7ctmhqtow4nnzukdfuyw3wr5bm73o5vlvbl5mgny, piece cid baga6ea4seaqnx4gnoeuqjyu7ctmhqtow4nnzukdfuyw3wr5bm73o5vlvbl5mgny: no deals found"}

This piece was created with pdptool doing a chunked upload.

2. Should we support this, but then if we do want to, how do we go about doing that.

If we support this in /piece/ then we're probably going to need a wrapper that is able to find the piece in the pdp roots list and then list its subpieces and stream those together as a response. Not hard, just something to plug into the existing /piece/ cachedreader.

[jennijuju]

each of which has their own PieceCID (CommP)

Why?

[jennijuju]

This was what magik suggested #467

[jennijuju]

It’s unclear if you are suggesting a user is storing one piece or multiple pieces.

The former doesn’t seem like to be a problem iiuc how the mkv2 aggregation task works - @LexLuthr should confirm tho. So for the latter, I think we should look at the curio PoDSI aggregation, and it might be a good idea to have PoDSI in the metadata upon AddRoot too - which is a thing @Kubuxu has suggested a couple times for PoRep

[bajtos]

Thank you @rvagg for the detailed description of the problem! 😍

My perspective is based on a builder that don't want to deal with inner complexities of Filecoin, they want an easy way to store and download large content.

Let's consider this scenario:

1. I have a 2 GB video file I want to stream from Filecoin.
2. I upload this file to PDP.
3. In my video player widget, I want to fetch a byte range of my 2 GB file to start playing the content quickly.
4. Ideally, I also want to verify that the byte range returned by the server belongs to my file. (An inclusion proof.)

As a builder:

• The fact that PDP pieces are limited to 254MiB is an implementation detail of PDP & Synapse that I don't want to deal with.
• I want to give Synapse SDK a ReadableStream with my 2 GB video file and get back a single CID I can use for retrieving my file later. (My mental model is Object Storage - I upload a blob of data and get back a key I can use to retrieve my blob.)
• In my video player, I want to use that single CID to identify the video to play, and fetch chunks as a byte range based on the size of the entire file.
• Content verification for byte-range retrievals is tricky. Some options that come to my mind:
  • Adapt Blake3's BAO format for CommP.
  • @masih's Commpact https://www.youtube.com/watch?v=ndLKjXQ1awI