Announce PieceCID to IPNI

[rvagg]

For piece discovery, there's a desire to announce PieceCID to IPNI and it would make sense (particularly for markets v2) that this is done for both PoRep and PDP pieces and that it's just done in the process of onboarding a piece. Ideally the second step of doing the full piece indexing (CAR & PoSDI indexing) is available too but is an optional step. But this means two different types of advertisement I think. We are registering a new piece transport codec for this purpose (Curio currently uses just metadata.IpfsGatewayHttp{} but we'll also make a metadata.FilecoinPieceHttp{} for the pieces).

Curio is now using the piece & size in the context ID for the advertisement of the IPLD blocks, this is to enable Spark to fetch piece lists for providers for retrieval sampling.

In talking to @LexLuthr we were in a quandary as to how to support both of these things at the same time. Ideally we always want to report PieceCID to IPNI, with IPLD block reporting is an optional extra.

1. Do they have to be separate advertisements? I assume we can't bundle them into one advertisement?
2. What ContextID do we use for piece reporting? Do we need to have something different to the IPLD block reporting, should it be the same, or...?
3. Alternatively, since we have PieceCID (CommPv2 compatible too) in the ContextID for existing advertisements, can we do a reverse mapping lookup via IPNI? i.e. the reverse of what Spark is doing now so we can go from PieceCID (it'd have to be CommPv2 since the size is in there) to provider(s) with an IPNI query?

[bajtos]

Let's discuss first what we use sampling for and why we need PieceCIDv2 in the ContextID.

In the world of IPLD CAR files as Pieces stored in PoRep deals (f05 or DDO), there is no on-chain link from PieceCID (CommP?) to the CIDs of individual IPLD blocks stored in the CAR file. The requirement to construct ContextID from PieceCIDv2 allows Spark to link on-chain deal metadata (PieceCIDv2) with IPNI advertisements and leverage the recently introduced reverse-index sampling API to get a random IPLD block inside the given PoRep deal.

These IPNI advertisements are using IPLD CIDs & IpfsGatewayHttp transport, and there are many entries in every deal.

If we want to put CAR files into PDP deals, then I agree we should extend this architecture from PoRep to PDP deals, too.

When it comes to Piece retrievals:

• Curio currently does not advertise Piece retrievals to IPNI.
• Assuming Pieces are atomic units and the only supported retrieval operation is retrieving a byte range of a given PieceCID:
  • There will be only one IPNI entry in each advertisement (the CommP CID (baga...)), and the advertisement will specify FilecoinPieceHttp transport.
  • Spark doesn't need reverse-index sampling. It can use PieceCID (CommP) from the on-chain state in the IPNI query, and use the PieceSize to pick a random byte range from the piece.

If we agree on this, then it does not matter what ContexID value Curio uses for the advertisements of Piece retrievals.

At the same time, I agree it would be nice to keep constructing ContextID from CommPv2, if it's feasible.

My knowledge of IPNI advertisements and data schema is limited. With that being said:

Do they have to be separate advertisements? I assume we can't bundle them into one advertisement?
What ContextID do we use for piece reporting? Do we need to have something different to the IPLD block reporting, should it be the same, or...?

It would be best to consult with @masih on this.

We want to advertise that IPLD block bafy1 can be retrieved over IpfsGatewayHttp and Filecoin Piece baga1 over FilecoinPieceHttp transport.

I think the only way to accomplish this is by having two different retrieval providers and thus two different advertisement chains. In that case, it's ok to re-use the same ContextID because the primary key is a pair (ProviderID, ContextID).

Alternatively, since we have PieceCID (CommPv2 compatible too) in the ContextID for existing advertisements, can we do a reverse mapping lookup via IPNI? i.e. the reverse of what Spark is doing now so we can go from PieceCID (it'd have to be CommPv2 since the size is in there) to provider(s) with an IPNI query?

You need to know IPNI ProviderID of the SP in order to user the reverse-index sampling API.

Once you know the ProviderID, you can find SP's addresses at https://cid.contact/providers/{PeerId} (example). There is no need to query the index or sample the reverse index.

Spark finds ProviderID either from miner's libp2p address or from the mapping maintained by Curio in a singleton smart contract.

• Our code: https://github.com/CheckerNetwork/index-provider-peer-id
• The on-chain registry: https://github.com/filecoin-project/curio/blob/395bc47d0f585cbc869fd4671dc05b1b2f4b18c2/market/ipni/spark/sol/README.md
• Curio docs on the topic: https://docs.curiostorage.org/curio-market/ipni-interplanetary-network-indexer-provider#ipni-provider-identification

[LexLuthr]

Well! After reading above, I remembered the most important part. PDP and PoRep do not share a provider ID of any kind. So, a new chain does sounds like a good idea with same contextID. I can bind the new providerID to the f4 wallet which is responsible for SP proof creation. This model will allow having multiple ad chains if required. This will require some changes on Curio's side but this can work. But, for a give providerID and contextID, if a user chooses to publish only the pieceCID then they should not be allowed to change it to normal payload later.

[rvagg]

ipni/go-libipni#267