On-chain Retrieval Expectations

[willscott]

Background

There is not a cryptographic protocol guaranteeing retrieval in the same way that there is with storage. The retrieval side of the Filecoin "storage+retrieval market" remains as a consequence undeveloped.

What this has led to is two primary issues:

1. The definitions of retrieval that are being used today are imprecise.
   a. fil+ discussion on retrievability does not specify what retrieval actually means.
   b. fil+ recent proposal for ac bot specifies retrieval via a "retrieval bot score" minimum, which does not specify the actual bandwidth provisioning expected of SPs.
2. The retrieval expectations of data on the network today are unknown. As a client, while I may have an agreement with a storage provider where i expect to reserve some amount of their bandwidth for retrievals of my data, I have no insight into what expectations there are of other clients. Without a common location where those other expectations can be made known, it is impossible for a client to know if an SP can or is meeting their obligations.

Problem

There is not an on-chain record of retrieval characteristics of stored data. without community buy-in, we reduce the ability of any l2 solution to succeed in promoting reliable retrieval on top of filecoin.

Proposal

The full proposal is linked here.

We are proposing 5 'tiers' of retrieval that a deal is classified into:

1. Offline - Data that is transferred via physical media. This data is not expected to be retrieved over the network.
2. Archival - Data that is expected to be retrieved infrequently. This data is only expected to need enough network bandwidth to be able to ensure replication. The most common data replication policies on filecoin today at 5x and 10x copy replication, to ensure that data remains even when some copies are unavailable. In order for contracts or users to ‘heal’ data replication policies in the face of failures, they would potentially need (n-1) copies of the data to be re-replicated.
3. Online Data - Data that is expected to have periodic retrieval by an ACLed list of users. This tier allows for negotiation of higher retrieval service levels without committing to make the data available to the network at large.
4. Public - Data that is expected to be openly retrievable by anyone. This tier allows for negotiation of higher bandwidth and low latency provisioning.
5. CDN - The details of this tier are still being worked out, but we reserve this nomenclature for a higher tier of retrieval service that equates to the expectations for bandwidth and latency that would be found at CDN service offerings.

This will reduce the implementation space, to better allow providers to provision, and to not have to guess at provisioning levels when most clients will not know exactly what level they will need apriori.

We encode these tiers in a deal proposal (or equivalent in a DDO world) so that there's a common location where clients and the network as a whole can understand an estimate of how much retrieval providers are signed up to provide based on their current storage load.

Including an initial tier estimate in the ask negotiation (which happens off chain, but is reflected in the need to include it in the proposal) means that SPs can better differentiate their pricing and charge against the bandwidth request associated with stored data.

[anorth]

Most of this doesn't quite make sense to me, although I do think that some parts could after some iteration and clarification.

One problem you point to is that discussions about retrievability are imprecise. I can see that as a problem, but don't see how consensus protocol changes are a great way to resolve that. Would the first step not be to upgrade the discussions to use precise concepts of bandwidth, latency etc? Similarly for clients negotiating storage deals - if the deal is to include expectations about bandwidth, latency, availability etc, they should be explicit in the deal.

I mean deal in a very general way here. The built-in market actor specifically is very limited, and it's quite unfortunate that it is a built-in actor and currently holds a privileged position in data onboarding. FIP-0076 and others aim to resolve this so that user-programmed smart contracts are equally capable of brokering and holding the metadata for a deal. Such markets could define any terms they want, including expectations for retrieval frequency, bandwidth etc. I can clearly see the value in an FRC defining some common standards around this across markets. (I would suggest that it directly address the attributes like bandwidth, frequency etc rather than bundle into a few categories, but an far from expert in user needs here). But change to the built-in market actor is expensive, disruptive, and limited. The linked FIP draft implies change to it by changing the DealProposal type that it deals in, but lacks detail around the necessary migrations etc.

Your point 2 identifies a problem that I'm not convinced needs to be solved (and certainly not at the consensus level). I think you're saying that a client needs to know about all of an SP's other deals with other clients in order to know if an SP can meet their obligations. This seems generally impossible, not solved elsewhere, and not much needed. An Amazon/Google/Azure client has no insight into those services' other obligations, and its unthinkable that they might know about all other customers. Whether the service meets client expectations is a question answered at the time clients request their data. I think it quite unlikely that any of these services is provisioned for all their clients to retrieve all data stored with them even on a schedule that would be quite modest for any individual client and data. Can AWS export all their data in even a year? Who knows? The impossibility of knowing whether a service can meet some future obligations is not relevant to their clients.

I think it's impossible anyway, even given a public blockchain. SPs will be able to take on arbitrary obligations beyond those captured by any central mandated standard. If squeezed, one would expect them to honour the expectations that are most economically valuable to them, but that may not be the standardised ones.

I can see that discussion about retrieval is imprecise. I also do agree that smart contracts that are acting as brokers, markets, or other deal-related things on chain should probably have some explicit encoding of expectations of retrieval (and other things). The problems I see is that those on-chain brokers don't exist or aren't doing this, and that even if they were there would be value in a standardised representation. However, the built-in market isn't the right place to implement this (or anything, if we can avoid it). Almost everything in this discussion is application-level concerns to be addressed by user-programmed contracts and conventions among them.

[willscott]

We've tried an opt-in L2 path towards retrieval over the last couple years with minimal success. Clients who do put in requirements are ending up feeling the need for external contracts, rather than feeling confidence than the network will meet or that they can even specify their expectations for data transfer in deals today. There have been a number of experiments run, including pools of vetted providers matched based on expected thresholds of service, selection based on past performance, and direct identification of individual providers. Individual client observed performance has been insufficient in several cases, leading to cross-client reputations emerging in both RIBS and Fil+ contexts.

• I'm happy to flesh out the migration needed for extending DealProposal. One caveat noted in the initial draft is that we are in a tricky place in terms of interleaving this proposal with FIP-0076. The mechanism for specifying retrieval expectations will be different depending on whether we are in a DDO world, and it would be great to get clarity on interleaving will take place and should be focused on.
• I don't think I agree with the comparison drawn to existing cloud systems. The Filecoin providers are not nearly as asymetric in scale as existing monolithic clouds, and while we cannot measure if AWS overall can export all of it's data (which would be a comparable to filecoin as a whole exporting all it's data), One can as a client observe the peering connectivity of an individual AWS data center to calculate limits on how much data stored in one availability zone could be potentially recovered in the event of a disaster.

At a higher level:

• One reason I want to define and push for this in the framing of a FIP is because FIPs are our current governance mechanism. There has been no community engagement or discussion generated by any FRC so far, so it has unclear value to submit artifacts in that format. Pressing the community to engage with a FIP will help continue the discussion we need to have as a community about what we're going to do in order to get retrievals where they need to be.
• I think if we as a network really believe that retrievals are an important part of the value-unlock for accelerating filecoin, we need to figure out what we're going to do as a network to incentivize reliable retrievals. Leaving it as bolted on outside of the core protocol is a reflection of the value the network is currently placing on this problem.

[f8-ptrk]

i am not sure what this proposal aims at. i mentioned it in a, now deleted, comment before:

there is no course of action on L1 to enforce this. it is technically impossible to prove retrieve-ability.

as alex mentions here https://github.com/filecoin-project/FIPs/pull/862/files#r1405505256 the FastRetrieval field of the current deal proposal format is already questionable to be kept on L1 - there is neither a way to directly incentivize it (right now, PoA comes to mind) nor a L1 way to punish miners that do not honor this promise. from the L1 side this is, most likely, expensive to be stored and in the end dead data. the fil+ program starts trying to somehow punish the fil+ clients for not enforcing it (and retrieve-ability itself) off chain with various attempts - bots, questionable attempts of nonsensical repeated retrieval, etc. - this is a separate discussion, but i see the same problem arising with this proposal: trying to enforce something that technically cannot be enforced on L1.

there is a reason clients use off chain agreements: it can, if needed, be litigated (lets be generous and classify whatever fil+ does as litigation...) over to enforce them.

if there is a way to enforce (incentivize/punish) retrive-ability on L1 in a mathematically sound way i am happy to hear about it. everything i have seen so far is far away from what one would want to base a L1 consensus on.

all this doesn't mean that i do not agree with the general idea behind this discussion - it for sure makes sense to have a discussion about how to categorize retrieval expectations. but i also agree with alex that the proposed tiers are way to unspecified in their current form. and i honestly do not see the community agree on tight enough specifications for this to be even a FRC. the pressing problem i see right now:

• there is no way to not specify this, it needs at least a tier that basically says undisclosed/unknown/do not care/none of your business/blank/... - a sensible default of sorts
• expectations are not static. clients need to be able to change the recorded expectations if they change. this will add some kind of challenge response scheme to the whole thing that will make it inherently more complex (same is true for FastRetrieval btw.)

in short: this is all L2, not L1

(when i say L1 i mean the filecoin L1 we have and will have. there are for sure other projects that do something on their L1 to do stuff in the data availability direction, the Binance storage chain thingy comes to mind here)

[willscott]

Related to this FIP and the discussion around the right places to push for improving the reliability of filecoin retrievals, we're starting up a biweekly synchronous meeting to track active threads and keep ourselves accountable. If you're interested in participating, the meetings can be found at https://lu.ma/retrieval-wg

[jennijuju]

Have we considered to add the tiers to a user deployed actor rather a builtin actor? With the recent work from fvm team towards actor upgrade - I think that gives a more flexible way to update teirs as new requirement evolves

[f8-ptrk]

as i read the proposal there is an expectation that having this implemented in a build in actor, eg. the market actor via deal proposal standard, will give this some kind of authority.

I think if we as a network really believe that retrievals are an important part of the value-unlock for accelerating filecoin, we need to figure out what we're going to do as a network to incentivize reliable retrievals. Leaving it as bolted on outside of the core protocol is a reflection of the value the network is currently placing on this problem.

i do not see a course of action now (i do not see one in the future either) to enforce this via L1 - so i agree with L2 being the right place for this. evm or native fvm doesn't matter in the end.

[laudiacay]

Have you guys watched my talk from ipfs thing in Brussels because I have a proposal for how to enforce and incentivize retrievals

Banyan will implement the above next year

But I think NONE of this info should be encoded into deal proposals on actor-level... we should do retrieval SLAs composably on the FVM... it should be an "added layer of agreement" not a "different deal"

[The-Wayvy]

under-rated comment

[The-Wayvy]

Don’t over-engineer the consensus algo.

This can be done at the application layer.

aggregators like Lighthouse are capable of serving retrievals just fine

[willscott]

previous examples like web3 storage have already demonstrated that this doesn't work if you can't offload cache to backing storage and have some expectation of being able to get it back.