Direct data onboarding: Onboarding data without built-in market deals (FIP-0076) #730
Replies: 21 comments 50 replies
-
|
current flow -> Direct FIL+ proposal is that right? @anorth |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for this question - drawing out product requirements like this is very valuable. Firstly, please note that the existing processes will keep working as they do today, so there is no need for clients to interact with the chain. However, in order to take advantage of the large total cost reduction here, participants would adopt a new flow. You are correct that, in the basic design here, the datacap allocation message comes from the client account. This means the client pays gas (but much less than PublishStorageDeals). The client doesn't necessarily need to be the one transmitting the message on chain: the could sign it and send to the SP for submission. But they would need to pay gas (perhaps SPs will reimburse this, since they get a large saving out of it). At the cost of a bit more design/code complexity, we could implement a means for an SP to both submit and pay gas for this message on behalf of a client. The flow would then be more similar today, where the client transmits a signed message to the SP for submission (the message being a FIL+ allocation instead of a deal proposal). There are a couple of ways we could do this. How important is it? |
Beta Was this translation helpful? Give feedback.
-
|
OK, I see, no problem. It is good to maintain the status quo user experience without adding burden. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @anorth!
Client payment is our network's top priority. It's essential for the network’s sustainability and overall health. Ignoring it would be a big mistake. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for highlighting this. Let me start out by assuring you that I share the goal of enabling efficient client-paid deals. This is a stepping stone towards that. The Direct FIL+ flow by definition doesn't include payments, but the changes come this close 🤏 to enabling efficient deals with client payments too. First let me note that the existing flows will continue to work and support payments. Also, nothing will prevent parties from arranging off-chain payment (or any other externally-enforced terms), while still enjoying the gas savings of avoiding the built-in storage market. I know we all want both cheap and on-chain payments, but these alternatives can support some uses until we get there. Consider the following matrix of possibilities:
We have expensive deals with on-chain payments today. This proposal enables cheap deals without payments. The first item under "Preparations for future enhancements" is "Data activation notifications to smart contracts". This capability is the missing link to enabling cheap, on-chain payments. It depends on the changes for Direct FIL+, but is separable from them. It does, however, involve some difficult security considerations to get right, so I have opted to exclude it from the initial scope of this proposal for now[1]. Processing on-chain payments, or any other deal terms, is extra work for the chain. It's inevitably going to cost a little bit more than not having payments. Empirically, almost all deals today involve no client payment (but still pay the cost of the built-in storage market); we should at least make them as cheap as we can. The changes for Direct FIL+ will eventually enable deals with payments and other terms at significantly lower cost (~half?) than the built-in storage market, and smart-contract markets will be able to improve and optimise much faster than the built-in one. [1] We could decide to bring it into scope now as one large change, but I think this would add significantly to the timeline of resolving the unnecessary costs for the dominant use case today. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @anorth for opening this! I have a couple of questions after the first quick pass on the technical design:
|
Beta Was this translation helpful? Give feedback.
-
Good question. Either (a) the client transmits it to them (instead of a deal proposal), or (b) they could observe it from chain state directly. To support (b), adding an actor event is a great suggestion. @ZenGround0 has suggested adding actor events to the verified registry independently of this proposal, and that would do it.
Yes. The new
The proposal as written only supports the built-in market (or no market), so it's always
See link to discussion of StorageAsk above. This proposal doesn't actually support alternative markets yet. For a future proposal that does, it's not a consensus thing so not part of a FIP. I think deal negotiation protocols will need to change too to leverage these capabilities, and an FRC would be appropriate when good patterns emerge. Observability of on-chain payment terms depends on what smart contracts will implement; IMO it's too early to standardise anything. Note that off-chain payments are and will remain possible, but obviously outside of FIP scope. |
Beta Was this translation helpful? Give feedback.
-
|
This proposal involves the client submitting a message to create a datacap allocation. In current flows, the SP submits the equivalent message when they publish storage deals. Submitting a message involves holding a token balance to pay gas, and interacting directly with a node; clients don't currently have to do any of that. A workflow where the SP submits all messages could be restored with the addition of a generic delegated execution capability, proposed in #738, or with implementation of explicit "voucher" handling in the datacap actor. |
Beta Was this translation helpful? Give feedback.
-
|
@dirkmc @nonsense, here is the client delegation flow thats proposed |
Beta Was this translation helpful? Give feedback.
-
|
Generic delegated execution turned out to be difficult, so cant be used to wave away this issue. We could support something mirroring the current workflow by explicitly adding support for "allocation vouchers", signed by the client and submitted to chain by the provider (or anyone). This is a relatively independent feature that we could add later. I have removed it from scope for the first direct data onboarding FIP in order to keep the complexity there manageable. We can add it later if needed. |
Beta Was this translation helpful? Give feedback.
-
|
Agree with the scope. For the actual delegation - What about letting the clients adding SP as their |
Beta Was this translation helpful? Give feedback.
-
|
"Governor" is the wrong term, but yes clients that can trust their SP could authorise the SP to spend an allowance of their datacap, in which case the SP could use transfer_from to create the allocation. Good observation. |
Beta Was this translation helpful? Give feedback.
-
|
Omg Idk where that’s coming from - sorry I meant operator lolol |
Beta Was this translation helpful? Give feedback.
-
|
Hey @anorth are you planning to move forward with this proposal and make it a "Draft"? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. It's complex and will take some time, though. |
Beta Was this translation helpful? Give feedback.
-
|
@anorth does this work with non-Fil+ deals? |
Beta Was this translation helpful? Give feedback.
-
|
The same cost savings are available whether or not FIL+ is involved yes. But it depends on what you mean by "deal". Currently the only on-chain mechanism for a deal is the built-in storage market actor, and if parties use that then they clearly must pay the associated gas costs. However nothing will prevent an SP from onboarding data without a built-in market deal, which will enjoy all the same savings. This proposal doesn't immediately create an on-chain alternative to the built-in market, but parties can negotiate off-chain. A subsequent FIP will enable the hooks needed for people to create user-programmed on-chain actors as markets, in which case costs will depend on those actors' implementation. This is the lower-right "future" box in the matrix above. |
Beta Was this translation helpful? Give feedback.
-
|
I personally oppose to changes that are doing plus not minus on L1. The idea of 730 felt a bit hacky. An intrusive patch on an already cumbersome protocol architecture. It complicates L1 with even more unnecessary messages and complexity. |
Beta Was this translation helpful? Give feedback.
-
|
at that time the FIL price was roughly $200. i chose the extreme to make the point. the numbers are for sure way to high. but it exemplifies the problem better then discussing subcent amounts. its way more complex than i make it seem. i agree. but the fundamental problem of [gas usage] / [deal pubish fees] / [FIL market price] interaction being potentially toxic to what we want to do (storing data) is numbers independent [edit] |
Beta Was this translation helpful? Give feedback.
-
|
the fundamental fear here is that once we leave the valley of sadness we are currently in, gas usage picking up to healthy levels, the market price going up - we have been there before!!! it was not funny - we will find ourselves, again, at a point where we have to push emergency upgrades and other stuff that was arguably not thought thru enough for long term sustainability to counter what we can/could/should fix now. see jacobs analysis of max possible onboarding throughput in iceland for example. i should be fixed now, not when it becomes a problem |
Beta Was this translation helpful? Give feedback.
-
|
I think the point i made is independent of FIL price though:
Doesn't gas remain a very small percentage of the overall cost, so while we're reducing that small percentage, it remains a minimal impact on the actual cost calculation that an SP is going to have to make? |
Beta Was this translation helpful? Give feedback.
-
|
@willscott it's complex. the cost for bringing up collateral are hard to denominate - if you get your collateral from glif for example its 15% yearly of the borrowed FIL amount, right now. its subject to changes and will most likely be coupled to other factors like blockrewards in the future the publishing message fees can make up to 50% of all onchain fees miners have to pay for onboarding capacity+data. (not sure if you have the data avail, i have not. [precom+provecom] vs [(prove update) + publish fees] over tiem would be interesting here). 50% should be in the ball park |
Beta Was this translation helpful? Give feedback.
-
This is too hard. I don't know how to do it. The built-in market is in heavy use today, we can't just break it. As mentioned above, there's also no consensus that this is the right outcome. Without learning from observing behaviour around L2 markets, I wouldn't be very confident that it's right. The only way to progress is somewhat incrementally. There is no progress in a big sweeping backwards-incompatible change. |
Beta Was this translation helpful? Give feedback.
-
|
Please can you give some more detail as to how this affects the ability of FEVM smart contracts to be able to ascertain the status of a particular piece_cid? e.g how will the following usecase be solved: "As a smart contract, I want to be able to prove that a given piece CID has at least 3 replicas and that they are all valid for at least 6 months from now"? |
Beta Was this translation helpful? Give feedback.
-
|
will this be enabled by the market being "informed of the data commitment"? how will that info be obtained via smart contract if we leave out market altogether? via the verified registry actor storing deal-like info? |
Beta Was this translation helpful? Give feedback.
-
|
It depends how the piece was onboarded. If the built-in market was used, then there will still be deal IDs. Use cases that need such a record on-chain should keep using the built-in market until the final notifications-to-untrusted-code is implemented in a future FIP.
The easy answer is make three deals with the built-in market, as today. (The market doesn't have a piece->ID index, but a smart contract that makes deals can have one, or the index can be generated off-chain. The market does have ID->piece association which can verify the mapping). For pieces that are onboarded with the direct flow, and with a verified data claim, an alternative is to inspect the claims. Each claim has an ID that functions much like a deal ID. Look up the claims on chain and you can see the minimum commitment epoch. A smart contract that is a verified client can also create the verified allocations by transferring datacap. Unfortunately, until we can implement on-chain notifications such a contract would need some external trigger to verify that the allocations have been claimed. For pieces that are onboarded with direct flow and no verified allocation, there is no record in chain state. You can't prove anything about piece commitments that didn't pay the cost of storing a piece->sector mapping in chain state somewhere (which is expensive). So use cases that want to be able to make such proofs should do so somewhere (built-in market, verified registry, their own state), and use cases that don't can be much cheaper. |
Beta Was this translation helpful? Give feedback.
-
|
Note that a separate change #496 championed by @ZenGround0 would allow after-the-fact proofs of inclusion of data even in the case where there is no on-chain association between piece and sector. |
Beta Was this translation helpful? Give feedback.
-
|
I am drafting a FIP for this proposal at #804. Implementation is in progress on the |
Beta Was this translation helpful? Give feedback.
-
|
starting a thread to discussing the In FIP it mentions
my question "Peer: if the notification causes the message to fail due to ErrOutOfGas or such, I'd guess the sector activation will be reverted too? or if the actor detects the gas usage will just exit the notify loop so to ensure the message can still be successfully sent for sector activation?" and from anorth: "If >=1 notification was specified then a failure will result in "aborting the entire operation". Aborting the entire operation means no sectors are activated. No there is no handling for out-of-gas or similar. That's why notifications are restricted to the built-in market actor to start. Your concerns about notifying untrusted code are valid, but there's no attempt to solve them here. @nonsense @rjan90 I am wondering in practical, if there would be any user/use case would want to keep a sector with data activated and alive, without the deals inside actually is activated/considered started on chain? is this piece still considered practically stored on filecoin? is there any practical reason we should have user to specify if |
Beta Was this translation helpful? Give feedback.
-
|
also relatedly my q "could you please specify if a sector contains both f05 deal pieces and non f05 pieces, when notification failed for f05 pieces yet the sector is successfully activated - what would the deal pub activated: ChainEpoch, value be? a from anorth "If the notification fails the deal is not activated. So there is no activated epoch, sector_start_epoch, or last_updated_epoch." unless all implementation this means a f05 state could be mismatched from sector state unless |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/filecoin-project/builtin-actors/blob/cd9ac2bb0afcca7a59465e57cee6569e69070d7a/actors/verifreg/src/lib.rs#L1070 |
Beta Was this translation helpful? Give feedback.
-
If out of gas, there is nothing the actor can do to succeed. The entire operation must abort. Issues like this are why notifications are only supported to the built-in market actor (trusted) for now. |
Beta Was this translation helpful? Give feedback.
-
What do you mean by sector state here? After this change, there is no per-sector chain state in the miner about deals, f05 state is the sole source of truth for deals. If you mean that the data may in fact be in the sector but the market deal not activated, then yes. It becomes "sparkling data" and appears the same as if no market notification had been specified in the first place. Dashboards that only look at f05 state can't represent all data stored on Filecoin, only that data that is part of an active on-chain "deal". Dashboards that want to represent all non-zero data in sectors need to inspect the onboarding messages instead. |
Beta Was this translation helpful? Give feedback.
-
|
also related to the above failure handling This also means if a prove commit sector message has say
I would be curious if we could attempt to make sector activation sector dependent, so that failure in sector A will always not impact sector B unless its message level abortion due to reason like out of gas. |
Beta Was this translation helpful? Give feedback.
-
|
Yes there is potential for wasted gas when one failure causes a message to abort. This is one reason an SP may wish to set There is a cost/complexity vs functionality tradeoff here, and the proposed behaviour tends towards simple and efficient. Because there are two stages to activation (verified claim and deal activation) there needs to be some decision about sequencing them. Because actors commit state independently, it's not generally possible to roll back part 1 if part 2 fails, without aborting the whole operation. And in our environment there are very large efficiencies gained from batching similar work across multiple sectors. It would be reasonably practical to make sector activation independent if we abandoned batches. We have good empirical evidence from earlier non-batched methods that batching is a huge efficiency gain. SPs who want sector independence and are willing to pay for it can submit their activations one-at-a-time. I would expect sophisticated SPs in the future, dealing with multiple markets etc, might select batches to reduce the chances of batch failure (e.g. batch by sectors that use the same market for their deals). So in summary, I did attempt to make sector activations independent, and decided it's not practical while retaining the efficiencies of batch operations. |
Beta Was this translation helpful? Give feedback.
-
my Q: "What would happen when a sector has datacap pieces onboarding without a f05 deal?" A from anorth: "the built-in market actor is notified of the sector termination if the sector has non-zero deal weight or verified deal weight another product consideration we should note is that: This will break most of the toolings & might complicate the implementation for RaaS - as verifreg allocation/claim currently doesn't have one specific state to track when an allocation/claim state is updated (an allocation is claimed, or a claim was made and active to early terminated). (cc @hammertoe plz provide builder needs and insight) should we consider adding claim/allocation state for it to contain the necessary information to serve as one of the deal abstraction MVP? |
Beta Was this translation helpful? Give feedback.
-
|
It's correct that a verified claim remains in state even if the sector is terminated. An off-chain party can detect this because the claim includes the provider and sector number, but the miner doesn't synchronously notify the verified registry of terminations. There are two reasons why I don't think it should:
|
Beta Was this translation helpful? Give feedback.
-
|
i support this FIP. it will allow for exploring more efficient solutions around data storage and data indexing on top of the filecoin network - without any apparent technical downside |
Beta Was this translation helpful? Give feedback.
-
|
The draft FIP is finally published at https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0076.md One thing I intend to add is an exported method for the built-in market that looks up the sector number for a deal |
Beta Was this translation helpful? Give feedback.
-
|
I have proposed adding methods for iterating the verified registry's allocations and claims collections in filecoin-project/builtin-actors#1468 (code here). @raulk left a non-editorial comment which I am moving here for discussion.
|
Beta Was this translation helpful? Give feedback.
-
|
No, I don't have a cost projection. I don't expect wise contract authors to invoke these on mainnet, though I accept that some probably will. Your assumption about expecting on-chain callers isn't correct with regards to my motivation. The methods primarily exist to encapsulate logic for observability purposes. And I think that exporting (i.e. providing an FRC42 method number and allowing any caller) methods for local invocation is the scalable and compatible way for us to do this into the future. Human users should be able to use readable names, and client implementers should not have to implement a new RPC for every exported built-in actor method: a single "invoke(name, paramsJson)" - should suffice. Everything you say about the cursor design is accurate. I don't think that makes it not useful, but nor that it's worth the effort to design a stable cursor (I don't think many contracts should be calling it anyway). You might say that the potential for unsafety means that we should prevent any invocation on chain. I don't agree with this as a reason to blanket deny the functionality (here). I don't think contracts should call it, but I'm not so sure of myself to forbid them doing so. But I don't care about it enough, so if this is going to cause problems during Last Call, I could disagree-and-commit to restricting callers to the system actor. |
Beta Was this translation helpful? Give feedback.
-
|
Gotcha. If it's just for introspection purposes, and we acknowledge various issues prevent this method from being useful/reliable for on-chain usage (some of which I and @Kubuxu identified in the linked thread), I would apply the principle of least privilege and restrict the caller to the system caller. I am relatively unopinionated on the FRC42-derived identifier front. I acknowledge your ultimate intention, but even in the scenario you described, the user needs to know an identifier ahead of time (whether a case-sensitive string or a number). What would actually move the needle is making the interface of an actor introspectable and enumerable. If I were forced to position myself more strongly, I'd lean against the usage of FRC42 here and in this way, because it conspicously sneaks in a visibility modifier, creating more entropy and one more edge case. We'd now have:
All in all, I think this tries to solve for a problem we don't have today, and whose actual eventual long-term solution is an IDL (which Ethereum has had for a long while). And a better short-term way of achieving what you want is to keep this method private and introduce a new standard to enable actors to enumerate their methods. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for articulating a number of good points. I'm going to just pull this out of the FIP. I added it in response to a (verbal) request from another core dev, but there's no persistent discussion I can point to to justify it, and I'm not that excited about it yet. It can come later, alongside better context. |
Beta Was this translation helpful? Give feedback.
-
|
Is it still true that
@dkkapur is that a reasonable limitation to shift towards? my understanding is that there are an increasing number of FIL+ datacap allocations that are associated with paid deals. Is the expectation that those payments will be out of band / separate? |
Beta Was this translation helpful? Give feedback.
-
|
@willscott user programmed storage markets can implement their own payment channels/logic. and today - if someone wants to use onchain payment, they can still use f05. |
Beta Was this translation helpful? Give feedback.
-
|
What I am calling "Direct FIL+" is by definition not involving a payment or any other smart contract. But parties don't have to use it:
I fully hope and expect that on-chain deal payments become far more widespread (regardless of Fil+), and am working towards that not being prohibitively expensive. |
Beta Was this translation helpful? Give feedback.
-
|
This table is still accurate. |
Beta Was this translation helpful? Give feedback.
-
|
Hi All, Here is a note analyzing the CE impacts of DDO. In this report, we identify several cases of what can happen to base_fee after DDO is implemented and look at the impacts to supply in each case. In general, from a supply perspective, DDO has a smaller effect since gas burn is a small portion of the total circulating supply. The FIP also enables higher throughput of data to be onboarded. We quantify the investment needed to support increases in data onboarding rates from the current levels. |
Beta Was this translation helpful? Give feedback.
-
|
@anorth The fip doesn't specify that with my gut feeling is that a similar aggregate fee should be implemented, otherwise im concerned an imbalance cost charge will be introduced again between PCS3 vs PCA, where PCS3 will charge less burn compare to PCA and creates additional unintentional disincentive to participate in built in storage market (as PCS3 doesn't support sectors with f05 deal ID/or support f05 notifictaion). |
Beta Was this translation helpful? Give feedback.
-
|
You're right it doesn't specify it. There's no debate that it should be the same. I've attempted not to re-specify existing behaviour where it is continued unchanged, the aggregate fee is just one of many such items. I've opened #730 (comment) to clarify that the new methods behave the same as the old ones except for the bits that are specified to be different. |
Beta Was this translation helpful? Give feedback.
-
|
kk thanks! I’m more asking to make sure the implementation matches - good to get extra clarification in fips I guess! |
Beta Was this translation helpful? Give feedback.
-
Synchronous Notifications versus CommitmentsCommenting for discussion (and for future readers), I had some concerns but resolved them while writing this up. So, there are two approaches to "activating" deals:
The first option has a significant benefit: it decouples notification from proving. This gives the SP quite a bit of flexibility:
Of course, it has a downside: we need to record a something on-chain and, to save space, that will involve some cryptography (likely pairings if we want constant-sized proofs). But, on further consideration, synchronous notification is actually strictly more flexible:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @ZenGround0 and I have thought about this quite a bit. I would like to enable both one day. The biggest reason not to support async now is that we don't store CommD on chain, and doing so is very expensive in state size (would add a little less than half again to sector state for affected sectors). See #496 for more in this direction. The next is that, even if we did, the sync notification is overall more efficient because of the trust that can flow. So it will scale a little better, at least in a mainnet context (subnets would want async). |
Beta Was this translation helpful? Give feedback.
-
|
I don't see this mentioned anywhere so... is there any reason not to remove the |
Beta Was this translation helpful? Give feedback.
-
|
Filed in filecoin-project/builtin-actors#1497. They'll be useful for recovery if we mess things up here, and also yes avoiding migration of sector info until necessary for some other reason. |
Beta Was this translation helpful? Give feedback.
-
|
Is there any motivation for the |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0076.md#sectorcontentchanged
shouldn't this say 0042? |
Beta Was this translation helpful? Give feedback.
-
|
It should, thanks. Could you make a PR to fix it? |
Beta Was this translation helpful? Give feedback.
-
|
sure i can. |
Beta Was this translation helpful? Give feedback.
-
Edit: FIP-0076
Background & goals
The only mechanism today by which storage providers (SPs) are permitted to commit data in sectors is to make a deal with the built-in storage market actor. The built-in market actor is expensive in gas, and offers only basic functionality. It supports only a single, simple deal schema and policies, and does not support a range of desirable market features. Requiring its use raises costs and limits the utility and value of sectors, and the applications that can be built on Filecoin. It cannot support the expansion in usage or functionality that network participants hope for.
Publishing deals is the largest on-chain cost of QA power onboarding, and currently consumes a large fraction (~half) of total chain bandwidth. Much of this cost is unnecessary in the common case of simple FIL+ verified deals with no client payment, because the verified registry actor already records deal-like information describing the DataCap allocation terms. The built-in market plays no necessary role in allocating or accounting for QA power.
If the built-in storage market actor were bypassed for this common case, QA power could be onboarded at a significant reduction in gas cost. This requires a change in the APIs used by storage providers to onboard sectors. Such new APIs could be simply extended to also support other smart contracts acting as markets, or other data applications.
Goals
Proposal
Direct data onboarding comprises changes to the Filecoin built-in actors, and the components that interact with them, to enable gas-cheap data storage, including Filecoin Plus. Clients make DataCap allocations directly with the verified registry, and SPs claim them directly when activating sectors. The built-in storage market actor is bypassed, and the verified registry’s on-chain allocation/claim record suffices to represent a simple “deal”. Direct FIL+ doesn’t support client payments, but most deals today have no on-chain payment anyway.
The necessary changes also unlock subsequent improvements that can support cheaply-scalable FIL+, and flexibility for other data applications to efficiently implement features on top of basic data storage commitments.
In brief, the proposal requires adding new methods for committing data into sectors, alternatives to
ProveCommitAggregateandProveReplicaUpdates. These new methods accept additional parameters from the storage provider which specify the pieces of data activated in the sector, any FIL+ verified allocations being claimed, and the address of any actors to notify about the successful activation. Deals are possible, but not necessary.Data onboarding chain interactions – current flow
Data onboarding chain interactions – Direct FIL+ proposal
I am currently working out a detailed technical design, ahead of drafting a formal FIP.
A change similar to that discussed in discussion #719 is a pre-requisite for this work.
Direct FIL+
For a Direct FIL+ allocation, the client & SP need not publish any deals. Instead, the client allocates DataCap to specific data pieces directly with the verified registry actor. The SP then refers to these DataCap allocations when activating a sector, and receives the appropriate quality-adjusted power if the proven data matches the allocation.
The Direct FIL+ flow does not support client payments for a deal.
The built-in market
The built-in market can still be used, e.g. if a storage fee is to be paid by the client. In this case, a deal is published as usual (which creates the FIL+ verified allocation). Instead of specifying the deal IDs during sector pre-commit, the SP specifies them at activation, along with verified allocation IDs. During sector activation, the market actor is informed of the data pieces and associated deal IDs, and activates the deal if they match.
This is a change from the current flow, where deal ID are specified at pre-commit, and the miner actor fetches the deal piece information when activating a sector, failing if they do not match. The market actor is informed of the data commitment, but does not control it.
Preserving existing workflows
The existing sector onboarding methods will keep working, while the new ones are made available as options. Participants will need to move to the new methods in order to enjoy the benefits of lower cost and increased flexibility, but will not need to do so in any rush. In the long run, I expect the new methods to be widely preferred and to retire the complexity of maintaining both, but not until a large majority of participants have migrated.
Preparations for future enhancements
Changing to new sector onboarding methods involves significant work from client implementers deal clients, and storage providers. We don’t want to do it too often.
If we provision new onboarding methods to support Direct FIL+, we can take the opportunity to prepare for future enhancements that would otherwise also require changes to these methods. Fully supporting these is beyond the scope of this proposal, but we can cheaply make preparations now to ease implementation and adoption in the future. This generally involves adding a placeholder parameter to the new methods, to be specified in full in a future FIP.
Data activation notifications to smart contracts
Because the market actor is no longer an authority on sector content, the same onboarding flow can support the notification of other, user-programmed actors. This workflow also permits deals to be created on the fly during sector activation, without requiring them to be published first. This could allow future markets to have ~half the gas cost of the current built-in-storage market, even while offering similar functionality.
Scalable verified data allocations
FIL+ verified data allocations that are much larger than a sector, with ~constant size. See #708.
Re-committing verified data that has been lost
Re-onboarding verified data that has been accidentally lost, without requiring the client to allocate new DataCap.
SP selection of Window PoSt deadlines
Support for SPs to choose the Window PoSt deadline to which a sector should be assigned, giving them control over Window PoSt scheduling.
Proven deletion of data
Support for proven removal of data from sectors.
Decoupling sector lifetime from data commitment durations
Adding flexibility to sector durations while retaining strong commitments to data.
Data inspection by smart-contracts
Support for optional storage of a sector’s data commitment in chain state, permitting query of sector content by smart contracts.
Discussion
Reduced gas cost for data onboarding
Estimating by analysis of some batched deal and onboarding messages from mainnet, the potential gas savings are:
The first-order impacts of this are reduced SP onboarding costs and reduced variability in those costs, improving total returns. Reduced gas use for onboarding reduces competition for chain bandwidth, likely leading to a reduction in base fee and transaction costs for other parties.
Second-order effects of reduced gas usage and base fee include reduced base fee burn rate (which accounts for ~12.5% of net circulating supply outflows) in the near term. Reduced onboarding costs may induce more onboarding (locking pledge accounts for the other 87.5% of supply outflows) and user demand for other transactions, more likely over a longer term.
On-chain observable state
Removing the necessity of a built-in storage market deal reduces the amount of information necessarily stored and observable on-chain for verified deals. However, most of the removed information is redundant.
A verified registry claim already stores the data CID and size, client and provider addresses, sector number, and epochs describing the minimum and maximum term for which the provider has committed to store the data. The only additional information that would be carried by a built-in market deal is:
Availability of 5-year maximum term for clients
A client using Direct FIL+ can immediately specify a 5-year maximum term, despite the maximum sector commitment duration and built-in market deal duration being limited to 1.5 years (3.5 years after FIP-0052). After initially committing a sector, an SP will be able to extend that sector out to the 5-year maximum while retaining full QAP, with no further client interaction.
Beta Was this translation helpful? Give feedback.
All reactions