Contract as a Deal Client Standard

[aashidham]

The goal of this proposal is to align on a new FRC that will create a new interface upon which any Solidity contract deployed on FVM can issue deals to participating SPs.

The motivation of this comes from the great variety of ways in which deals can be created on the FVM today. We wanted to define a composable, extensible interface where a great many use cases for building deals can be supported.

Some examples of this include: deal replicator contracts, proxy contracts that can be leveraged by many DataDAOs, repair workers, and many more such services.

We are proposing an interface DealProposer to look like the following:

interface DealProposer {
    event DealProposalCreate(bytes32 indexed id);

    // returns deal proposal given proposal id. Should return CBOR-encoded DealProposal struct.
    function getDealProposal(bytes32 id) public virtual returns (bytes);
}

Note that the DealProposal struct will be CBOR-encoded and look like the following:

pub struct DealProposal {
    pub piece_cid: Cid,
    pub piece_size: PaddedPieceSize,
    pub verified_deal: bool,
    pub client: Address,
    pub provider: Address,

    pub label: Label,

    pub start_epoch: ChainEpoch,
    pub end_epoch: ChainEpoch,
    pub storage_price_per_epoch: TokenAmount,

    pub provider_collateral: TokenAmount,
    pub client_collateral: TokenAmount,
}

We expect DataDAOs, perp storage solutions, repair workers and other deal-issuing contracts on FVM to implement the DealProposer interface.

We expect the flow to look like the following:

1. Clients write a DealProposalCreate event to the event log whenever they have a DealProposal they want to send to an SP.
2. SPs poll the event log for DealProposalCreate events.
3. Once they have found an event, SPs call the client contract's getDealProposal function with the id stored in the event.
4. Clients then conditionally (after likely doing some validation logic on their end) release the DealProposal payload associated with the DealProposalCreate event.

The overall flow looks like the following:

Here is an example DataDAO contract that inherits DealProposer:

contract DataDAO is DealProposer {
        // payloads maps between bytes32 id and a CBOR-encoded DealProposal struct
        mapping(bytes32 => bytes) public payloads;
        
        // note that _deal is a CBOR-encoded DealProposal struct
         function makeDealProposal (bytes _deal) public {                    
                     // creates a unique ID for the deal proposal -- there are many ways to do this
                    bytes32 _id = keccak256(abi.encodePacked(block.timestamp, msg.sender));
                    payloads[_id] = _deal;
                    
                    // writes the proposal metadata to the event log
                    emit DealProposalCreate(_id);
         }

         // this returns a CBOR-encoded DealProposal struct
         function getDealProposal(bytes32 id) public returns(bytes) {
                  return payloads[id];
         }
}

Notes:

1. The client has complete freedom in how the deal is implemented and stored. The above example stores the deal in state. It could also be generated on the spot, stored in a database, or some other method altogether.
2. The client may need extra interaction to claim a deal. This might mean making an off-chain call or some other private implementation. The client is totally free to implement this as a separate set of methods specific to their use case.

Open questions:

1. Right now, the DealProposalCreate event only contains an id. Is there more worth adding to the event signature? Keep in mind each additional topic needs to be 32 bytes at most. Having more descriptive events may allows SPs to filter events so that they can more selectively respond. Some ideas:
   (a) MinerID: to let a participating SP filter by its own ID. This is valuable for clients to emit events that are specifically for a constrained set of SPs.
   (b) string: a human readable string (of at most 32 characters) that describes the storage deal. Can be optionally filled by clients. Relevant strings can then be filtered by SPs. May lead to confusion / splintered topics since this is a freeform field. May need further standardization if included.
   (c) pieceSize: the size of the deal. SPs can then filter by the size of deals they want to store.
   (d) verifiedDeal: whether or not the deal is verified. SPs may not want to entertain unverified deals.

More open questions and notes to be added, but thought I would kick off the conversation! :)

[scotthconner]

I like how this is defined as an FRC in user-land. One question I have is why the solidity/event interface uses CBOR encoding. Would it not make it more consumable by ethereum toolchains and existing methodologies to not use CBOR for ethereum events? Are there interactings with other lotus APIS that makes this either desired or required?

Security: imagine a contract is malicious, and the event data is garbage, wrong, or points to unscrupulous miner IDs, clients, etc. Are there any open attack vectors by interacting with the DealProposal struct that would phish, scam, or mislead someone interacting with it? I need to look more closely at how SPs would use this information - but I'm thinking about a miner participating in a deal proposal where the client collateral or the deal verification is spoofed in some way.

Are there additional interfaces or interactions that we might need to expose as part of this standard to enable some level of notarization on these events? If the other APIs the SP would use to further broker a deal are secure enough to say, not trick an SP into taking a mis-represented deal, then maybe that isn't needed.

Having a user-space standard protocol for pub/subbing deal marketplace activity seems critical. The general direction of the proposal is valuable and fit for purpose.

[jennijuju]

Thank you for getting this started.

DealProposalSubmit

This name may be easily misleading to newer developers as it sounds like "Submit a deal proposal to the market"

If you were going to store the deal proposal in the contra, then I think we do not have to emit the event and just have a getAllDealProposals method, and make the get deals and their indexing + process off chain.

Some of the deal proposal variable should be optional and can be generated by the SP on the fly, as they get the data and construct the PSD message. This is also limiting DataDAOs to filecoin built in storage market structures and may prevent future user programmable storage market flexibility. One thing might worth considering is to add piece CID and exposing a metadata field. and the metadata can be defined by datadao operators and store info like: where the data is, how the SP will get the data, and others.

[aashidham]

Proposal:

• DealIssuer -> DealCreator
• DealProposalSubmit -> DealProposalCreate

[jennijuju]

For the initial version that we will use with space warp let’s go with DealProposer and DealProposalCreate

And iterate with the protocol and application developers over the next couple weeks

[aashidham]

Confirmed:

• DealIssuer -> DealProposer
• DealProposalSubmit -> DealProposalCreate

[nonsense]

At the moment the deal proposal libp2p protocol /fil/storage/mk/1.2.0 includes a transfer struct - https://boost.filecoin.io/boost-architecture/libp2p-protocols

I think we should probably include a location reference to the deal proposal standard over the FVM too in some form (more fields in the DealProposal struct, or some method to fetch location ref via id). Whoever proposes a deal (DataDAO or another actor), should also be responsible for saying where to find the data (at least optionally). If such a location reference is omitted maybe it could explicitly mean that the data is on IPFS for example...

In any case having the data available at multiple locations is going to make it easier for SPs to fetch it. If no location reference is specified, how are SPs supposed to find the data? It pushes a lot of extra work onto them.

[brendalee]

ignore that comment - @ribasushi kindly reminded me that this is for on chain stuff 😄

[aashidham]

@raulk @jennijuju @nonsense thoughts on the approach we should take (out of the three shown above) for giving the SP location information? We should def take one of these on, but not sure which is most idiomatic.

[aashidham]

Bumping this thread -- thanks

[brendalee]

would be interested to hear others input based on potential future use cases for fvm, want to make sure that the design is flexible for future use cases.

Based on my observation: for the SPs that are doing pull based storage deals today (aka not directly on Filecoin network/they have some manual interaction with the client), most of them download data from the client with a link to the data the client provides, or have an even more manual data transfer step (such as getting hard drives from the client).

the three options all could support this existing case (if I'm understanding correctly), so would be interested to better understand explicit tradeoffs here based on what you're seeing folks build

[aashidham]

Incorporated the location information into a new event signature here: #604 (comment)

Please have a look and comment.

For now, incorporated 2nd option (with 2nd sub-option chosen) from the post here: #604 (reply in thread)

[aashidham]

@raulk @jennijuju I wanted to flag a potential idea for an escrow deposit somewhere in this deal flow. Perhaps it is out of scope of this particular FRC, but perhaps not.

We do need some way for a client contract to trustlessly pay an SP and for the SP to only receive payment once they have published the storage deal and sent a ProRep.

Otherwise the client can ask for a deal they have no intention of paying for. An SP can also receive payment and take no action to publish the deal or submit proofs. What kinds of mechanics do we have to avoid this?

[jennijuju]

Lets do a separate FRC to propose a client contract interface. (I think once lotus finishes the initial prototype we will have a better understanding what it could look like. I am also open to get a discussion started earlier, next week or so.)

[cbtan21]

@aashidham @jennijuju this is exactly what BigData Exchange (BDE) has built, although on BDE, currently it is the SP that is paying the data client. This can change anytime based on market dynamics, demand and supply.

however, the transaction part, i.e. the proceeds from the auction, is currently separated from the deal making part. if we can somewhat incorporate these two for large datasets (for eg. 1 PiB), where one data client can send to multiple spids, which all belong to the same sp, and programmatically check the status of deal making and automatically releases the funds upon deal making completion (which can be tiered, staggered, or fully completed), that will be very helpful for both the data clients and SPs.

[jennijuju]

This particular standard is built to work with the existing builtin storage market, which currently only supports zero/positively priced storage deals. And we are hoping more applications/users will start to pay storage providers for their service: people will fund a client contract that acts as a dao to initiate deals to store data for public good via paid deals.

If Big Data Exchange may leverage this client contract for your auction (move auction onchain) too , which is totally possible as you can define the SP -> client payment logic in this contract - that would be amazing—looking forward to see your solutions!

[nonsense]

I suggest we drop the CBOR encoding, unless there is a very good reason to include it.

It seems like the Deal Proposal submitted on chain and the one actually pushed with a PSD will be different - for example the provider field should be empty on chain, whereas an actual provider which takes the proposal will populate it prior to submitting PSD on-chain.

In this case it doesn't make sense to introduce another encoding (namely CBOR).

[aashidham]

@raulk thoughts on this? Would defer to engineering perspective here.

My understanding was that we want to do CBOR encoding because we want to ideally take the deal proposal from getDealProposal and allow the SP to publish that deal proposal, with modifications made as needed (eg the population of the provider field).

With CBOR encoding, we incentivize as much parity as possible between the DealProposal struct described in the OP and the struct that is actually published, since all of this remains in Rust instead of being translated back and forth between Rust and Solidity.

[raulk]

On the topic of CBOR vs Ethereum ABI, these are my views:

1. Most external clients of this FRC (e.g. Boost) anyway need to encode/decode Deal Proposal objects, so adding the Ethereum ABI might be extra overhead for them. If they consumed this in an Eth ABI, they would probably need to add a dependency (although they probably need to add it anyway to consume the ABI-encoded event).
2. Most importantly, CBOR is the lingua franca of the Filecoin protocol, so I would prefer FRCs to target CBOR for future-proofing. Otherwise, future Wasm actors would need to understand Ethereum ABI to interact with Deal Proposals, which is rather questionable.
3. For contract-to-contract interactions, devs can either build a proxy translation contract, or the Filecoin Solidity libraries can offer translation logic.
4. A CBOR-encoded object enables pass-through scenarios in client applications (e.g. feeding the proposal into the deal pipeline in Boost, although in practice you will want to deserialize it and validate it, most likely).

I would recommend that the interface return a CBOR-encoded DealProposal inside opaque Solidity bytes.

[aashidham]

Based on the output of conversations in filecoin-project/boost#1160 (comment) and #604 (comment), I propose we include the following inside the event signature. These are optional fields that don't have to be filled in but can allow SPs to better process deal proposals:

• A string representing the location of the data (either as an https://domain.com/file.car or ipfs hyperlink) (can be left as the empty string if not available)
• A uint representing the size of the deal in GiB (can be left as 0 if not available)
• A bool representing whether or not this is a verified deal (can be left as false if unknown)
• A uint representing the nanoFIL the client is offering for the deal per GiB per epoch of storage (can be left as 0 if no price offered)

This would mean the event signature would look like this:

event DealProposalCreate(bytes32 indexed id, string link, uint indexed size, bool indexed verified, uint indexed price);

[nonsense]

I don't see a reason why some of the params should be emitted as event fields, and others should be fetched from the contract state with eth_call.

Ideally we want a future-proof standard for notifications that deal proposal was published (i.e. DealProsposalCreate), which then Boost (or any markets node) can handle. Including additional fields in the event, means we have to abstract the whole event monitoring in Boost, which I think doesn't make much sense.

Ultimately splitting deal proposal fields in-between event fields and state, doesn't make much sense to me, and I don't see a good reason to do it. It's much cleaner to just define that DealProposalCreate will emit and id and one can fetch all deal proposal fields using that id.

eth_call is quite cheap too. We can easily call it for every single DealProposalCreate -- we don't seem to be getting any benefits by pushing "some" of the proposal fields inside the event.

[aashidham]

The location of the data (link) will not be present in any proposal field. If we remove everything else (due to overlap with the deal proposal fields) we should at least leave this one.

Nonetheless, I don't think it is problematic to have fields repeated in the event signature and the payload.

Just like eth_call can be used to filter SPs, the event signature can be useful for SPs to filter client deal proposals. This is the SP side of the handshake, and more detail here can be used to ignore some proposals altogether. You’re right that eth_call is very cheap, but it seems like a waste to only have the id the event param in this part of the handshake.

Additional fields also add valuable annotations to the event log, which keeps some proposed deal metadata on-chain. Not the most critical, but it is valuable for indexers / chain-watchers to see proposed deal metadata in comparison to the actual deal metadata.

[brendalee]

if we are concerned about having too much in the event signature, we could remove the string for data location.

It seems like the main purpose of having these fields is for potentially indexers/chain watchers to get more aggregated info on what's happening on chain. I do think size of deal, verified deal, and price will be important for indexers/chain watchers. I don't know if these folks will care about where the data is sitting though.

For filtering client deal proposals, it seems like having this information in the event signature isn't necessary (based on Anton's proposal with eth_call. so the main use case for this would be for non SPs who may care about these events, but don't want to make additional calls to get additional info. Not an expert so please correct me if this understanding is wrong!

[aashidham]

Based the thread above, and Anton's convos here: https://filecoinproject.slack.com/archives/C03AQ3QAUG1/p1676639087961419

I think we should align on:

• A bytes representing the size of the deal in GiB (can be left as 0 if not available)
• A bool representing whether or not this is a verified deal (can be left as false if unknown)
• A uint representing the nanoFIL the client is offering for the deal per GiB per epoch of storage (can be left as 0 if no price offered)

This would mean the event signature would look like this:

event DealProposalCreate(bytes32 indexed id, bytes size, bool indexed verified, uint price);

[anorth]

I think @nonsense original points are spot on (I can't resolve the Slack link to a conversation - what channel was it in?). Following my points below, it'll be impossible to design an event schema that has data appropriate to all future deal-making actors, except by making it just a pointer to where that data can be fetched from the emitter. Unlike the DealProposal type, this event actually could become a standard for all market-like actors in the future, if it's simple enough.

As you've already noted, you need to make provision for all the further fields being optional anyway. Adding anything beyond a pointer here ends up in an uncanny valley where there's some info, but not enough to rely on and it can't always be specified anyway. It would be simpler if there's just one way to do it: follow the pointer to fetch the full data from the emitter.

As another point: why bytes32 for the id? Why not arbitrary bytes? (this question might only make sense when this is viewed as an FVM standard rather than EVM - there might be some EVM limitation of which I'm not aware).

[aashidham]

The spec for this document is being iterated on here: https://www.notion.so/pl-strflt/Client-Contract-FRC-458e625f13b14c70bfdfe7ed64007b6c

The scope of this FRC has expanded to include datacap, paid deals, and client verification.

This prior thread and notion doc will be referred to in the subsequent FIP discussion and FRC draft that follows.

[jennijuju]

FYI- im making the notion page publicly viewable as well

[jennijuju]

fyi im updating the discussion title and such according to the latest discussion.

[jennijuju]

"DealProposer interface to standardize storage deal proposals on FVM" -> "Contract as a Deal Client Standard"

[anorth]

This is a good discussion and I'm happy to see people attempting to align on some standards here.

We wanted to define a composable, extensible interface where a great many use cases for building deals can be supported.

It's not going to be easy to predict everything useful that clients, providers or markets will want in the future, so I think there's some risk of locking into something too narrow if the intended scope is large. That is, if the scope of this standard is narrow and focussed on what's possible today, it will be relatively easy and expected for new standards to emerge in the future that support more things. OTOH, if the scope here is large and it tries to be the standard for everything, the chance of getting that right is very low, but it might gain such wide adoption that the barrier to alternative standards arising is too high, and the ecosystem is locked into an unfit standard.

As currently written, this proposal is entirely specific to the built-in market actor. That's currently the only way to get an SP to store data, but it will not be for too long. Motivation for this are in #241 and #298. I expect and hope the built-in market actor to become entirely redundant and unused in the future.

Thus, I think this proposal should either:

• very explicitly target the built-in market actor, or
• generalise now to support arbitrary market-like actors in the future, with whatever unpredictable capabilities they might want

In order to target the built-in market actor, the main thing that needs to change is the positioning and narrative. The proposal should explain that it's a standard for arranging deals on the particular terms that the built-in market understands, and no more. It should leave conceptual room for new standards to emerge later.

In order to target a broader scope, I think this proposal would need to (at least):

• Add a field to deal proposal specifying the address of the actor to which the deal should be submitted
• Add data specifying the method number to which the deal should be submitted. For this to be standard, that means also defining an FRC-0042 method for deal submission that similarly needs to be general enough to handle all possible future deal proposal schemas (we could then backport this to the built-in market)
• Add space for schema-less proposal metadata into which a client can encode market-specific deal parameters which cannot be predicted from here. An opaque byte array intended to be deserialised according to market-specific schema?

I would recommend explicitly scoping this proposal to the built-in actor. The work to define a good standard for future user-programmed market actors is large, and the chances of getting it right are quite low. IMO it is premature to attempt to standardise anything more here before native actor developers have a chance to build things, experiment, and discover patterns that do and don't work.

See some related comments about the Ask protocol.

[anorth]

There might be something I haven't seen about a standard conversion, but how does an event emitted by an EVM contract get converted into an event emitted at FVM level? As an FVM standard, I want to see what this looks like in FVM-native terms, as defined in FIP-0049.

@raulk is there an obvious conversion that doesn't need to be stated here?

[raulk]

On mobile now, but it's specified in FIP-0055!

[anorth]

Thanks I see this here in FIP-0054

// Values are encoded as DAG-CBOR byte strings.
ActorEvent {
    entries: [
        (0x03, "t1", <first topic word>),   // when LOG1, LOG2, LOG3, LOG4
        (0x03, "t2", <second topic word>),  // when LOG2, LOG3, LOG4
        (0x03, "t3", <third topic word>),   // when LOG3, LOG4
        (0x03, "t4", <fourth topic word>),  // when LOG4
        (0x03, "d", <data word>),           // event data, omitted with LOG0
    ],
}

I think this event proposal should thus specify the schema of events in this format, since that's what SPs will be looking at.

[brendalee]

Adding a few comments here based on some observations from the nft.storage perpetual storage project. With 11 SPs participating taking 100 or so deals a day, we already see a lot of wasted bandwidth on both the client and SP side as many SPs attempt downloading data, but end up not successfully getting the deal since another SP already claimed + published. We need to have some plan to address the frontrunning problem as it's already surfaced and causing frustration.

We should have some guidance on how to handle the basic use cases around SP selection so it's a bit more deterministic / not wasteful for client + SP resources with the competing for deals situation...

1. whitelist of x SPs, with small # (much less than x) replicas per deal
2. whitelist of x SPs, with approx x replicas per deal
3. free for all for any SP to take deals (more complicated most likely)

Ideally we have a solution for 3, which works for 1 and 2 as well. Based on some work done by @jennijuju and team, seems like soft acceptance + timeouts was what we landed on to solve this frontrunning problem.

[raulk]

Although having to cover gas fees even in the case of rejection makes (2) a no-go for me.

[brendalee]

With the current state of the network your flag on gas fees + rejection is correct, would not be something SPs are happy about. 😆

I think just looking at from some of the use cases we know about, longer term it would be worth providing two references/options for builders (and clients perhaps)

• Permissioning: we currently do have clients who only want to work with a subset of SPs (they are at the moment manually coordinating this)
• transactionless selection option: enable more decentralized use cases like perpetual storage/data daos!

My personal opinion would be to focus on option (3) first and establish a good reference that others can leverage, especially since the 1st use case is handled (although very poorly) today.

[aashidham]

We are working on an SP election addition to the FRC that will be visible publicly with the publication of our first draft of the FRC.

The approach relies on any Boost SP being able to "subscribe" to deal proposals from a CC. The CC can then order the subscribed SPs for each emission of a deal proposal. The first in the order can then receive the deal. If that SP fails to submit a PSD from the deal within a specified time interval, the next SP can then receive the deal.

This solves for a few product requirements:

• The CC doesn't need to pick out SPs ahead of time; SPs can subscribe to any deployed CC there is.
• The SPs are picked without favoritism (unless the CC wants to show deliberate preference for a certain SP)
• Each SP gets a chance to make the deal, without wasting bandwidth in downloading the data for the deal.

[aashidham]

Any comments on this approach? Public draft now available for viewing at: https://www.notion.so/pl-strflt/SP-Frontrunning-Draft-Proposal-ce4bf07156764142a1d95d483b13ae8b

[zhiqiangxu]

@aashidham I think the standard also needs to consider pricing negotiation, if the pricing is unacceptable for the SP in "order", there should be a fast forward way instead of waiting for timeout, which would be too slow.

[RobQuistNL]

[nonsense]

off-chain and on-chain solutions are quite different, we can't just use Spade / Evergreen on-chain.

[lordshashank]

hi everyone,
I am not able to get why do we have to give car file link while making a deal in client contract as i think it would be better to give a file link in params and miner converting it to car as it will save some gas with less params and also i as service provider would not have to pin both car and actual file (one for miner and one for retrieval).

[lordshashank]

@scotthconner @aashidham @brendalee, etc had a worth lokking discussion on this at https://filecoinproject.slack.com/archives/C04QGFF5EDR/p1680722575169839

[lordshashank]

As a storage solution provider my point was on pinning of car file as usually one will have to pin one file for retrieval and one car file for deal proposal. proposal is to pin one file only and send commp by creating a car file but not pinning it and miner himself creating this car file for deal from the link of actual file (security risk could be tackled as commp must match). I think that pinning both files is apparently doubling the ipfs storages needs and this car file is needed only until the the deal making is completed.

[alexanderwilliom]

@aashidham
I have a question:
Currently, users need to query ask orders and select the corresponding miner to sign and make a deal transaction. If a user initiates an order through a Deal Client Contract without specifying a miner to perform the deal, the miner need to query the Deal Client Contract's interface to retrieve the customer's order. If a miner wants to accept a storage order from a Contract, does the user need to sign again, or can the miner start storing and calling publish_deal on the market actor? And will call back to the method handle_filecoin_method(uint64, uint64, bytes) of the Deal Client Contract? Right? Is my assumption correct?

[aashidham]

Yes the client will authenticate through the dispatch method inside handle_filecoin_method. This will happen before the storage deal is actually published, and client authentication is a prerequisite for the PSD message to be written on-chain.

@shrenujbansal @scotthconner can provide more technical details.

[KarthikeyaGundumogula]

I have a question, at what point in the deal flow the client makes payment to store data, makeDealProposal or at the time of dealNotify