Skip to content

Commit

Permalink
updates, some PR review changes
Browse files Browse the repository at this point in the history
  • Loading branch information
whyrusleeping committed Aug 14, 2018
1 parent 8e30942 commit d198292
Show file tree
Hide file tree
Showing 3 changed files with 46 additions and 24 deletions.
2 changes: 1 addition & 1 deletion drafts/mining.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ ourPower := getMinerPower(pTipSet, minerID)

// Now check if the ticket is a winner!
// mild amount of hand waving in treating the hash output as a number
if sha256.Sum(ticket) % totalPower < ourPower {
if ToFloat(sha256.Sum(ticket)) * totalPower < ourPower {
// winner winner chicken dinner!
} else {
// bad luck, try again next block
Expand Down
6 changes: 3 additions & 3 deletions drafts/offchain-repair.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

### Motivation

The motivation to take the repair protocol off-chain comes from a number of different things, all of which end up making the space on chain required to do an on-chain repair protocol unfeasible. In order to provide repair services, the protocol needs a way to detect faults, and a way to repair them. Fault detection here is not changed, it still primarily depends on the absense of proofs from a miner to detect when a given sector (or set of sectors), but fault repair gets overhauled. In order for fault repair to work, when a fault is detected, storage brokers need to be able to figure out exactly what data has been lost. Under the current construction, this data is provided by miners when they post sector commitments. Each sector commitment includes a list of deals that are stored in that sector (and the proof must verify this, very difficult to do). The deals referenced by the commitment each contain the hash of a piece, meaning there is one hash per piece somewhere on chain, plus some extra associative data. In that regard, the space required on chain is linear to the amount of data stored by the network, this is not a great scaling factor.
The motivation to take the repair protocol off-chain comes from a number of different things, all of which end up making the space on chain required to do an on-chain repair protocol unfeasible. In order to provide repair services, the protocol needs a way to detect faults, and a way to repair them. Fault detection here is not changed, it still primarily depends on the absence of proofs from a miner to detect when a given sector (or set of sectors), but fault repair gets overhauled. In order for fault repair to work, when a fault is detected, storage brokers need to be able to figure out exactly what data has been lost. Under the current construction, this data is provided by miners when they post sector commitments. Each sector commitment includes a list of deals that are stored in that sector (and the proof must verify this, very difficult to do). The deals referenced by the commitment each contain the hash of a piece, meaning there is one hash per piece somewhere on chain, plus some extra associative data. In that regard, the space required on chain is linear to the amount of data stored by the network, this is not a great scaling factor.

This proposal shifts the space required on chain to be relative to the number of files stored instead of the amount of data being stored, and also provides an easy ramp to aggregating many different things (for example, client data may be aggregated by storage brokers into a single 'put' to storage miners). It also simplifies the proofs that are required for sector commitments, simplifies the client storage flow, provides easier to reason about incentives for repair actually happening, and paves the way for future optimizations.

Expand Down Expand Up @@ -52,7 +52,7 @@ type BrokerAsk struct { // TODO: better name pls

A client wishing to store data should select a storage broker `B`, contact them, and send them a storage proposal containing the bid ID, if `B` is willing, they respond that they tenatively accept the deal, and the client starts transferring the data to the broker. Once `B` receives all the data, they verify that it matches what is specified in the bid correctly (filesroot, total size, and erasure params are all correct), and sends back a signed `BrokerDeal` deal, which the client can then post on-chain to seal the deal. Once that deal lands on-chain, `B` is now on the hook for the storage, and the client can now leave with an economic assurance that their data is safe. Note: This is just a slight variant on the 'make deal' protocol.

Next, the broker must select some set of storage miners to store the clients data on. For each storage miner `S`, `B` sends a signed storage deal proposal, including the broker deal, and the data to be stored. Once `S` receives the data, they provide `B` with a signed agreement to store the data at offset X in sector Y. Once the given sector is sealed, they also provide `B` with a proof that the data is stored as agreed. If `S` fails to seal the data as agreed or fails to provide a correct proof, `B` can post the agreement on-chain, and `S` must respond with the proof or be slashed. Once `B` receives a correct proof for the storage of the data, they then send `S` a series of time-locked payment channel updates that can be cashed out contingent on the absense of any fault claims on-chain made by `B` against `S` for the file in question.
Next, the broker must select some set of storage miners to store the clients data on. For each storage miner `S`, `B` sends a signed storage deal proposal, including the broker deal. if `S` responds 'accept', then `B` sends them the data to be stored. Once `S` receives the data, they provide `B` with a signed agreement to store the data at offset X in sector Y. Once the given sector is sealed, they also provide `B` with a proof that the data is stored as agreed. If `S` fails to seal the data as agreed or fails to provide a correct proof, `B` can post the agreement on-chain, and `S` must respond with the proof or be slashed. Once `B` receives a correct proof for the storage of the data, they then send `S` a series of time-locked payment channel updates that can be cashed out contingent on the absense of any fault claims on-chain made by `B` against `S` for the file in question.

Note: The broker should hold onto the data from the client until it has the proof of the data being sealed into sectors. Otherwise a miner could agree, and then fail and lose the data.

Expand All @@ -62,7 +62,7 @@ Separate from making deals, the storage broker must periodically complete a numb

Storage miners are continuously performing proofs of spacetime to convince the chain they are storing the data that they claim to be. If they lose some data, then they may either report a fault, or simply fail to provide proofs (TODO: incentivize miners to report failures up front, makes things easier). In either case, the network learns that a certain set of data is missing and needs to be repaired.

At this point, any proofs that a storage broker has claiming that that data is being correctly stored are now invalid and the broker must create new deals. When the storage broker notices this, they should begin the repair process. This process involves looking at the erasure coding parameters for the data in question, using that to fetch the required pieces from the other storage miners that have them, then using whatever erasure coding routines they need to regenerate the missing data. Once the missing data is recovered, the broker needs to now create a new deal with another storage miner. After that deal is struck, the broker should now have everything they need to continue generating repair proofs, and can continue on as normal.
At this point, any proofs that a storage broker has, claiming that that data is being correctly stored, are now invalid, and the broker must create new deals. When the storage broker notices this, they should begin the repair process. This process involves looking at the erasure coding parameters for the data in question, using that to fetch the required pieces from the other storage miners that have them, then using whatever erasure coding routines they need to regenerate the missing data. Once the missing data is recovered, the broker needs to now create a new deal with another storage miner. After that deal is struck, the broker should now have everything they need to continue generating repair proofs, and can continue on as normal.

TODO: should the storage broker have to be responsible for slashing the miner? Or should they react to someone else slashing their miner when they fail to post proofs? (ref: Honest vs Malicious failures)

Expand Down

0 comments on commit d198292

Please sign in to comment.