Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The IPFS<>Filecoin Interface #143

Open
whyrusleeping opened this Issue Feb 14, 2019 · 0 comments

Comments

Projects
None yet
2 participants
@whyrusleeping
Copy link
Member

whyrusleeping commented Feb 14, 2019

IPFS<>Filecoin Integration

Filecoin is the incentive layer to answer the question “Who keeps the data in IPFS around?”. To make this possible, we need a way for content in IPFS to be preserved by Filecoin, and for Filecoin to make that content accessible to the IPFS network. In other words, we need an interface that enables the common IPFS developer use cases like storing a data backup or configuring distributed web hosting and content delivery for their app. Not all IPFS developers will want to integrate with Filecoin directly, so our interface can also support an ecosystem of pinning services that automate the storage and availability of data in Filecoin - and do smart caching to make popular content more available to the IPFS network.

The Interface

The interface is mostly defined by the set of utilities that make it easy to move data from IPFS into Filecoin, and vice versa. The interface can take a file (referenced by an IPFS hash) and store it in Filecoin, and then pay to make that content accessible by retrieving files from Filecoin and caching them for the IPFS network. Many people will want to offload responsibility for retrieving and caching files stored in Filecoin to make them available to the IPFS network, which will create an ecosystem of pinning services, or “bridge nodes” running both IPFS and Filecoin, that simplify that process.

At a high level, the interface allows you to persist data, so that you know it is being stored even if you delete your local copy of it. And it allows you to ensure the availability of data even when you're not the one hosting it.

Store by hash

A common use case for IPFS developers is ensuring redundant, decentralized storage of data in IPFS. The interface supports that by offering a “pay to preserve” option that stores data directly in Filecoin.

Pay to preserve

Anyone may request that the interface store a given IPFS hash, for a certain period of time. This process starts by fetching the data through the IPFS network, then contracting miners for its storage in the Filecoin network. Proofs for this deal would be returned to the requester.

Retrieve by hash

Filecoin is a two sided market. This means that payment is required for storage, and separately for retrieval. This allows miners to compete on price separately in both markets. So, separately, interface users would need to pay for retrieval.

For developers who want to integrate directly with Filecoin as an extended storage layer, they can use the interface to ensure content stored in Filecoin is made available to the IPFS network. This would allow a storage-constrained IPFS node to offload data to Filecoin until requested on the IPFS network. This involves fetching the data from Filecoin retrieval miners on demand and serving it to requesting IPFS nodes, with smart caching strategies to avoid frequent fetches (which would be costly).

Interfaces

The interface should provide roughly the following function signatures:

type FilInterface interface {
  Preserve(c Cid, d Duration, p Payment)

  EnsureAvailability(c Cid, d Duration, p Payment)

  ListPreserved() []Pair<Cid, EOL>

  ListAvailable() []Pair<Cid, EOL>

  IsPreserved(c Cid) (bool, EOL)

  IsAvailable(c Cid) (bool, EOL)
}

An interface node providing availability for some content would have extra logic added to their datastore that looks roughly like this:

// someone requests an IPFS hash from us
func (fd *FilInterfaceBlockstore) Get(c Cid) (Block, error) {
  b, err := fd.normalBlockstore.Get(c)
  if err == nil {
    return b, nil
  }

  // we didn’t have it locally, let’s check if we’ve agreed to keep it available
  if !fd.AvailabilityManager.ProvidingData(c) {
    return nil, ErrNotFound
}
  // we’re responsible for this data, get it from filecoin
  b, err := fd.FilecoinRetrieval.Get(c)
 if err != nil {
    // Oh no! we have failed to uphold our agreement!
    return nil, err
  }

  // We got it! Cache it locally for later
  fd.normalBlockstore.Put(b)

  // Return it to the requestor
  return b, nil
}

Pinning service on Filecoin

Anyone can use these primitives to run a pinning service on Filecoin (using many storage miners and many retrieval miners). The pinning service effectively serves the role of stradling the Filecoin<>IPFS interface to listen for requests for specific hashes on the IPFS network, fetch that content from the appropriate Filecoin retrieval miners, and serve/cache the requested data on the IPFS network to make it available.

Pay for Availability

Pinning service users can request that a given file remain available for a certain time period, for a certain amount of money. The interface would then serve requests for that file through the ipfs network either by keeping the file cached locally, or retrieving it on-demand. This smart caching mechanism would comprise most of the functionality of the ‘pay for availability’ feature.

Pay for duration instead of per retrieval

For pinning services, paying per retrieval is very difficult to prove, and also provides a worse user experience overall. The pinning service operators can abstract that away, allowing users to simply pay for retrieval for a certain length of time. The cost per ‘availability hour’ is left entirely up to the operator, who should look at storage costs, bandwidth costs, and general usage statistics in order to compute a fair price. Operators might include a bandwidth cap per hour/day to prevent a single piece of hosted content from consuming disproportionate resources.

Trusting pinning service operators

A malicious pinning service operator may simply choose to take the money and not serve the files. Paying for retrieval in filecoin requires a bit of trust between the client and the retriever, since there is no easy way to prove that someone did or did not give some data in exchange for money. This means most retrieval miners (and thus, most pinning services operating on top of the retrieval market) will likely be fairly well known entities.

Accountability

If the pinning service produces signed statements of its activities, it could become possible to ‘call out’ operators for misbehavior. A reputation system may manifest itself for clients to be able to select reliable pinning services.

Future Pinning Service Improvements

Payment contingent on storage

The current design allows a malicious operator to ‘take the money and run’ instead of storing content. To compensate, pinning services will have to be ‘well known’ - you won’t use a pinning service again if they are malicious.

That said, we could design a system where the clients payments can only be redeemed if the files are actually stored the way that the client wants. This would be similar (but trickier) to the way that filecoin clients pay filecoin storage miners.

Verifiable retrieval

Currently, content requesters send payments to retrieval the miners without guarantees that they will respond with the requested data. An area of future work and investigation is to ensure that retrieval and/or availability is verifiable in some way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.