by Venus Team
Sep 2021
As the rebranding of filecoin terminology spearheaded by FIP0018 settled, consensus has been reached across communities (developers, providers, ecosystem partners etc) to push for taking on more storage deals to improve the public perception on the fact that most of the network storages are still committed capacities (CCs). Given the above sentiment, design and implementation of droplet module have been put into the spotlight. A clear long-term roadmap is due for Venus community to discuss and iterate on, also as a means for better communications with filecoin ecosystem in general.
While Lotus is leading the way of implementing a full-fledged market module according to the spec, Venus has been making much efforts to catch up and closing the gap in regard to markets. Right now, Lotus supports a 1-to-1 (client-to-provider) storage and retrieval model where burdens of discoveries are mostly on storage clients and match making services like Estuary. Negotiation is a fairly manual process and does not support much flexibility. As venus-team is picking up the reminiscences of the Filecoin Component Architecture, emergent ways of how market could facilitate the dynamics between storage providers and storage clients are constantly being integrated into the long-term vision of Venus filecoin.
Current roadmap for droplet is loosely broken into the following phases.
For phase 1, droplet will deliver a complete deal making experience as what lotus offers. This includes compatibility with lotus client where one can make deal with droplet using lotus client, retrieve deal/data in the same way as lotus retrieves its data, setup storage ask and etc.
- Implementation of the one-to-one model of lotus market like module and fully interoperable with lotus implementation, which means compatibility with lotus client and more
- droplet deployed as independent module, like sophon-cluster and venus-wallet
- Implementation of a reliable market module that runs a separate process from the main storage process
- A clear module boundary that allows interoperability and user customizations
- Flexibilities of market module to interact with existing venus infrastructures using RPC APIs
- Supports for mainnet, calibration and Nerpa
- Lightweight client: compatibility with Lotus and support for droplet unique features including client running separately as a process and remove dependencies for node; great for bootstrapping tests on deal making process
For phase 2, droplet is taking the following approach.
platform-to-peer: droplet as deal making backend for middle-man services like Estuary connecting client and provider. As deal market matures, instead of ineffectively advertising one's storage system in #fil-deal-market, storage middleman services like Estuary and Filswan are taking up the roles for distributing datacap more effectively to storage providers looking for deals. Given venus' unique architecture where multiple providers are sharing same infrastructure (chain services), droplet is in a good position to provide before mentioned deal making backend for a storage middle-man service.
platform-to-platform: droplet as storage backend for a storage integrator (a storage provider who offers different kinds of storage products to its end user, for example, filecoin, S3, tape and etc).
- Taking advantages of Venus' distributed architectural nature, a gateway service backend built on top of current infrastructure
- Compact API: separation of node and droplet data enabling local storage of some of the deal related meta data
- Data transfer support for different protocols in addition to
Graphsync* - Meta data stored locally in HA database like mySQL by droplet
- droplet as deal gateway for storage providers using venus chain services (venus shared modules)
- Deal match making: multiple copies for store and faster retrieval
For phase 3, droplet will look into ways to automate deal flow between client and provider using a peer-to-peer approach, giving up its role as a gateway in phase 2. Additionally, venus pool can be positioned as a retrieval node which is fully aware of deal meta that chain services helped to record.
- Goals for phase 3 is not as clear cut; require more iterations as filecoin develops smart contracts and others
- auto-match deal market: a service to provide algorithmically (as opposed to manually verifying data using current fil-plus framework) verified data storing/retrieval from peer to peer
- droplet as gateway for IPFS: options for paid IPFS node
- In time, a platform model as an easy and quick way for matchmaking might fall out of favour and a faster layer 2 protocol could be built on top of Venus to make true p2p data storage with standardized storage services governed by blockchain
- New econ market on layer 2
Design draws inspirations from the original filecoin component document and filecoin storage market module.
- module and component: "A module means a code library that may be compiled and linked against. A component means a distinct process presenting or consuming RPC API." In this document, the distinction is not as clear. Will need revamping of the Venus documentation to redefine all terms.
- GraphSync: "The default underlying transfer protocol used by the Scheduler. The full graphsync specification can be found here."
In a multi-process architecture, the storage component would form the miner operator’s entry point for all mining and market operations (but not basic node operations) pertaining to a single storage miner actor. It depends on a node to mediate blockchain interactions. The storage component drives these interactions. If viewed as a system of services, the storage component is the consumer of a service provided by a node. Thus, the storage component will depend on an RPC API be provided by a node. This API is likely to include streaming operations in order to provide continually changing blockchain state to the component. The mimblewimble/grin project is another example of this multi-process node/miner architecture.
Similar to what is described for storage component above, droplet will be dependent on a RPC API provided by a node ie. chain services, venus shared modules. Blockchain interactions will be handled by venus chain services which can also be extended to handle authentications among others.
In phase 1, louts market deal flow will be mirrored in Venus. Maintenance of the market and evolution with the network.
In phase 2, proposing and accepting a deal will work as following.
- [Provider] Add storage ask bidding policy along with other deal acceptance policy
- [Client] Query asks from droplet with filters like geo locations, redundancy, deal lifespan and etc
- [droplet] Aggregate requirements from both providers and clients, matchmaking on demand
- [droplet] Provider(s) and client go through rounds of real-time bidding to match-make
- Once matched, provider proceeds to store data as in the one-to-one model
Platform model implementation of droplet may store metadata on the deals it distributes to providers under its wings. Like Airbnb, it may include metrics that a reputation system of both client and provider can be built upon. Metrics like storage success rate, retrieval success rate, fast retrieval enabled? and etc.
venusmodule to provide node servicessophon-messagermodule to provide data servicessophon-gatewayto provide signature servicessophon-clusterto provide sealing and data lookup servicesgo-fil-marketcompatible with lotus one-to-one model (For compatibility with lotus only)- piece data from external deals
- datastore using HA databases for deal meta data
Note that go-fil-market included as dependencies is solely for the use of compatibility with lotus. droplet will be bundling other unique features along with compatibility with lotus.
venus-marketrenamed todroplet
- Community's concerns on infrastructure failure, node redundancy can be setup
- Data transfer complexities
- Some of the goals in later phases may be moving target as ecosystem evolve




