[2020 Theme Proposal] IPFS Cluster Applications

[NukeManDan]

Note, this is part of the 2020 Theme Proposals Process - feel free to create additional/alternate proposals, or discuss this one in the comments!

Theme description

Many people come to IPFS believing that simply the act of adding/pinning a file enables instance distributed, redundant, permanent storage of arbitrary data among (presumably) peer nodes. This is sadly far from the truth, and can lead to people leaving the community feeling let down once realized - not likely to return. But with the right applications, incentives, and defaults set to easily enable peer groups to self organize and provide this idealized dream of IPFS for themselves (at least).

This is, IMHO, the first step to true general purpose decentralized applications.

Core needs & gaps

As an end user and/or dapp creator, I want the default behavior to host and request for others to host a set of common, collectively valued, data among a peer group. At present, this takes a lot of research and configuration to achieve, if at all.

Why focus this year

• IPFS Cluster is recently getting to the point that it can deliver this solution.
• Great work on permissioned and private networks on IPFS ([2020 Theme Proposal] Privacy #44, textile, peergos, ect.) that can enable configurable clusters sharing private and secure files among a sub-network.

Milestones & rough roadmap

• A minimal forkable set of examples akin to, or incorporated with, those found in JS IPFS to build off of.
• An full-fledged application incubated by our PL/IPFS community that uses cluster to highlight this use case. Examples:
  • An auto-replicating "who is online" guestbook webapp. It allows for some to join the peer group , add your peer ID, sign this (cryptographicly). to showup online you must have all data for the app hosted on your node so others can get the app from you, and you must be online to remain on the log. A cute way to illistrait a true serverless dapp
  • A community shared database. Would include assets that all participants would store and relay redundantly. This could include things like a group website, photo album, chat app, and wiki/docs - so no server is needed. Only at least one member of the community to have their node running for the resources to be accessible. (Something I personally would love to get involved in and gather community support to build - see here - could fit nicely into community engagement goals ([2020 Theme Proposal] Solid foundation for future growth #42) )
  • A group password/keystore backup where sharded anonymous data are spread randomly across a small permissioned network such that only the owner of the keystore could know and privately collect the chunks needed to reconstruct their data. No one else on this network could (trivially) discover any keystore file, despite holding fragments of many of them.

Desired / expected impact

How will we measure success? What will working on this problem statement unlock for future years?

• Increased grass-roots use of IPFS
• Increased IPFS clients/nodes providing a useful service while online, thus high uptime and avalibility on the network to be expected
• Decreased reliance on central gateways, increased community hosted gateways
• Decreased reliance on central servers/resources for dapps in general with use of clusters of dapp users. - true dapps!

[momack2]

I love all these app ideas! @hsanjuan, any specific feedback on how to build them on cluster with the new consensus CRDT model?

I don’t quite understand why these don’t seem like good fits for apps built ON ipfs vs incubated by the same working groups focused on core implementations, performance, documentation, etc. Seems like building community tools like these would be a better fit for focused development by groups like @andrewxhill on textile, @Sharipova from anytype, or someone else excited to get the UX right at a layer above core protocol usability/performance. Can you clarify?

[NukeManDan]

built ON ipfs vs incubated by the same working groups focused on core implementations... Can you clarify?

I mean to say getting dedicated time and effort from the core put into starting and growing an application or two linked above. Specifically having forkable fully functional examples of cluster applications would be great to have directly developed by the core and cluster group specifically.

In my mind the guestbook app would be very simple to implement on top of a simple templated example that established a cluster. @hsanjuan - might you be willing to try and get something like this put together? Glad to contribute 😁

[kishansagathiya]

Great ideas @NukeManDan

One such example would be pinbot-irc. It let's IRC users, pin, unpin, recover or get status of things on a cluster. It has been fairly useful, has reasonably small code and can be use for any IRC channel.

[NukeManDan]

I like it @kishansagathiya!
Although I should also clarify that I think we need GUI applications that are very simple to use to get a wider audience. In that light, an extension to IPFS Desktop or WebUI to enable cluster interactions would be real cool. Who would be the ones to talk with about this?

[hacdias]

@NukeManDan we have actually created an issue about this some time ago in IPFS Desktop repository!

[hsanjuan]

Hi @NukeManDan , I work on "IPFS Cluster". These proposals come close to some of the things we've been wanting to do in the short term.

easily enable peer groups to self organize and provide this idealized dream of IPFS for themselves (at least).
IPFS Cluster is recently getting to the point that it can deliver this solution

Correct! Cluster can be used to create large users group that subscribe or (if they trust each others enough) collaborate on hosting the content each of them adds to the network.

An auto-replicating "who is online" guestbook webapp. It allows for some to join the peer group , add your peer ID, sign this (cryptographicly). to showup online you must have all data for the app hosted on your node so others can get the app from you, and you must be online to remain on the log

Much of the power of IPFS is to be able to get things from anywhere. In that sense I find the language here weird. Fetching data on the network should not depend on a particular node nor exist a requirement to stay online as long as it is well replicated.

A community shared database. Would include assets that all participants would store and relay redundantly. This could include things like a group website, photo album, chat app, and wiki/docs - so no server is needed. Only at least one member of the community to have their node running for the resources to be accessible. (Something I personally would love to get involved in and gather community support to build - see here - could fit nicely into community engagement goals (#42) )

One of my short terms objectives is to create "collaborative clusters" to backup and distribute particular archives on the IPFS network. We would run a a couple of "trusted/authoritative" peers and anyone would run a simple command (GUI?) to join and replicate a pinset. That pinset can be, for example, one of the wikipedia languages, xkcd, a stash of world literature works or cat pictures. A website would show the different archives, instructions to join and estimated size needed on the local computer.