IPFS Files Support

[segator]

Should be very useful if IPFS cluster have api for IPFS FIles

[hsanjuan]

Hi @segator , I think I know what you mean (not 100% sure) , but it would be great if you provide a little bit more details so that everyone that lands on this issue can understand:

• What do you expect from this feature? Do you expect the same API as IPFS files but working instead with the ipfs cluster pin store?
• What's your usecase for this feature?
• How critical is it to have this feature? Is your use-case fully blocked because of it or are there workarounds?

[segator]

you are right, I didn't the best to explain it.
Let me try again.
With the IPFS Files API you can create virtual FS asociation files to IPFS Object ID's but this relation-database if I understand correctly is writed in the local repository of your IPFS Daemon, What I mean to say is that should be interesting if you could have this feature in a cluster, so you define and replicate this VirtualFS ObjectID-->Path relation between some of the nodes part of the cluster, is a way to don't lose this virtualFS in case of ipfs node failure.

Use case, share a virtual FS around the cluster.
Right now it's not criticaly, there are some features more importants that you already have in mind.

Let me know if you understand it now.

[icidasset]

Hey folks, I'd like to 2nd this request. To answer @hsanjuan questions:

What do you expect from this feature? Do you expect the same API as IPFS files but working instead with the IPFS cluster pin store?

Yes exactly.
Same MFS HTTP API as on IPFS daemon.
Add to MFS via cluster, equals adding to MFS on every IPFS instance in the cluster.

What's your use-case for this feature?

A user "authenticates" on a web application by storing the application data encrypted on IPFS using MFS (to be clear, the app does this for you). This app data includes settings, added-by-user content, etc. User of said web application would like to sync that app data on her/his other devices.

How critical is it to have this feature? Is your use-case fully blocked because of it or are there workarounds?

I'd say very critical. I haven't found another way to do this. If you do know a work-around, please let me know.

Thanks ✌️

[RubenKelevra]

I would also like to see this feature. As it makes accessing and modifying of the content of a cluster much more useful, in terms of complexity of the application using it.

[pengisgood]

I'm also looking forward to this feature. +1

[paz]

I'm looking to sync a folder across multiple servers, with the benefit of bittorrent-like p2p. Is this what I'm looking for?

[hsanjuan]

I'm looking to sync a folder across multiple servers, with the benefit of bittorrent-like p2p. Is this what I'm looking for?

Yes, I think so...

A summary of my thoughts so far to get around this:

• One approach is to make sure all cluster peers set mfs root to a value, but IPFS does not have an API for this.
• Other approach is to implement a parallel MFS (ipfs-cluster-ctl files {ls,cp,stat,write} etc) and ensure that is correctly pinned by IPFS, but then IPFS would not realize it's MFS (including IPFS Desktop and tools that rely on MFS). It would also duplicate a lot of functionality. Both MFS could be synced, but would have to do traversal/diff using the current MFS API which does not seem like a very good approach having the root CID.
• Another approach is to let the user run MFS operations and broadcast them to all peers with cluster, but that is really easy to get you out-of-sync when peers are down or multiple cluster peers are sending different mfs operations.

The only thing clear is that I would really like to have a set-mfs-root method in IPFS.

[dokterbob]

At ipfs-search.com we've been working on strategies of sharing metadata (thumbnails and nsfw-data for now, all metadata in the future) and we're considering to do this through (sharded) MFS folders, so that web-clients can transparently look them up without invoking our API.

So we'd have: /ipns/<trusted_root>/<content_cid> pointing to the cid of the metadata/thumbnail/etc.
This could be a pattern for very efficient caching of any kind of API the results of which are immutable.

We have this implemented in nyats, our PoC thumbnailer, but it currently is impossible to scale this beyond 1 node, and thus to have production-quality resilience. In order to do so (without a master), we'd need a consensus layer (like the one ipfs-cluster already has) and, based on that:

1. Consistent (e.g. CRDT-based) updating of MFS folders amongst cluster nodes.
2. Consistent publication of IPNS for folders (e.g. cluster-wide private key and simultaneous publication).

I personally feel that this would be a killer feature, as it would enable proper distribution of IPFS' very centralised and (code) silo'd content publication/distribution side (e.g. Fleek and, previously, Temporal). The key to doing this is combining mutability with trust which, in theory, IPFS does really well.

An alternative would be rolling our own, but this might be the kind of thing which could benefit hugely from having a standardised feature scope.

Love to hear your thoughts on this, and whether or not it's in-scope for ipfs-cluster or belongs somewhere else.
Also happy to have a call or a chat (@dokterbob:matrix.org). @RubenKelevra @whyrusleeping @hsanjuan

[RubenKelevra]

I think publishing these information with a signature in the DHT makes more sense @dokterbob:

https://discuss.ipfs.tech/t/ipfs-records-for-urn-uri-resolving-via-a-dht/10456

So large sides like "archive.org" or "nasa.com" can publish the metadata for their cids with a signature in the MFS and ipfs-search can index them.

@Winterhuman and I discussed a couple of days ago that it should even be possible to have DNS records themselves stored there. But this concept is still in it's early stage.