Feature: ipfs file system support

[klauspost]

ipfs is an interesting concept, and is would seem that it could be feasible to have support for it.

Even though the reference implementation is written in Go, it should probably be implemented via the REST interface.

[gcornut]

Ok I'm looking into implementing the IPFS backend for rclone and I did some research that I'm going to share here along with my initial implementation plan.

To start, I think I will use the go-ipfs-api to access local or remote IPFS API node.

I see three possible FS access in IPFS:

1. Shared immutable read-only FS via IPFS path (path in /ipfs/<HASH>)
2. Shared mutable FS via IPNS path (path in /ipns/<HASH>)
3. Node-local mutable FS via IPFS files API (ipfs files ls /local/ipfs/file/path)

Each of these FS access could be considered as different "buckets" from an IPFS remote but you could only have 3 static bucket (ipfs, ipns and local).

Some limitations with IPFS:

• Can't get or set modification time of files
• No SHA1 or MD5 hash directly accessible
  => we could add a new IPFS-specific hash type to rclone (like for dropbox) using the go-ipfs implementation (I don't think we should use the REST API for that since we would need the IPFS endpoint URL in fs/hash.go :/)
• Have to handle uploading, pinning and unpinning of IPFS content separately from the IPFS files API operations (create and delete files)

I think I will start a first draft implementation without hash support or buckets (only accessing the node-local FS and not /ipfs or /ipns).

Then after, I'll look into:

• Buckets (to access local, ipfs and ipns paths separately)
• Performance (flushing only once the IPFS FS?)
• File hash
• Alternative pinning options (always pin, never pin, pinning of IPFS dir rather than individual files?, etc.)

[ncw]

@gcornut wrote:

Ok I'm looking into implementing the IPFS backend for rclone and I did some research that I'm going to share here along with my initial implementation plan.

Great :-)

To start, I think I will use the go-ipfs-api to access local or remote IPFS API node.

That looks like a good library.

I don't know much about IPFS so apologies in advance for any stupid questions ;-)

I see three possible FS access in IPFS:

1. Shared immutable read-only FS via IPFS path (path in /ipfs/<HASH>)
2. Shared mutable FS via IPNS path (path in /ipns/<HASH>)
3. Node-local mutable FS via IPFS files API (ipfs files ls /local/ipfs/file/path)

Each of these FS access could be considered as different "buckets" from an IPFS remote but you could only have 3 static bucket (ipfs, ipns and local).

I see what you mean. Would someone tend to use one of those methods exclusively? If so maybe a parameter might be better "access = ipfs" which could go in the config file or as a command line flag --ipfs-access ipns?

Some limitations with IPFS:

• Can't get or set modification time of files

Is there a time of upload or something like that? That is still useful if there is.

• No SHA1 or MD5 hash directly accessible
  => we could add a new IPFS-specific hash type to rclone (like for dropbox) using the go-ipfs implementation (I don't think we should use the REST API for that since we would need the IPFS endpoint URL in fs/hash.go :/)

Which hash function is it? I saw sha2-256 mentioned in the code. It is relatively easy to add a new hash function to rclone.

• Have to handle uploading, pinning and unpinning of IPFS content separately from the IPFS files API operations (create and delete files)

I don't understand this one!

I think I will start a first draft implementation without hash support or buckets (only accessing the node-local FS and not /ipfs or /ipns).

Something which can read and write files will mean you can run the integration tests against it which will help enormously when building up the code.

Then after, I'll look into:

• Buckets (to access local, ipfs and ipns paths separately)

Or a parameter?

• Performance (flushing only once the IPFS FS?)
• File hash
• Alternative pinning options (always pin, never pin, pinning of IPFS dir rather than individual files?, etc.)

Can you explain what pinning is (or link a doc?)

If you haven't read it already check out the CONTRIBUTING guide and in particular the section on writing a new backend

Sounds fun :-)

[gcornut]

@ncw Thank you for your encouragements!

I don't know much about IPFS so apologies in advance for any stupid questions ;-)

No problem, I too apologies because I actually never programmed in Go before but after reading some of the code I found it not really difficult to understand so I decided I could give it a try ^^.

I started the implementation on a fork using the rest.Client instead of the go-ipfs-api because it is actually incomplete (I will check into contrubuting into go-ipfs-api I guess ^^):
https://github.com/gcornut/rclone/blob/ipfs-backend/backend/ipfs/ipfs.go

I see what you mean. Would someone tend to use one of those methods exclusively? If so maybe a parameter might be better "access = ipfs" which could go in the config file or as a command line flag --ipfs-access ipns?

Yes, these three different file access are used independently. The parameter could be an option but in the implementation I think I will have to handle the FS operations totally differently.

Is there a time of upload or something like that? That is still useful if there is.

No the IPFS files don't have any metadata besides the IPFS hash and the file size.

Which hash function is it? I saw sha2-256 mentioned in the code. It is relatively easy to add a new hash function to rclone.

I will have to investigate further on how to calculate the IPFS hash in rclone but from what I've seen IPFS first splits files into chunks, each chunk is hashed with sha2-256, the chunks are regrouped into an object representing the file which is itself also hashed and finally even the directory is represented as an object which is hashed (forming a Merkle DAG).

Of course the go-ipfs implements this hashing but I guess it would too big as a dependency to be added into rclone ^^

Can you explain what pinning is (or link a doc?)

To simplify, an IPFS node has a data repository which is garbage collected. When you add your own data into IPFS, this data will be pinned to make sure it won't be garbage collected. In contrast, when you fetch IPFS data, it will be store in your IPFS data repository but unpinned and could be garbage collected.

More info at https://docs.ipfs.io/guides/concepts/pinning/

[gcornut]

I'm thinking I could implement rclone FS operations on IPFS by forking of the original root folder (fork of the DAG), applying all necessary modifications and then I could update the mutable FS (being the local mutable file system or the IPNS shared mutable FS) in an atomic fashion. And I could even detect and crash on concurrent modification of the mutable FS since I could check at the end that the initial folder hash hasn't changed under me.

The only problem is that this it would require to do some operations at the very end of every rclone process writing stuff into IPFS. @ncw Do you have any Fs method executing at the end of the rclone process?

[ncw]

No problem, I too apologies because I actually never programmed in Go before but after reading some of the code I found it not really difficult to understand so I decided I could give it a try ^^.

:-) Go is a reasonably simple language, especially if you've ever done C or Java.

I started the implementation on a fork using the rest.Client instead of the go-ipfs-api because it is actually incomplete (I will check into contrubuting into go-ipfs-api I guess ^^):
https://github.com/gcornut/rclone/blob/ipfs-backend/backend/ipfs/ipfs.go

Nice!

No the IPFS files don't have any metadata besides the IPFS hash and the file size.

That is fine - rclone can cope with that.

I will have to investigate further on how to calculate the IPFS hash in rclone but from what I've seen IPFS first splits files into chunks, each chunk is hashed with sha2-256, the chunks are regrouped into an object representing the file which is itself also hashed and finally even the directory is represented as an object which is hashed (forming a Merkle DAG).

That sounds like that isn't a content hash if it includes the directory. What rclone wants is a content hash that it can calculate locally and compare to the remote hash to detect corruptions.

I'm thinking I could implement rclone FS operations on IPFS by forking of the original root folder (fork of the DAG), applying all necessary modifications and then I could update the mutable FS (being the local mutable file system or the IPNS shared mutable FS) in an atomic fashion. And I could even detect and crash on concurrent modification of the mutable FS since I could check at the end that the initial folder hash hasn't changed under me.

Sounds interesting!

So could the read only parts of rclone work without the mutable file system?

The only problem is that this it would require to do some operations at the very end of every rclone process writing stuff into IPFS. @ncw Do you have any Fs method executing at the end of the rclone process?

Sure there is the atexit lib

Could this be done at regular intervals too?

[gcornut]

That sounds like that isn't a content hash if it includes the directory. What rclone wants is a content hash that it can calculate locally and compare to the remote hash to detect corruptions.

Well it is a content hash since any two IPFS nodes adding the same file to the network will produce the same IPFS hash. So in IPFS the files have content hash (which acts as addresses) but also file chunks and folders have hashes (so you can detect corruption between two folder by comparing the IPFS hash). I just need to find a way to add this IPFS hash function locally in rclone in a way which is compliant with the IPFS spec but without having to import the whole go-ipfs implementation (I will ask the authors of IPFS about that).

Sounds interesting!

So could the read only parts of rclone work without the mutable file system?

Yes the IPFS backend could work on the read only part of IPFS (with the /ipfs/<HASH> addresses) and also on the two mutable file system in IPFS (the IPNS and MFS).

I think I like what you proposed with the CLI parameter. By default the IPFS backend could work on the MFS (local mutable FS) and it could switch context with a parameter like --ipfs-root=/ipfs/<HASH> (read only) or --ipfs-root=/ipns/<HASH> (read/write and shareable).

Some examples of possible CLI interactions:

$ rclone ls ipfs: 
# => Equivalent to `ipfs files ls` listing files from IPFS MFS

$ rclone ls --ipfs-root=/ipfs/<HASH> ipfs: 
# => Equivalent to `ipfs ls /ipfs/<HASH>` listing files from IPFS hash

Also I can easily generate public links to IPFS and IPNS addresses using the public HTTP gateway (http://ipfs.io/ipfs/)

Sure there is the atexit lib

Nice, I will use this.

Could this be done at regular intervals too?

It's a good idea!

[ncw]

Well it is a content hash since any two IPFS nodes adding the same file to the network will produce the same IPFS hash. So in IPFS the files have content hash (which acts as addresses) but also file chunks and folders have hashes (so you can detect corruption between two folder by comparing the IPFS hash). I just need to find a way to add this IPFS hash function locally in rclone in a way which is compliant with the IPFS spec but without having to import the whole go-ipfs implementation (I will ask the authors of IPFS about that).

OK, that makes sense :-)

I think I like what you proposed with the CLI parameter. By default the IPFS backend could work on the MFS (local mutable FS) and it could switch context with a parameter like --ipfs-root=/ipfs/<HASH> (read only) or --ipfs-root=/ipns/<HASH> (read/write and shareable).

Some examples of possible CLI interactions:

$ rclone ls ipfs: 
# => Equivalent to `ipfs files ls` listing files from IPFS MFS

$ rclone ls --ipfs-root=/ipfs/<HASH> ipfs: 
# => Equivalent to `ipfs ls /ipfs/<HASH>` listing files from IPFS hash

Any backend parameter such as "--ipfs-root" can be stored in the config file too, so if you uses one particular thing a lot you could make a new remote for it (eg ipfsFred:).

If the ipfs backend has parameters which all have sensible defaults which are likely to work, then we'll be able to use it without config, eg

rclone ls :ipfs:

For an example of this see the http backend

[gcornut]

Implementation progress report 2019/02/01

https://github.com/gcornut/rclone

Done

• Rclone Fs/Object operations implemented on top of the IPFS DAG object API
• Persist DAG on IPFS MFS on rclone exit

Pains points (making integration tests fail)

• IPFS object size are not the same as local FS file size (discussion discuss.ipfs.io, we might have to wait for the future IPFS unixfs-v2 specs 😢)
• IPFS object don't have modification time (or any other timestamp)

To do

• Implement optional Fs interfaces (Copier, Mover, PublicLinker, etc.)
• Handle IPFS object pinning/unpinning (avoiding potential garbage collection while working on the DAG)
• Persist DAG on IPFS IPNS
• Persist DAG periodically rather than only at the end of the rclone
• Add IPFS hash type into rclone (example using go-ipfs)
• Add documentation

Future IPFS unixfs-v2 specs

The future IPFS unixfs-v2 specs will bring more FS features (like file modification time, true file size, content type and more).
If the problem with the IPFS object size vs local file size can't be solved until unixfs-v2 we will probably have to wait before finishing the IPFS backend for Rclone.
In the meantime, I can continue working on other features listed above (but with failing integration tests)

[ncw]

Implementation progress report 2019/02/01

https://github.com/gcornut/rclone

Done

• Rclone Fs/Object operations implemented on top of the IPFS DAG object API
• Persist DAG on IPFS MFS on rclone exit

Great :-)

Pains points (making integration tests fail)

• IPFS object size are not the same as local FS file size (discussion discuss.ipfs.io, we might have to wait for the future IPFS unixfs-v2 specs cry)

It is possible to work around this. However if we don't know the actual size of the object without decoding it then it will likely make rclone mount misbehave.

Could you try making objects of all the sizes from 0-1024 (say) and we can have a go at making a formula?

• IPFS object don't have modification time (or any other timestamp)

That shouldn't be a problem. Just return a fixed time 1970-01-01 00:00:00 or the time the backend was initialised whichever you think is more appropriate, and make sure the Precision returns ModTimeNotSupported and the unit tests will work fine.

To do

• Implement optional Fs interfaces (Copier, Mover, PublicLinker, etc.)
• Handle IPFS object pinning/unpinning (avoiding potential garbage collection while working on the DAG)
• Persist DAG on IPFS IPNS
• Persist DAG periodically rather than only at the end of the rclone
• Add IPFS hash type into rclone (example using go-ipfs)
• Add documentation

Sounds like you are making great progress :-)

Future IPFS unixfs-v2 specs

The future IPFS unixfs-v2 specs will bring more FS features (like file modification time, true file size, content type and more).
If the problem with the IPFS object size vs local file size can't be solved until unixfs-v2 we will probably have to wait before finishing the IPFS backend for Rclone.
In the meantime, I can continue working on other features listed above (but with failing integration tests)

We can always put workarounds for the integration tests I'm sure!

[beorn]

Any update on this? :)

[gcornut]

I stopped working on the implementation over a year ago just because I lost interest :/
I never got it reviewed but it's implements all read and write operations. I think the only missing part is the implementation of the IPFS hashing algorithm in rclone (necessary for rclone's sync operation).

If anyone wants to finish the implementation or just look at it, it's right here: https://github.com/gcornut/rclone