Properly Handling Filesystems

[jbenet]

So far, we've only been handling "regular files" and "directories". We've recently added symlink support (#1627, thanks @whyrusleeping). Named pipes and other special files are still not supported and continue to hang (#1072).

• ipfs shouldn't hang
• we should decide how to handle them.

i think git ignores everything else. But since ipfs is more general than git (i.e. we may store filesystems representing containers or entire VMs) we must go through all special files and decide how to handle them. (I suspect that -- to properly represent containers and VMs -- we will need to be able to ingest + reproduce them all). Worth mentioning that we should be able to handle other non-UNIX platform-specific things in an extensible way, like solaris Doors, xattrs in linux/bsd/aix/osx, and so on.

I suggest the following process:

• enumerate all filesystem files + attributes that we may want to track
  • defer most things (like MIME types, xattrs, etc) to the "Metadata" object
• suggest patches to unixfs that will address the requirements
• select one + implement it

The standard file types in UNIX are (am i missing any?):

• Regular files. added as unixfs File type.
• Directories. added as unixfs Dir types.
• Symlinks. added as unixfs Symlink type
• Named pipe. (unixplex pipe)
• (Unix) Sockets. (duplex pipe)
• Device files.

Non-UNIX but widely used:

• Doors in Solaris
• xattrs in Linux/BSD/OSX/AIX/...
• whatever windows does

Transparent resolution across object shards

This is not super relevant, but comes to bear here. We will have to teach tools to transparently resolve across shards. This is already done for "chunked files" (i.e. files that are "large" and thus sharded across multiple objects). And we will have to do it for "chunked dirs" too (not yet supported). (Other datastructures, like keychain's Signatures will need similar resolving as well).

FWIW, most git tools do this transparent resolution for the commit.tree pairing-- that is, commits have one tree, so referencing <git-commit>/foo/bar really means <git-commit>/tree/foo/bar in most git tools, including github.

cc @whyrusleeping @rht @wking @tv42 @mildred

[mildred]

You can add to the list resource forks

[tv42]

Resource forks are just bastard xattrs, these days. The only difference is that they run the API as a "write B at offset N" instead of "replace with value B". See https://godoc.org/bazil.org/fuse#SetxattrRequest Position.

[tv42]

Also, Apple wants people to use sqlite or something, and not resource forks, these days.

[crackcomm]

@jbenet I can see permissions are entirely discarded in unixfs spec and implementation, obviously mainly because there is no way to use them properly, all the time. Files with different permissions will have different hashes in the end but that is not a big problem. Lack of permissions in IPFS file system does not help anyway. Many important use cases are impossible because of this gap.

Personally I am waiting for the day I can host entire linux on IPFS. Currently IPFS would be but cannot be used in container runtimes.

[cchris-org]

I stuck in same issue. the only solution for me is tar whole container:
tar . -c I ipfs tar add
sorry, no pipe symbol on my phone :)
but this could not be a permanent solution.

[magik6k]

The plan to address this issue is currently discussed in ipld/legacy-unixfs-v2#1