[BOUNTY] Implement UnixFSv1.5 in go-ipfs

[Stebalien]

This issue has a bounty!

Successfully closing this issue by producing a production-ready, mergeable PR can earn you not only the undying love of the IPFS community — it can net you a financial reward. See the current list of bounty issues and their values here.

The need in brief

Add UnisFSv1.5 metadata support per the spec https://github.com/ipfs/specs/blob/master/UNIXFS.md#data-format, which includes: specifying metadata, respecting existing metadata, and displaying metadata.

This aims to support large data import use cases (ex package managers), who need to retain file metadata, particularly last modified time (mtime), in order to selectively sync only data that has changed. Up until now if you wanted to host a large data set on IPFS, like a package manager's repository, it would be difficult to update - but with file mtime you can only reimport the changed files, making things much more efficient.

Deliverable

• Implement the new mode & mtime fields and obey the new rules around default modes & mtimes as per the metadata spec.
• Plumb this metadata all the way through from ipfs add on one IPFS node to ipfs get on another.
• An end-to-end sharness test that validates that this metadata is preserved.

Requirements

• This feature must be interoperable with js-ipfs's implementation of the UnixFSv1.5 spec.
• The behavior of ipfs add must not change except when the user has requested that metadata be preserved via the appropriate flags (see the implementation details below). Specifically, the final hash produced by ipfs add must not change.
• The final hash of files added with this feature enabled must match the hash produced by javascript.

Guidelines

• Please use the IPFS Go Contributing Guidelines as your north star — adherence to these guidelines are crucial when it comes to awarding a potential bounty!
• Please DO NOT bundle a general code refactor with the completion of this bounty!

[momack2]

For context, here is the js-ipfs unixfsv1.5 implementation done by @achingbrain: ipfs/js-ipfs#2621, and the blog post describing the shipped work: https://blog.ipfs.io/2020-02-13-js-ipfs-0-41/#unixfs-v1-5

If you want to start working on this, ping @Stebalien for a quick walkthrough of where to start and what already exists to build on top!

[Stebalien]

Implementation Notes

Implementing this feature will touch several sub-components of go-ipfs:

• The go-ipfs-files library will need to be updated to:
  • Support modification times and modes.
    • When reading from and writing to disk.
    • When reading from the web (WebFile).
    • When sending/receiving over HTTP using the "multipart" file types.
  • Expose modification times and modes via the Node interface.
• Implementations of the Node interface from go-ipfs-files will need to be updated:
  • github.com/ipfs/go-unixfs/file/unixfile.go
  • github.com/ipfs/go-ipfs-http-client/apifile.go
• The protobuf definitions in github.com/ipfs/go-unixfs/pb/unixfs.proto will need to be updated and regenerated according to the spec.
• The go-unixfs (de)serialization logic in go-unixfs will also need to be updated to (de)serialize modes/mtimes.
• The go-ipfs file adder (github.com/ipfs/go-ipfs/core/coreunix/add.go) will need to be extended to handle modification times.
• The go-ipfs "add" command (ipfs add) will need two new flags (defaulting to false):
  • --preserve-mtime -- record the file modification time when adding.
  • --preserve-mode -- record the file mode when adding.

The new go-ipfs-files Node interface should be:

// Node is a common interface for files, directories and other special files
type Node interface {
	io.Closer

	// Mode returns the file's Mode
	Mode() os.FileMode

	// ModTime returns the files last modification time.
	//
	// If the last modification time is unknown/unspecified, ok is false and
	// the mtime is the Unix epoch (default timestamp).
	ModTime() (mtime time.Time, ok bool)

	// Size returns size of this file (if this file is a directory, total size of
	// all files stored in the tree should be returned). Some implementations may
	// choose not to implement this
	Size() (int64, error)
}

[kalmi]

@Stebalien Hi, @momack2 said to ping you for a walkthrough if one would like to start working on this, but I think it is safe to assume that your last comment is that walkthrough, and I will just go ahead. Thanks!

[achingbrain]

ipfs add also needs the following new options:

• --mode - specify a file mode
• --mtime - specify the seconds component of the mtime
• --mtime-nsecs - specify the nanosecond component of the mtime

And there are two new mfs commands:

• ipfs files touch [path] (update mtime to the current time)
  • --mtime - optionally specify the seconds component of the mtime to update to
  • --mtime-nsecs - optionally specify the nanosecond component of the mtime to update to
• ipfs files chmod [mode] [path]

[momack2]

Hey @kalmi! How are things going? Did you get your questions answered / are you unblocked?

[kalmi]

Hi!

Thanks. I'm not blocked, I had some personal issues last week, sorry for the silence. I plan to spend most of the upcoming long weekend on this. So far I have got ipfs get to use perm modes passed from the tarwriter, and I might have overdone the handling of the edge cases that can happen during ipfs get, but probably not. I want it to update modes for files and directories that already exist without doubling the number of syscalls, and that proved to be a bit tricky.

Status (WIP, "done" is very loosely defined at this point, this list is bound to be incomplete):

• go-unixfs
  • Regenerate unixfs.pb.go
  • Update go-unixfs serialization logic 
• go-ipfs-files
  • Implement ModTime and Mode (default-returning) stub for all Node implementers
  • Change Node interface
  • Use ModTime and Mode in tarwriter (used by ipfs get)
  • serialfile
  • WebFile
    • Check if there is such a thing in js-ipfs
    • Use headers (What headers?)
  • multipart
    • Check used headers in js-ipfs
    • Use headers
  • filewriter
    • Find where this is actually used
  • others?
• go-ipfs-http-client (yet to investigate)
• ipfs get (this mostly touches whyrusleeping/tar-utils in a backward compatible way)
  • mode perms
    • umask is respected (maybe too much, even for preexisting file, that might be misguided)
    • mode perms are updated for preexisting directories and files
  • set non mode perm bits
  • set mtime
  • sharness
• ipfs add
  • go-ipfs/core/coreunix/add.go updated
  • --preserve-mtime implemented
  • --preserve-mode implemented
  • --mode
  • --mtime, --mtime-nsec
  • sharness
• mfs commands
  • touch
  • chmod
  • sharness
• Investigate OS time resolution differences
• Consider abusing NTFS alternate streams to store mode
• FUSE?

[Stebalien]

I'd drop the last two for now. Everything else looks good to me.

Note: If you're trying to find where different types/functions are being used, I recommend grep.app. E.g., .NewMultiFileReader.

[momack2]

Hey @kalmi! Any status update? Are you blocked?

[kalmi]

@momack2 I am making progress, albeit unfortunately it is slower than I originally expected due to external factors. Now I am at the point where I believe the get side of things to be mostly done, and I am moving on to extending add, and I hope to be able to write back the added files with metadata somewhen in the next few days. I wanted to test aginst jsipfs before this as a sanity check, but I had some bad experiences it with it, and eventually decided to push it to the end. In the meantime, I will use the hashes from its test cases for the go-ipfs tests.

I am making sure that all changes to the libraries are backward compatible. The Node interface change specified by Stebalien is not, but I guess that's reasonable and acceptable. When I have everything working, I will start making PRs in the proper merge order.

Do you have a reasonable timeframe in mind for this that I should be aware of?

[momack2]

Hey @kalmi! It'd be great to include this feature in the next release, go-ipfs 0.6. I think the feature cut for that release is ~2 weeks from Wednesday. Do you think that's a reasonable goal?

Note, the sooner to put up draft PRs for review (and the more modular/well tested they are) the easier it will be to land this quickly. =]

[achingbrain]

I wanted to test aginst jsipfs before this as a sanity check, but I had some bad experiences it with it

Could you open an issue on ipfs/js-ipfs please and we'll see if we can get to the bottom of the problem you had?

[Stebalien]

@kalmi Any status updates?

[aschmahmann]

Thanks @kstuart the implementation notes in this issue are hopefully clear enough to guide you in your implementation.

I'd recommend putting up work in progress PRs early and poking folks (e.g. me) for feedback even at the earlier stages instead of waiting to put up a mostly completed PR. This should make things easier to review and make sure that we're on the same page.

I'd also recommend breaking the different steps/sections up into different commits and/or PRs so that they are easier to review + merge.

[kstuart]

Draft PRs submitted, as I wrote this on the hoof I'll not be adding any further functionality for now but focusing on exploratory testing and authoring tests, probably not till the weekend though.

For anyone wanting to try out the new features you'll need all the related PRs, you should then be able to use the functionality detailed in #7754, testing on Windows would be welcome.

For webfile, as I'm not aware of an existing X- header for communicating file mode I'm initially proposing X-FileMode, this is easily changed.

[achingbrain]

I'm initially proposing X-FileMode

We had to change the js implementation around passing file mode/mtime to the HTTP API - because in the browser you have to use FormData objects to construct requests from multiple Blobs in order to prevent buffering file data into memory. Each Blob you add becomes a multipart part - but you cannot specify per-part headers with the FormData API.

So what we ended up doing was encoding the mode/mtime as a querystring and appending it to the field name because it was essentially the only field we could modify that didn't take user input (like the filename for example), so your request ends up looking something like:

POST /api/v0/add HTTP/1.1 
Host: localhost
Content-Type: multipart/form-data;boundary="boundary" 

--boundary 
Content-Disposition: form-data; name="file-0"; filename="/file-with-no-metadata.txt" 

...file data 
--boundary 
Content-Disposition: form-data; name="file-1?mode=0755"; filename="/file-with-mode.txt" 

...file data
--boundary
Content-Disposition: form-data; name="file-2?mode=0755&mtime=500"; filename="/file-with-mode-and-mtime.txt" 

...file data
--boundary
Content-Disposition: form-data; name="file-3?mode=0755&mtime=500&mtime-nsecs=4982"; filename="/file-with-all-the-metadata.txt" 

...file data
--boundary--

If go-IPFS could implement the same thing it would mean this feature will work across both implementations and in browsers.

[kstuart]

@achingbrain, for multipart I've implemented the querystring on field name, the name itself doesn't include an index as the way parts are constructed doesn't lend itself easily to doing so, in this context though I wouldn't expect that to be an issue, e.g.
Content-Disposition: form-data; name="file?mode=0644&mtime=1604320500&mtime-nsecs=55555"; filename="beep.txt"

For the webfile case e.g. ipfs add http://example.com/somefile.txt the current implementation uses the following response headers when present:

• Last-Modified -- to set the last modified timestamp
• X-FileMode -- (as an octal string) to set the unix file mode

Does the js implementation have a position on this?

[achingbrain]

No, looking for those headers in the response sounds perfectly reasonable.

The X- prefix has been deprecated for a few years but my opinion isn't that strong.

File-Mode might be more conventional than FileMode.

[lidel]

Given that we already return X-Ipfs-Path I'd go with Ipfs-File-Mode (assuming we want to drop X- at some point anyway) just for the sake of consistency and emphasis this metadata comes from IPFS.

[achingbrain]

This isn't for what we return though, it's what we will look for in HTTP responses when we are importing from a URL - expecting third parties to prefix headers with IPFS- just in case someone tries to ipfs add the URL seems unreasonable.

[achingbrain]

@kstuart I've put a repo together that'll allow you to run the interface tests js-IPFS runs but with minimal setup: https://github.com/achingbrain/ipfs-interface-tests

There's a readme with instructions but in brief once you've got everything running you'll just need to remove the skips for all the tests that mention mode, mtime or metadata in test/interface.spec.js

Please let me know if you have problems running it or need any help.

[kstuart]

@achingbrain That's great, and very insightful.

Should I be concerned about the preloadNode issue?

[achingbrain]

Should I be concerned about the preloadNode issue?

If you pull from the repo this should be fixed now, sorry about that

[kstuart]

@achingbrain looks like there are still some issues running the tests, specifically when running the browser tests process doesn't exist:

FAILED TESTS:
  interface-ipfs-core over ipfs-http-client tests against go-ipfs
    .add
      ✖ should add with mtime as hrtime
        Chrome Headless 91.0.4455.2 (Linux x86_64)
      TypeError: process.hrtime is not a function
          at Context.eval (interface-ipfs-core/src/add.js:269:29)

Could you please investigate, the package browser-hrtime might provide a solution.

[zacharywhitley]

Any idea what the current status of this is? I'd really like the feature and it seems soooooooo close. I'm not interested in the bounty but would really like to see it get over the finish line and if there's any way I could give it that final push I'd be happy to help out.

[kstuart]

@zacharywhitley so far this year it has been challenging finding the time to focus on this, I'll be looking to move the PRs out of draft within the next 6/8 weeks, if not sooner.

[markg85]

@kstuart I'm really curious about your progress here!

I would very much like to use this additional stat information in places where i'm (going to) list content from folders stored on IPFS.

Just for my information. This additional stat info you're providing is also there when using the HTTP API, right? Not just internal to the unixfs layer? I'm asking because i intent to add IPFS browse support in KODI as a VFS plugin where it in the backend would use the HTTP API (it's near certain that i will be doing this). Likewise i want to add a KIO SLAVE (it's for KDE's file browsing capabilities) to support IPFS natively in KDE applications too, it too would fetch file data via the HTTP API. It's unlikely i'll be doing the KDE side of things anytime soon though, that really is just a wishlist item for me personally that keeps getting pushed back.

[kstuart]

@markg85 yes, if stored with mode and/or time that information is returned from the HTTP API. As per screenshot it returns the same fields as js-ipfs, except for WithLocality field.

I need to review Mtime value, but aside from that you can rely on those fields being present if they were stored.

[zacharywhitley]

Really looking forward to this. Quick question that I could probably answer by looking at the branch but you can make things easy on my if you don't mind my masking. Is this going to include an update to fuse to pass the info on to mounts? That's what I really want it for. I'm looking to replace SDKMan with IPFS so installing a new JDK is as simple as pinning the directory for that version and updating a couple of env vars.

[markg85]

Nice!
Can't wait for this to land and an IPFS version be released with this!
I don't want to rush you... but which version to you "soft target" for including this? 0.11?

[kstuart]

@zacharywhitley it's not in scope for this ticket, don't know what the plan is there.

@markg85 all I can say is I'm currently working on it, barring any major issues it shouldn't be much longer.

[zacharywhitley]

Thanks. That's fine. I was planning on writing a separate fuse implementation that also supported MFS so as long as it's there.

[djdv]

@zacharywhitley
For what it's worth, a while ago I had code for this within the go-ipfs repo.
There's a rough demo of that version here: https://www.youtube.com/watch?v=FXi3yYO2H1w
(this would later see some changes)

More recently, I've been asked to make a standalone version of this, and have been porting those features over to it.
A rough demo of that is here: https://www.youtube.com/watch?v=q-hKANpM9Xg
While not done yet, the port of the MFS code is planned and should probably happen soon-ish.
Although I'll need to coordinate some means of actually syncing changes back to the go-ipfs node in cases where you want to target it's ipfs files root rather than just IPNS keys, or arbitrary CIDs. Basically something like ipns name publish for the files root, where I can just tell it to replace its root CID with another one.

If you need help with yours, feel free to ping me.

[zacharywhitley]

Oh wow. That's fantastic and thank you for the offer of help. I will definitely keep in touch. You're from Philly? I went to Drexel University and used to work up in King of Prussia.

I've always thought that IPFS mount was an unfortunate rough spot for IPFS that didn't get the love it deserved. It's called IPFS and you go to use it as a filesystem and it's a little disappointing. I guess this is more an MFS thing but I always thought it would be nice if it also supported snapshots. I'm thinking of it creating a hidden directory every time the root changes to move the old root to something like .snapshot-20210826-3489. It's immutable. I should be able to go to an old version.

[djdv]

You're from Philly?

I am, I'm at the borders of Tacony and Holmesburg. :^)

It's called IPFS and you go to use it as a filesystem and it's a little disappointing

Strongly agree. It was the only reason I downloaded it in the first place, only to be greeted with a "not supported yet" message.
And from what users tell me, the only platform left that it does work on isn't practically usable.
Someone posted a few days ago about a 75-80% CPU reduction using my initial unoptimized version, and a fraction of the time.
Compared to the current release. #5003 (comment)
Not too bad.

snapshots

Yeah this would be really cool. It's something I like a lot in systems like ZFS, and shouldn't be too difficult to come up with a scheme for in IPFS.

the port of the MFS code is planned and should probably happen soon-ish

For what it's worth I at least ported over the read-only portions of this for now. They're here.
It seemed to work, but I haven't wired up a way to pass a CID to it yet via the CLI. I need to fetch it from node using a different API. Unfortunately it looks like no progress has been made on this ipfs/interface-go-ipfs-core#19
So I'm going to have to come up with some roundabout solution for now. Likely I won't mount and sync the root, but instead some subdirectory like /mount, until we can do something in the IPFS APIs to be able to set the root, or manipulate it directly.