feat: ipfs-unixfs get or "walk over anything"

[koivunej]

Add /get endpoint, which passes modified conformance tests (PR not yet created). There are "a few" commits throughout the days but I sadly did not find a way to make this any smaller.

Most importantly this adds ipfs_unixfs::walk::Walker which can be used to walk the whole dag-pb/unixfs tree in a fashion useful when exporting files to filesystem or creating a tarball out of those. If you look at earlier commits, this used to be ipfs_unixfs::dir::walk::Walker. Initially it had a much less ergonomic API as well, but it has been ironed out.

This PR includes some minor changes to path walking to fix the issues created by refactoring ipfs_unixfs::UnexpectedNodeType and eyeballing the code correctness.

Reviewing:

• I'd start with the 1k walk.rs file, look at the tests first
• for a more complicated version of the walks, you can find the /get implementation

Making the Walker api more ergonomic resulted in a few more internal unwrap's which I think are all good.

Pending in this PR:

• add hamtshard checks when loading those to Walker
• get windows specific code to compile
• remove the double buffering in tar helper

Later:

• unify InnerKind in walk.rs Root versions and use Cids in all of them (or none of them)
• find a way to expose more of the Entry values at ContinuedWalk level; not sure how to do this without it becoming a self-referential struct, perhaps Walker::continue_walk could become Walker::continue_walk(&mut self, ...)? The current item could always be moved to the ContinuedWalk.
• migrate IpfsPath and ipfs_http::v0::refs::walk_path over to ipfs under ipfs::Ipfs::resolve and merge the IpfsPath, leave only supported features
• cleanup ipfs::Ipfs::{get, add} which only work on single block files

Open questions:

• is the fact that this contains code from tar clearly enough marked, should we add notices elsewhere? An external fork of tar would be one way as the approach taken here is wildly different from async-tar or tokio-tar, which switch from std::io traits to the async versions. Created issue Exposing currently private builder.rs, header.rs fns alexcrichton/tar-rs#235

[ljedrz]

nit: unwrap_or_else can be used here

[koivunej]

Now that I have the http api and conformance tests going in another branch based on this, and there'll be a lot of rework of unmerged code, I'll probably scrap this review and open a new one a bit later.

[koivunej]

It appears I ended up just continuing where I left off. I'll try to write a better PR description before ready for review.

[koivunej]

This seems to be ready for review, with the linked tar-rs issue and such! Even the CI agrees: only transient failures left (macos sigsegv and nightly probably timeouts -- 23min without a test log).

[ljedrz]

Wouldn't we want to provide the underlying bytes of a symlink that is not UTF-8? It might be "mangled" in a way that is not visible to the user, so if this is not immediately thrown only when given a direct symlink, it would be a good idea to indicate where the issue is.

[koivunej]

This error will actually be eaten by hyper as it currently doesn't support returning trailer headers. No one would ever see the bytes. The tar functionality is not exposed in the ipfs crate. This is by design, we currently can't do it using current tar-rs interfaces.

[koivunej]

The "design" idea was that we could provide the means to get to local filesystem, which we almost do with ipfs::unixfs::ll::walk::Walker (or ipfs_unixfs::walk::Walker). Exporting to filesystem safely is a bit different task which I don't want to grow this massive PR into.

[koivunej]

Slight rebasing, merging.