Support for ipfs add <url>

[NatoBoram]

Version information:

go-ipfs version: 0.4.19-
Repo version: 7
System version: arm64/android
Golang version: go1.11.5

Type:

enhancement

Description:

I noticed that IPFS Cluster supported ipfs-cluster-ctl add <url>, but that feature isn't in IPFS. I think it's a pretty amazing command and I'd rather use that than urlstore.

Relevant : ipfs-cluster/ipfs-cluster#701 (comment)

[Stebalien]

Completely agree. Personally, I'd like to replace urlstore add with ipfs add --nocopy URL but that's a bit trickier.

FYI, if you'd like to implement this, it shouldn't be hard. We already have a "WebFile" file type (defined in go-ipfs-files). All we need to do is to modify the commands library (go-ipfs-cmds) to detect URLs passed as file arguments and use the "WebFile" type instead of the normal file type.

[Jorropo]

@Stebalien I will try for my first contrib here.

[jmank88]

@Jorropo I've started hacking on this as well. Happy to collaborate if you want.

I've got go-ipfs-cmds/cli.Parse recognizing urls and using files.WebFile. It works (produces expected idential CID and is retrievable) but seems messy (unexpected parent paths added, https://->https:/) and doesn't work with --nocopy. Here is an example run with a fake url:

> ipfs add https://google.cloud.storage/v0/b/bucket/o/Qm...?alt=media&token=XXXXX
added Qm... https:/google.cloud.storage/v0/b/bucket/o/Qm...?alt=media
added Qm... https:/google.cloud.storage/v0/b/bucket/o
added Qm... https:/google.cloud.storage/v0/b/bucket
added Qm... https:/google.cloud.storage/v0/b
added Qm... https:/google.cloud.storage/v0
added Qm... https:/google.cloud.storage
added Qm... https:
> ipfs cat Qm...
hello world!

The url is recognized and a files.WebFile created (by cli.Parse), but then the content is streamed over multipart/form-data and http.parseRequest creates a files.multipartDirectory on the other end. What I'm really interested in is the --nocopy case, but I don't see an obvious way of sending the url over the http request for http.parseRequest to create it's own files.WebFile. @Stebalien Am I overlooking something simple? Or will the existing format need to be extended?

[Stebalien]

With respect to the file paths, I'm not sure, I'd have to see the code.

For nocopy, we do actually (fow now) send the files over the HTTP API. We just also send the full path of the file in an abspath mime header. We then hash the data like usual except, instead of putting it into our local datastore, we throw it away and record a reference to the file from which it came.

We handle ipfs urlstore slightly differently. There, we send a URL as text (normal string argument) and download the file on the daemon.

The easiest path for ipfs add --nocopy URL would likely be to do what ipfs add --nocopy FILE does and just stick the URL in the abspath header. There'll be a slight performance hit but we have to download the file regardless (to hash it).

TL;DR: Make WebFile implement FileInfo.

[jmank88]

TL;DR: Make WebFile implement FileInfo.

Great! And that should work for cluster-ctl automatically as well, no?

[Stebalien]

I believe so. Actually, I really don't know.

[Jorropo]

@jmank88 personaly I just copied the code from urlstore command to add and with an filestore.IsUrl test if that an url, if it is just fire the good one, I'm moving this code to an external file to not produce duplicate (wich is ironicaly very hard for me because I'm new with go).

[jmank88]

• return url as AbsPath from WebFile to implement FileInfo go-ipfs-files#13
• support url paths as files.WebFile go-ipfs-cmds#154
• set WebFile fpath to URL base go-ipfs-cmds#158
• support add --nocopy <url> #6118 folded in to coreapi unixfs: change Wrap logic to make more sense  #6019

[jmank88]

@Stebalien second PR is up as a draft. Blocked waiting for release of go-ipfs-files v0.0.2.

[jmank88]

@Stebalien Third is up. Blocked waiting for go-ipfs-cmds v0.0.4.

[Stebalien]

All done and merged.