support --nocopy when adding URLs #735
Conversation
| @@ -349,6 +349,10 @@ cluster "pin add". | |||
| Value: defaultAddParams.ReplicationFactorMax, | |||
| Usage: "Sets the maximum replication factor for pinning this file", | |||
| }, | |||
| cli.BoolFlag{ | |||
| Name: "nocopy", | |||
| Usage: "Add the URL using filestore. Implies raw-leaves. (experimental)", | |||
There was a problem hiding this comment.
Doesn't it add to the urlstore? I've had some problems figuring out how this is now working in ipfs (I haven' t been able to find the place where the URL is taken from the multipart request and used for the urlstore - I suspect it's somewhere in unixfs), but if urlstore is not enabled, adding urls like this on ipfs fails.
There was a problem hiding this comment.
WebFile now implements FileInfo, and returns the URL via FileInfo.AbsPath(), which is why the upgrade to go-ipfs-files v0.0.2 is required. This is passed along via the abspath header, which ends up in the ReaderFile. It is then extracted and set as DagBuilderParams.fullpath https://github.com/ipfs/go-unixfs/blob/master/importer/helpers/dagbuilder.go#L85-L88, which in combination with the nocopy flag, triggers the same underlying mechanism as urlstore, but urlstore is not directly used. Only filestore is required https://github.com/ipfs/go-ipfs/blob/master/core/coreapi/unixfs.go#L52-L54. Here is the read logic: https://github.com/ipfs/go-ipfs/blob/master/filestore/fsrefstore.go#L138
There was a problem hiding this comment.
As far as I can tell, enabling urlstore makes the command available, but that command just leverages filestore internals to actually do the work.
There was a problem hiding this comment.
Thanks, that's very useful. I have opened ipfs/kubo#6139
There was a problem hiding this comment.
I wonder, when adding a file like this, is the DAG builder outputting a single special unixfs block (symlink)? This is my intuition but I haven't verified it.
I don't see issues with this PR (thanks for submitting). Having a test would be great (adding a web file from a url with the api/rest/client and then checking that only a single symlink block has been generated and put on ipfs. But it's probably more complicated than the PR itself and I'm happing merging now if we're sure this works.
Not sure I fully grok your terminology, but it creates a |
Ah right, I understand now. Adding will chunk and build the resulting DAG blocks in the same way, but those blocks are filestore nodes, and instead of having the data payload they have a link to source+offset. Thanks! |
ipfs/kubo#6065 added URL support to the
addcommand, along with--nocopyas well, replacing the need for the experimentalurlstore. This PR proposes supportingipfs-cluster-ctl add --nocopy <url>.