feat(transports) add support for IPFS and IPNS

[achingbrain]

This PR has come together following our conversation on Twitter.

It adds support for resolving tarballs with CIDs or packuments with IPNS names.

So a users' package.json would be able to contain this sort of thing:

{
  "name": "my-module",
  // ...
  "dependencies": {
    "express": "ipfs://Qmfoo",
    "hapi": "ipns://QmFoo#^18.0.0",
    "http-server": "^0.11.0"
  }
}

Here express is essentially doing something similar to"express": "https://example.com/some-tarball.tgz" whereas hapi is using an IPNS name that would resolve to a packument and the semver range ^18.0.0 would be resolved from that packument and for http-server, the npm cli would talk to the public registry as normal.

To prevent the need to bundle IPFS with npm it uses ipfs-http-client, the idea is that the user would already be running an IPFS node (perhaps through IPFS Desktop or a normal install). They can then either specify the API port on each invocation with --ipfs-url=http://localhost:5001 or just once with npm config set ipfs-url http://localhost:5001.

I'd love to know what you think & work through any required changes, etc.

[achingbrain]

Requires npm/npm-package-arg#37 merging before CI will pass.

[zkat]

whoa, cool. This is a really cool and thorough PR, great job!

So, how does this perform compared to registry or git packages? 🤔

[achingbrain]

Sorry for my delay in replying.

The performance can vary. I tried using installing big-number via a bunch of different methods (chosen because it's reasonably popular and doesn't have any dependencies).

Each time I deleted ~/.npm, package-lock.json, node_modules, as well as ~/.ipfs to ensure every run was uncached, apart from the 'local node' test which required the ~/.ipfs dir to be intact.

I ran each test 10 times, eliminated outliers (anything more than +/- 2x the mean) and took the average & standard deviation.

Data comes from npm's timing logs (e.g. npm timing npm Completed in XXXXms).

http registry

Avg: 830.4ms
Std dev: 22.94ms

git package

Avg: 2703.4ms
Std dev: 196.2ms

Local IPFS node that already has the tarball

Avg: 1022.4ms
Std dev: 15.7ms

Direct connection to IPFS node on the same LAN that has the tarball

Avg: 1977ms
Std dev: 1267ms

Direct connection to remote node that has the tarball (e.g. over the Internet)

Avg: 2697.6ms
Std dev: 1559.9ms

Searching the DHT for the tarball

Avg: 5239.2
Std dev: 2467.1ms

---

So best case it's about 20% slower than http but almost 3x faster than git.

Looking content up via the DHT is the worst, but once you've used it to find a node that can resolve content for your CID you make & keep a connection to them - if you're requesting a bunch of modules chances are that remote node will also be able to resolve your subsequent requests too so over time performance should tend towards the Direct connection to remote node timings, which would make it vaguely comparable to git packages in speed.

[zkat]

Thank you! That's super useful to see. It's nice that it's not much worse than git. I'm definitely pretty surprised that it's still slower than a regular registry dependency when using a local node, or even a LAN connection, since I would've expected IPFS to really shine in that use case. Is that something we can expect to see improving in the future? Are there any cases where you can see that performing better than regular http, now or in the future?

Going forward, I really like having had this as a proof of concept, and I think it's really driving answers to some questions about where distributed package distribution (pun intended) can fit in with the overall npm story. I do think that before accepting something like this, we need to have a better understanding of that story and people's workflows. Ideally, too, if we decided to accept something like this, I imagine I would also want to see support for IPFS-fetched tarballs in registry-based packages so, for example, one could have a centralized index and IPFS-based tarballs.

Anyway, thank you again. I look forward to continuing this conversation. I hope you don't mind me not merging this right now and I want to reiterate how much I appreciate you taking the lead with this work and doing the research necessary to move it forward. Cheers!

[achingbrain]

LAN connection performance

The IPFS node does a bunch more work in the background than HTTP so there's always going to be some overhead (e.g. bitswap, connection negotiation to multiple peers, hashing, verification, etc), plus the npm process is going over HTTP to a local IPFS daemon so there's added latency there.

I can see it being better than regular HTTP if you have lots of nodes on the same network with the content you are after - it will take a chunk from node A, a chunk from node B, etc.

It would be way better than HTTP if you disconnected the Internet or had slow DNS - IPFS is really good with flaky Internet connections. If you were on a plane or in a field or something and someone nearby has the content you are after it would be the only game in town, apart from USB sticks I suppose 😉

But you are right, performance can definitely improve and it's very much a focus at the moment though it could be done outside the scope of this PR.

I would also want to see support for IPFS-fetched tarballs in registry-based packages

This sounds a bit like what I was trying to get at here - we've done some experimenting with adding content identifiers on-demand to the npm registry (search for "cid": here https://registry.js.ipfs.io/pacote for example) so the client would use the meta data from the npm registry for version selection, then IPFS or HTTP or both to retrieve the tarball and it would be completely transparent to the user.

I look forward to continuing this conversation. I hope you don't mind me not merging this right now

Sure - I'm happy to unpack and discuss any ideas in this thread further.

It would be really great to get IPFS/dweb support into the npm cli though - what would it take to get this merged?

[wighawag]

Please have a look at the discussion for proper url scheme : ipfs/kubo#1678

in particular : ipfs/kubo#1678 (comment)

So instead of ipfs://Qmfoo... it should be fs:/ipfs/Qmfoo...
and ipns://QmFoo#^18.0.0" should be fs:/ipns/QmFoo#^18.0.0"

[victorb]

@zkat

Is that something we can expect to see improving in the future? Are there any cases where you can see that performing better than regular http, now or in the future?

I'm pretty sure that's a implementation-specific issue.

For Open-Registry, I wrote a WIP called Bolivar that basically acts as a HTTP>IPFS gateway locally, but connecting directly with some peers to facilitate content-discovery.

In every case I've tested, Bolivar is the fastest way of installing packages, even for first-time hits, and the proxy being connected to Open-Registry instead of directly to npm.

[lidel]

lease have a look at the discussion for proper url scheme : ipfs/go-ipfs#1678

@wighawag oops, the issue you linked is from 2015, we need to update it.
FYSA since then fs:/ got deprecated and we switched to ipfs://, ipns:// and universal dweb:/:

• relevant discussion was summarized in Background on Address Scheme discussions (dweb, ipfs:// and more) ipfs/specs#152
• memo about current consensus can be found at ipfs/in-web-browsers/ADDRESSING.md