New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

npm on IPFS #2

Open
jbenet opened this Issue Apr 18, 2015 · 61 comments

Comments

Projects
None yet
@jbenet
Member

jbenet commented Apr 18, 2015

  • ipfs-blob-store
  • ipfs-daemon-ctrl control whatever daemon is installed
  • plug in ipfs-blob-store to reginabox
  • figure if we can 302 publish
  • mirror the registry
  • bundle it all into one module
  • make physical nodes

cc @bengl

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Apr 18, 2015

Member

just idle thought? Or discussion?

Member

whyrusleeping commented Apr 18, 2015

just idle thought? Or discussion?

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Apr 18, 2015

Member

@whyrusleeping notes from discussion in person :)

Member

jbenet commented Apr 18, 2015

@whyrusleeping notes from discussion in person :)

@bengl

This comment has been minimized.

Show comment
Hide comment
@bengl

bengl Apr 23, 2015

As mentioned in the first checkbox, one blocker for this is that someone needs to make a ipfs-blob-store adhering to abstract-blob-store. I'm working on this on and off in my spare time. Slowly. If someone beats me to it, yay!

bengl commented Apr 23, 2015

As mentioned in the first checkbox, one blocker for this is that someone needs to make a ipfs-blob-store adhering to abstract-blob-store. I'm working on this on and off in my spare time. Slowly. If someone beats me to it, yay!

@seidtgeist

This comment has been minimized.

Show comment
Hide comment
@seidtgeist

seidtgeist May 26, 2015

@jbenet Hmm... After you explained the ipfs docker mount hack yesterday, what if we mount ipfs on node_modules and require('hash')? Why am I even asking and not just trying this? brb

seidtgeist commented May 26, 2015

@jbenet Hmm... After you explained the ipfs docker mount hack yesterday, what if we mount ipfs on node_modules and require('hash')? Why am I even asking and not just trying this? brb

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping May 26, 2015

Member

@ehd how did that work out?

Member

whyrusleeping commented May 26, 2015

@ehd how did that work out?

@seidtgeist

This comment has been minimized.

Show comment
Hide comment
@seidtgeist

seidtgeist May 26, 2015

@whyrusleeping It works :)

$ cat add.js
'use strict';

module.exports = function(a, b) {
  return a + b;
};

$ ipfs add add.js
added QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x add.js

$ cat prog.js
'use strict';

var add = require('QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x');

console.log(add(1, 2));

$ node prog.js
3

Two things:

  • I'd rather not rely on the filesystem or fuse, that's doable
  • I'd rather require names instead of SHAs and move the mapping to metadata

seidtgeist commented May 26, 2015

@whyrusleeping It works :)

$ cat add.js
'use strict';

module.exports = function(a, b) {
  return a + b;
};

$ ipfs add add.js
added QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x add.js

$ cat prog.js
'use strict';

var add = require('QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x');

console.log(add(1, 2));

$ node prog.js
3

Two things:

  • I'd rather not rely on the filesystem or fuse, that's doable
  • I'd rather require names instead of SHAs and move the mapping to metadata
@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping May 26, 2015

Member

@ehd thats great! what would the ideal interface be for you to want to use ipfs for js modules?

Member

whyrusleeping commented May 26, 2015

@ehd thats great! what would the ideal interface be for you to want to use ipfs for js modules?

@seidtgeist

This comment has been minimized.

Show comment
Hide comment
@seidtgeist

seidtgeist May 26, 2015

Ideally we'd have a simple bootstrapping js/node program in ipfs that can
load other things from ipfs using require. Reminds me of the StackStream
opcodes actually ☺️ Anyway, without a file system the ideal
delivery mechanism would be the true ipfs network, so baking a simple ipfs
client into a runnable JavaScript bundle sounds like an idea worth trying.
Initially this could be done by just using the global or local ipfs http
gateway for loading code. I'm not sure whether a dynamic module loader
(require on demand) or a dynamic bundler (compile program, then start
program) would be more viable. Whatever runs in <script src="browser-fetchable-address-on-ipfs"/> or vanilla node I guess.

Another related idea we've bounced: storing computational results in ipfs.
Say you have blobs for js-compile and js-source-with-links; you can load
both and run js-compile(js-source-with-links), then store the result and
use that as the above entry point. It's basically how all browser side JS
already works, just that it's very sad how we store build results in dumb
files without linking to how they've been conceived (creation metadata! cc
@jbenet).

Side note: Ironically I'm in a place without wifi and just looked up
stackstream on my mac's browser history, but of course I can't access the
already cached page. Idea: any ipfs browser/browser extension should pin
history items. Immediate benefit over any other real world browser.

seidtgeist commented May 26, 2015

Ideally we'd have a simple bootstrapping js/node program in ipfs that can
load other things from ipfs using require. Reminds me of the StackStream
opcodes actually ☺️ Anyway, without a file system the ideal
delivery mechanism would be the true ipfs network, so baking a simple ipfs
client into a runnable JavaScript bundle sounds like an idea worth trying.
Initially this could be done by just using the global or local ipfs http
gateway for loading code. I'm not sure whether a dynamic module loader
(require on demand) or a dynamic bundler (compile program, then start
program) would be more viable. Whatever runs in <script src="browser-fetchable-address-on-ipfs"/> or vanilla node I guess.

Another related idea we've bounced: storing computational results in ipfs.
Say you have blobs for js-compile and js-source-with-links; you can load
both and run js-compile(js-source-with-links), then store the result and
use that as the above entry point. It's basically how all browser side JS
already works, just that it's very sad how we store build results in dumb
files without linking to how they've been conceived (creation metadata! cc
@jbenet).

Side note: Ironically I'm in a place without wifi and just looked up
stackstream on my mac's browser history, but of course I can't access the
already cached page. Idea: any ipfs browser/browser extension should pin
history items. Immediate benefit over any other real world browser.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Jun 2, 2015

Member

Update on the list above: @krl made https://github.com/ipfs/node-ipfsd-ctl


@ehd all of that sounds awesome!

Ideally we'd have a simple bootstrapping js/node program in ipfs that can
load other things from ipfs using require.

Yeah! like a special require that uses ipfs directly. Could patch npm to make requires that start with "/ipfs/..." require straight out of ipfs.

Btw, what @bengl and I want to do above with reginabox will work with stock npm and require.

Reminds me of the StackStream
opcodes actually ☺️ Anyway, without a file system the ideal
delivery mechanism would be the true ipfs network, so baking a simple ipfs
client into a runnable JavaScript bundle sounds like an idea worth trying.

that sounds awesome

Initially this could be done by just using the global or local ipfs http
gateway for loading code. I'm not sure whether a dynamic module loader
(require on demand) or a dynamic bundler (compile program, then start
program) would be more viable. Whatever runs in <script src="browser-fetchable-address-on-ipfs"/> or vanilla node I guess.

@krl's https://github.com/ipfs/node-ipfsd-ctl can make ephemeral ipfs nodes using the go-ipfs package, so you dont even have to install ipfs beforehand. it's huge (cause go-ipfs is huge) but once we have node-ipfs this will be a drop-in replacement.

Another related idea we've bounced: storing computational results in ipfs.
Say you have blobs for js-compile and js-source-with-links; you can load
both and run js-compile(js-source-with-links), then store the result and
use that as the above entry point. It's basically how all browser side JS
already works, just that it's very sad how we store build results in dumb
files without linking to how they've been conceived (creation metadata! cc
@jbenet).

Absolutely! And-- i want to make a way to do browserify with partial results separated out. THOUGH, rabin fingerprinting will get us 80% of the way there without the worries anyway.

Side note: Ironically I'm in a place without wifi and just looked up
stackstream on my mac's browser history, but of course I can't access the
already cached page. Idea: any ipfs browser/browser extension should pin
history items. Immediate benefit over any other real world browser.

Yeahhh i really want this. wonder what the overhead + pains of doing this might be... hard to get right because people dont hash things or use cache headers correctly :/

Member

jbenet commented Jun 2, 2015

Update on the list above: @krl made https://github.com/ipfs/node-ipfsd-ctl


@ehd all of that sounds awesome!

Ideally we'd have a simple bootstrapping js/node program in ipfs that can
load other things from ipfs using require.

Yeah! like a special require that uses ipfs directly. Could patch npm to make requires that start with "/ipfs/..." require straight out of ipfs.

Btw, what @bengl and I want to do above with reginabox will work with stock npm and require.

Reminds me of the StackStream
opcodes actually ☺️ Anyway, without a file system the ideal
delivery mechanism would be the true ipfs network, so baking a simple ipfs
client into a runnable JavaScript bundle sounds like an idea worth trying.

that sounds awesome

Initially this could be done by just using the global or local ipfs http
gateway for loading code. I'm not sure whether a dynamic module loader
(require on demand) or a dynamic bundler (compile program, then start
program) would be more viable. Whatever runs in <script src="browser-fetchable-address-on-ipfs"/> or vanilla node I guess.

@krl's https://github.com/ipfs/node-ipfsd-ctl can make ephemeral ipfs nodes using the go-ipfs package, so you dont even have to install ipfs beforehand. it's huge (cause go-ipfs is huge) but once we have node-ipfs this will be a drop-in replacement.

Another related idea we've bounced: storing computational results in ipfs.
Say you have blobs for js-compile and js-source-with-links; you can load
both and run js-compile(js-source-with-links), then store the result and
use that as the above entry point. It's basically how all browser side JS
already works, just that it's very sad how we store build results in dumb
files without linking to how they've been conceived (creation metadata! cc
@jbenet).

Absolutely! And-- i want to make a way to do browserify with partial results separated out. THOUGH, rabin fingerprinting will get us 80% of the way there without the worries anyway.

Side note: Ironically I'm in a place without wifi and just looked up
stackstream on my mac's browser history, but of course I can't access the
already cached page. Idea: any ipfs browser/browser extension should pin
history items. Immediate benefit over any other real world browser.

Yeahhh i really want this. wonder what the overhead + pains of doing this might be... hard to get right because people dont hash things or use cache headers correctly :/

@bengl

This comment has been minimized.

Show comment
Hide comment
@bengl

bengl Jun 2, 2015

Could patch npm to make requires that start with "/ipfs/..." require straight out of ipfs.

You'd have to patch node. All the require logic is self-contained in node, not npm.

There are also many lifecycle scripts and compilation steps and all kinds of other things that are done to modules during the install process. In some modules, the results of these scripts are different depending on where they are in the node_modules tree.

I think it's much more practical to keep a strict separation between the source of packages (ipfs/ipns, in an ideal world) and the current install directory. Requiring them directly might work for JS-only modules with no lifecycle scripts. But then you're not supporting a huge chunk of available modules.

I'm only really talking about node here, but for people using frontend build pipelines that depend on npm, the same problems can arise.

So yeah, I'm not sure grabbing node dependencies directly from ipfs at run-time is realistic.

bengl commented Jun 2, 2015

Could patch npm to make requires that start with "/ipfs/..." require straight out of ipfs.

You'd have to patch node. All the require logic is self-contained in node, not npm.

There are also many lifecycle scripts and compilation steps and all kinds of other things that are done to modules during the install process. In some modules, the results of these scripts are different depending on where they are in the node_modules tree.

I think it's much more practical to keep a strict separation between the source of packages (ipfs/ipns, in an ideal world) and the current install directory. Requiring them directly might work for JS-only modules with no lifecycle scripts. But then you're not supporting a huge chunk of available modules.

I'm only really talking about node here, but for people using frontend build pipelines that depend on npm, the same problems can arise.

So yeah, I'm not sure grabbing node dependencies directly from ipfs at run-time is realistic.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Jun 2, 2015

Member

@bengl ah thanks! makes a lot of sense.

i guess we could do it to specific packages people decide to do this with (i.e. people explicitly publish them + require them with hashes, so they take on the responsibility of not making all the lifecycle stuff happen?) not sure.

Member

jbenet commented Jun 2, 2015

@bengl ah thanks! makes a lot of sense.

i guess we could do it to specific packages people decide to do this with (i.e. people explicitly publish them + require them with hashes, so they take on the responsibility of not making all the lifecycle stuff happen?) not sure.

@bengl

This comment has been minimized.

Show comment
Hide comment
@bengl

bengl Jun 2, 2015

Yeah for sure. ipfs-pm/ipfspm/ippm. You'd pretty much have to make a clear separation from npm at that point, I'd think.

I still would like to see the ipfs leveraged as the package source, as we discussed before. Should be easy once ipfs-blob-store exists :)

bengl commented Jun 2, 2015

Yeah for sure. ipfs-pm/ipfspm/ippm. You'd pretty much have to make a clear separation from npm at that point, I'd think.

I still would like to see the ipfs leveraged as the package source, as we discussed before. Should be easy once ipfs-blob-store exists :)

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Jun 2, 2015

Member

@bengl maybe we can hack that (ipfs-blob-store) out this week. I'm in bay area atm. (cc @krl @travisperson @mappum @diasdavid in case they're inspired)

Member

jbenet commented Jun 2, 2015

@bengl maybe we can hack that (ipfs-blob-store) out this week. I'm in bay area atm. (cc @krl @travisperson @mappum @diasdavid in case they're inspired)

@travisperson

This comment has been minimized.

Show comment
Hide comment
@travisperson

travisperson Jun 2, 2015

Member

Definitely, I took a look at the spec today. @bengl do you have any of the work you've made towards it published anywhere?

Member

travisperson commented Jun 2, 2015

Definitely, I took a look at the spec today. @bengl do you have any of the work you've made towards it published anywhere?

@bengl

This comment has been minimized.

Show comment
Hide comment
@bengl

bengl Jun 2, 2015

So really we're looking at potentially two different use-cases for ipfs-blob-store. Might be better off as two different modules. I'm not sure.

Use Case 1: Raw Blob Store

In this case, we're really just plopping blobs onto IPFS and getting their IPFS paths back in the metadata for later retrieval. So in createWriteStream, you wouldn't provide a key in the opts as that's what you'd be getting back from IPFS (i.e. the IPFS path). In createReadStream, the key you'd pass in would be the IPFS path, and it would stream back the blob at that path.

Use Case 2: IPNS Mutable File System Thing

In this case (which could use the previous case under the hood), in the constructor for the blob store, we'd provide all the necessary data to read and write to an IPNS namespace (keypair? just the hash? Sorry, my IPNS knowledge is rather limited). Here, createWriteStream would use the key option from the args to determine where to put the blob, and the metadata returned would be kind of irrelevant, but would have the same key. Behind the scenes it would update IPNS properly. So the key would be something like /path/in/namepsace. Similarly for createReadStream.


Unfortunately I don't spend as much time on this as I'd like, so I'm unfamiliar with how things actually work in IPNS-land right now. Sorry!

Here's the abstract-blob-store API reference.

bengl commented Jun 2, 2015

So really we're looking at potentially two different use-cases for ipfs-blob-store. Might be better off as two different modules. I'm not sure.

Use Case 1: Raw Blob Store

In this case, we're really just plopping blobs onto IPFS and getting their IPFS paths back in the metadata for later retrieval. So in createWriteStream, you wouldn't provide a key in the opts as that's what you'd be getting back from IPFS (i.e. the IPFS path). In createReadStream, the key you'd pass in would be the IPFS path, and it would stream back the blob at that path.

Use Case 2: IPNS Mutable File System Thing

In this case (which could use the previous case under the hood), in the constructor for the blob store, we'd provide all the necessary data to read and write to an IPNS namespace (keypair? just the hash? Sorry, my IPNS knowledge is rather limited). Here, createWriteStream would use the key option from the args to determine where to put the blob, and the metadata returned would be kind of irrelevant, but would have the same key. Behind the scenes it would update IPNS properly. So the key would be something like /path/in/namepsace. Similarly for createReadStream.


Unfortunately I don't spend as much time on this as I'd like, so I'm unfamiliar with how things actually work in IPNS-land right now. Sorry!

Here's the abstract-blob-store API reference.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Jun 2, 2015

Member

@bengl yep, you've got it right :)

We'd want a third module: unixfs-blob-store which would read an ipfs object as unixfs and stream out the full file (or compute StackStream)

Member

jbenet commented Jun 2, 2015

@bengl yep, you've got it right :)

We'd want a third module: unixfs-blob-store which would read an ipfs object as unixfs and stream out the full file (or compute StackStream)

@seidtgeist

This comment has been minimized.

Show comment
Hide comment
@seidtgeist

seidtgeist Jun 2, 2015

Pardon my late replies, I’m on a vacation until Thursday.

You’d have to patch node. All the require logic is self-contained in node, not npm.

That’s true. But you can change the meaning of require through compilation, e.g. like webpack does. The JavaScript bootloader could do this when loading modules from ipfs.

But then you’re not supporting a huge chunk of available modules.

That’s unfortunately true, but I think we can do lots of useful stuff without modules that rely on native compilation. Most compilation I see when running npm install is (dev)dependencies installing native file watchers. If native dependencies are a must, there could be pre-compiled binaries per runtime that could be grabbed from ipfs. Most backend or hybrid JS code I’ve written runs in both node and browsers and doesn’t depend on native dependencies. I could do lots of interesting stuff without them. Maybe @grncdr can also comment on that.

seidtgeist commented Jun 2, 2015

Pardon my late replies, I’m on a vacation until Thursday.

You’d have to patch node. All the require logic is self-contained in node, not npm.

That’s true. But you can change the meaning of require through compilation, e.g. like webpack does. The JavaScript bootloader could do this when loading modules from ipfs.

But then you’re not supporting a huge chunk of available modules.

That’s unfortunately true, but I think we can do lots of useful stuff without modules that rely on native compilation. Most compilation I see when running npm install is (dev)dependencies installing native file watchers. If native dependencies are a must, there could be pre-compiled binaries per runtime that could be grabbed from ipfs. Most backend or hybrid JS code I’ve written runs in both node and browsers and doesn’t depend on native dependencies. I could do lots of interesting stuff without them. Maybe @grncdr can also comment on that.

@travisperson

This comment has been minimized.

Show comment
Hide comment
@travisperson

travisperson Jun 3, 2015

Member

First stab at ipfs-blob-store

https://github.com/ipfs/ipfs-blob-store

Currently uses raw blocks, and is only content addressable. It does not use ipns at the moment, passes 45 of the tests (out of a bunch more, I don't fully understand tape).

Member

travisperson commented Jun 3, 2015

First stab at ipfs-blob-store

https://github.com/ipfs/ipfs-blob-store

Currently uses raw blocks, and is only content addressable. It does not use ipns at the moment, passes 45 of the tests (out of a bunch more, I don't fully understand tape).

@jbenet jbenet referenced this issue Jun 30, 2015

Open

Cool demos #18

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Sep 15, 2015

Member

i think it's time to resurrect this effort.

cc @mappum @daviddias @RichardLitt

Member

jbenet commented Sep 15, 2015

i think it's time to resurrect this effort.

cc @mappum @daviddias @RichardLitt

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Sep 15, 2015

Member

npm uses tar right? ipfs tar is so ready.

Member

whyrusleeping commented Sep 15, 2015

npm uses tar right? ipfs tar is so ready.

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Sep 15, 2015

Member

👍

Member

diasdavid commented Sep 15, 2015

👍

@bengl

This comment has been minimized.

Show comment
Hide comment
@bengl

bengl Sep 15, 2015

👍 💥

bengl commented Sep 15, 2015

👍 💥

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Sep 17, 2015

Member

Looking for some clarifications:

this is so awesome!

Member

diasdavid commented Sep 17, 2015

Looking for some clarifications:

this is so awesome!

@bengl

This comment has been minimized.

Show comment
Hide comment
@bengl

bengl Sep 17, 2015

Just registry-static. I maintain it as part of my day job, so LMK if you need anything there. reginabox is just a fancily-pre-packaged version of registry-static.

  • Is registry-static ready to handle publishes? (not fully getting it from docs)

No. registry-static is not actually equivalent to an npm registry. It's only useful for mirroring from the public registry (or any other registry that supports couchdb change feeds). For a full-on registry replacement for internal or alternative usage, you need to either run your own couchdb, or something like cnpm.

  • Is there a way to proactively cache the whole npm built into registry-static?

Yes, that's the first thing registry-static does when you start it up (but it takes like 2 days).

bengl commented Sep 17, 2015

Just registry-static. I maintain it as part of my day job, so LMK if you need anything there. reginabox is just a fancily-pre-packaged version of registry-static.

  • Is registry-static ready to handle publishes? (not fully getting it from docs)

No. registry-static is not actually equivalent to an npm registry. It's only useful for mirroring from the public registry (or any other registry that supports couchdb change feeds). For a full-on registry replacement for internal or alternative usage, you need to either run your own couchdb, or something like cnpm.

  • Is there a way to proactively cache the whole npm built into registry-static?

Yes, that's the first thing registry-static does when you start it up (but it takes like 2 days).

@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Sep 18, 2015

Hi. Trying to make sure I grok the big picture here, in terms of what an ipfs-npm request looks like. Please correct me as I go. ;)

  1. User installs the ipfs-npm command, or whatever we call it
  2. User runs ipfs-npm foobar
  3. I'm not clear on how look-up happens, to map a module name to an IPFS address. One option is to use IPNS, and claim that "all ipfs-npm modules are published to this pubkey. This means all clients know that to get foobar the path will always be /ipns/<static pubkey we control>/foobar. Do we want that kind of control? It would mean centralizing the publish operation.
  4. Easy part: we grab the module tarball from IPFS and install.

How does ipfs-blob-store fit in? Is this what reginabox uses to retrieve the blobs?

noffle commented Sep 18, 2015

Hi. Trying to make sure I grok the big picture here, in terms of what an ipfs-npm request looks like. Please correct me as I go. ;)

  1. User installs the ipfs-npm command, or whatever we call it
  2. User runs ipfs-npm foobar
  3. I'm not clear on how look-up happens, to map a module name to an IPFS address. One option is to use IPNS, and claim that "all ipfs-npm modules are published to this pubkey. This means all clients know that to get foobar the path will always be /ipns/<static pubkey we control>/foobar. Do we want that kind of control? It would mean centralizing the publish operation.
  4. Easy part: we grab the module tarball from IPFS and install.

How does ipfs-blob-store fit in? Is this what reginabox uses to retrieve the blobs?

@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Sep 18, 2015

@whyrusleeping helped me better understand the intent here. Looks like I was getting ahead of myself; it sounds like the first goal is to opt for decentralized before distributed: let users run their own npm registry that uses IPFS as the blob store rather than the local fs.

Lingering Q: how will our reginabox map module names to IPFS addresses?

noffle commented Sep 18, 2015

@whyrusleeping helped me better understand the intent here. Looks like I was getting ahead of myself; it sounds like the first goal is to opt for decentralized before distributed: let users run their own npm registry that uses IPFS as the blob store rather than the local fs.

Lingering Q: how will our reginabox map module names to IPFS addresses?

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping
Member

whyrusleeping commented Sep 18, 2015

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Oct 13, 2015

Member

Just PR'ed ipfs-blob-store using mfs (mutable file system) ipfs-shipyard/ipfs-blob-store#5

@bengl I'm being unable to attach a custom blob-store to registry-static, looking at the --help menu, the --blobstore option doesn't appear (for ref posted the help menu below) and when I run something like » registry-static -d npm.ipfs.io -o /npm-ipfs --blobstore=registry-ipfs-blob-store.js, I get this error:

module.js:338
    throw err;
    ^

Error: Cannot find module 'registry-ipfs-blob-store.js'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:286:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/usr/local/lib/node_modules/registry-static/lib/args.js:108:25)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)

Although I have the module exporting the blob store:

$ ls -al                                                                                                                                                                                                                                                                         
total 464
# ...
-rw-r--r--   1 david  staff      84 Oct 12 19:36 registry-ipfs-blob-store.js
# ...
$ cat registry-ipfs-blob-store.js
var ipfsBlobStore = require('ipfs-blob-store')
module.exports = ipfsBlobStore().mfs

Could you give me a hint on how to do it? Thank you :)

note: mfs is only available on IPFS dev0.4.0

$ registry-static --help
registry-static@2.0.0

Examples:
  registry-static -d my.registry.com -o /var/www/registry
  registry-static -c /path/to/myconfig.json


Options:
  --help, -h    Show help
  --config      Use a config file
  --domain, -d  The domain replacer                                 [required]
  --dir, -o     The output directory                                [required]
  --registry    The registry to mirror from                                     [default: "http://registry.npmjs.org/"]
  --limit, -l   Limit the number of concurrent downloads                        [default: 10]
  --user, -u    Set the user that this process should change to
  --group, -g   Set the group that this process should change to
  --hooks       Provide a hooks module.
  --clean       Clear the sequance file
  --check       Crawl the tarballs and check all of their shasums
  --sync        Crawl the json files and look for outdated index files
  --report      Used with --check, write a json report for all missing files
  --log         The file to log the output to
  --quiet       Turn down the logger
  --index       An index.json for the whole registry                            [default: "/usr/local/lib/node_modules/registry-static/defaults/index.json"]
  --error       A 404 file                                                      [default: "/usr/local/lib/node_modules/registry-static/defaults/404.json"]
  --replport    Port to listen on for REPL
  --one         Sync only the provided package, right now, then quit
  --spawn, -s                                                                   [default: 20]
  --tmp, -t                                                                     [default: "/var/folders/_y/h419_yvx6tb6wjkgjr6gp0p40000gn/T"]
Member

diasdavid commented Oct 13, 2015

Just PR'ed ipfs-blob-store using mfs (mutable file system) ipfs-shipyard/ipfs-blob-store#5

@bengl I'm being unable to attach a custom blob-store to registry-static, looking at the --help menu, the --blobstore option doesn't appear (for ref posted the help menu below) and when I run something like » registry-static -d npm.ipfs.io -o /npm-ipfs --blobstore=registry-ipfs-blob-store.js, I get this error:

module.js:338
    throw err;
    ^

Error: Cannot find module 'registry-ipfs-blob-store.js'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:286:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/usr/local/lib/node_modules/registry-static/lib/args.js:108:25)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)

Although I have the module exporting the blob store:

$ ls -al                                                                                                                                                                                                                                                                         
total 464
# ...
-rw-r--r--   1 david  staff      84 Oct 12 19:36 registry-ipfs-blob-store.js
# ...
$ cat registry-ipfs-blob-store.js
var ipfsBlobStore = require('ipfs-blob-store')
module.exports = ipfsBlobStore().mfs

Could you give me a hint on how to do it? Thank you :)

note: mfs is only available on IPFS dev0.4.0

$ registry-static --help
registry-static@2.0.0

Examples:
  registry-static -d my.registry.com -o /var/www/registry
  registry-static -c /path/to/myconfig.json


Options:
  --help, -h    Show help
  --config      Use a config file
  --domain, -d  The domain replacer                                 [required]
  --dir, -o     The output directory                                [required]
  --registry    The registry to mirror from                                     [default: "http://registry.npmjs.org/"]
  --limit, -l   Limit the number of concurrent downloads                        [default: 10]
  --user, -u    Set the user that this process should change to
  --group, -g   Set the group that this process should change to
  --hooks       Provide a hooks module.
  --clean       Clear the sequance file
  --check       Crawl the tarballs and check all of their shasums
  --sync        Crawl the json files and look for outdated index files
  --report      Used with --check, write a json report for all missing files
  --log         The file to log the output to
  --quiet       Turn down the logger
  --index       An index.json for the whole registry                            [default: "/usr/local/lib/node_modules/registry-static/defaults/index.json"]
  --error       A 404 file                                                      [default: "/usr/local/lib/node_modules/registry-static/defaults/404.json"]
  --replport    Port to listen on for REPL
  --one         Sync only the provided package, right now, then quit
  --spawn, -s                                                                   [default: 20]
  --tmp, -t                                                                     [default: "/var/folders/_y/h419_yvx6tb6wjkgjr6gp0p40000gn/T"]
@seidtgeist

This comment has been minimized.

Show comment
Hide comment
@seidtgeist

seidtgeist Nov 3, 2015

Had an idea, been wondering whether that's being tried or pursued:

I'd like to have a PEER that offers a FILE of PACKAGE at VERSION as such:

/ipns/PEER/npm/PACKAGE/VERSION/path/to/FILE

Would something like this make sense? In a way, it would be useful to have a good way to access npm packages canonically, and have a trusted authority for feelgood-immutable npm@version names to ipfs objects.

seidtgeist commented Nov 3, 2015

Had an idea, been wondering whether that's being tried or pursued:

I'd like to have a PEER that offers a FILE of PACKAGE at VERSION as such:

/ipns/PEER/npm/PACKAGE/VERSION/path/to/FILE

Would something like this make sense? In a way, it would be useful to have a good way to access npm packages canonically, and have a trusted authority for feelgood-immutable npm@version names to ipfs objects.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 4, 2015

Member

Agreed. In time this could be a key npm itself controls.

I wouldn't count too much on specific IPNS names yet until we have easy ways of delegating record signing through auth agents, so we can keep important keys more secure and isolated from the Internet.


Sent from Mailbox

On Tue, Nov 3, 2015 at 9:55 AM, Stephan Seidt notifications@github.com
wrote:

Had an idea, been wondering whether that's being tried or pursued:
I'd like to have a PEER that offers a FILE of PACKAGE at VERSION as such:

/ipns/PEER/npm/PACKAGE/VERSION/path/to/FILE

Would something like this make sense? In a way, it would be useful to have a good way to access npm packages canonically, and have a trusted authority for feelgood-immutable npm@version names to ipfs objects.

Reply to this email directly or view it on GitHub:
#2 (comment)

Member

jbenet commented Nov 4, 2015

Agreed. In time this could be a key npm itself controls.

I wouldn't count too much on specific IPNS names yet until we have easy ways of delegating record signing through auth agents, so we can keep important keys more secure and isolated from the Internet.


Sent from Mailbox

On Tue, Nov 3, 2015 at 9:55 AM, Stephan Seidt notifications@github.com
wrote:

Had an idea, been wondering whether that's being tried or pursued:
I'd like to have a PEER that offers a FILE of PACKAGE at VERSION as such:

/ipns/PEER/npm/PACKAGE/VERSION/path/to/FILE

Would something like this make sense? In a way, it would be useful to have a good way to access npm packages canonically, and have a trusted authority for feelgood-immutable npm@version names to ipfs objects.

Reply to this email directly or view it on GitHub:
#2 (comment)

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 4, 2015

Member

(The root ipns name would point to another ipns key, the current signing key from a rotation, etc. that way can revoke it by moving ipns -> ipns ptr at a given time, bounded only by record lifetimes)


Sent from Mailbox

On Tue, Nov 3, 2015 at 9:55 AM, Stephan Seidt notifications@github.com
wrote:

Had an idea, been wondering whether that's being tried or pursued:
I'd like to have a PEER that offers a FILE of PACKAGE at VERSION as such:

/ipns/PEER/npm/PACKAGE/VERSION/path/to/FILE

Would something like this make sense? In a way, it would be useful to have a good way to access npm packages canonically, and have a trusted authority for feelgood-immutable npm@version names to ipfs objects.

Reply to this email directly or view it on GitHub:
#2 (comment)

Member

jbenet commented Nov 4, 2015

(The root ipns name would point to another ipns key, the current signing key from a rotation, etc. that way can revoke it by moving ipns -> ipns ptr at a given time, bounded only by record lifetimes)


Sent from Mailbox

On Tue, Nov 3, 2015 at 9:55 AM, Stephan Seidt notifications@github.com
wrote:

Had an idea, been wondering whether that's being tried or pursued:
I'd like to have a PEER that offers a FILE of PACKAGE at VERSION as such:

/ipns/PEER/npm/PACKAGE/VERSION/path/to/FILE

Would something like this make sense? In a way, it would be useful to have a good way to access npm packages canonically, and have a trusted authority for feelgood-immutable npm@version names to ipfs objects.

Reply to this email directly or view it on GitHub:
#2 (comment)

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Nov 4, 2015

Member

@ehd that is similar to what we are doing, where we have one PEER that has the entire npm by a similar folder structure (the one used in registry-static) and then the others, by copying that root path /npm have access to the folder structure and respective hashes, which then can be server by 1 or more nodes.

Member

diasdavid commented Nov 4, 2015

@ehd that is similar to what we are doing, where we have one PEER that has the entire npm by a similar folder structure (the one used in registry-static) and then the others, by copying that root path /npm have access to the folder structure and respective hashes, which then can be server by 1 or more nodes.

@seidtgeist

This comment has been minimized.

Show comment
Hide comment
@seidtgeist

seidtgeist Nov 4, 2015

I wouldn't count too much on specific IPNS names yet until we have easy ways of delegating record signing through auth agents, so we can keep important keys more secure and isolated from the Internet.

(The root ipns name would point to another ipns key, the current signing key from a rotation, etc. that way can revoke it by moving ipns -> ipns ptr at a given time, bounded only by record lifetimes)

Sounds great. Can't wait.

@ehd that is similar to what we are doing, where we have one PEER that has the entire npm by a similar folder structure (the one used in registry-static) and then the others, by copying that root path /npm have access to the folder structure and respective hashes, which then can be server by 1 or more nodes.

What do you mean by "the others"?

Here's an interesting project by @mjackson: http://npmcdn.com – It's a heroku app with a CDN in front, serving files out of npm packages.

Idea: Use trusted computation instead of trusted names for this problem (Distributed request network?)

f = hash of a javascript function (could be any lang with sufficient metadata or linking (unikernel!))
x = hash of an input

Someone else executes it for me, caches the result. I trust them to provide y = f x

If I paid them to do this for me, and they provided a binding legal contract: Very nice and trustworthy. You know who's got at providing a useful and stable environment for running code? CI services. There probably is a nicer blockchainy/ethereum smart contract way, but I'm not well versed here.

Something like npmcdn would be trivial with this: f is the lookup function, x is {name, version, pathToFile}, y is the computation result.

/ipcompute/funhash/inputhash -> result?

There are many things we could use that work like npmcdn

Benefit:

  • Anyone could do the computation
  • Code, inputs and outputs hosted on IPFS
  • Reproducible/verifiable
  • Fault-tolerant (have multiple nodes computing & caching)

Q:

  • Is this just something on top of ipns? pub-sub? req-res? notifications?

seidtgeist commented Nov 4, 2015

I wouldn't count too much on specific IPNS names yet until we have easy ways of delegating record signing through auth agents, so we can keep important keys more secure and isolated from the Internet.

(The root ipns name would point to another ipns key, the current signing key from a rotation, etc. that way can revoke it by moving ipns -> ipns ptr at a given time, bounded only by record lifetimes)

Sounds great. Can't wait.

@ehd that is similar to what we are doing, where we have one PEER that has the entire npm by a similar folder structure (the one used in registry-static) and then the others, by copying that root path /npm have access to the folder structure and respective hashes, which then can be server by 1 or more nodes.

What do you mean by "the others"?

Here's an interesting project by @mjackson: http://npmcdn.com – It's a heroku app with a CDN in front, serving files out of npm packages.

Idea: Use trusted computation instead of trusted names for this problem (Distributed request network?)

f = hash of a javascript function (could be any lang with sufficient metadata or linking (unikernel!))
x = hash of an input

Someone else executes it for me, caches the result. I trust them to provide y = f x

If I paid them to do this for me, and they provided a binding legal contract: Very nice and trustworthy. You know who's got at providing a useful and stable environment for running code? CI services. There probably is a nicer blockchainy/ethereum smart contract way, but I'm not well versed here.

Something like npmcdn would be trivial with this: f is the lookup function, x is {name, version, pathToFile}, y is the computation result.

/ipcompute/funhash/inputhash -> result?

There are many things we could use that work like npmcdn

Benefit:

  • Anyone could do the computation
  • Code, inputs and outputs hosted on IPFS
  • Reproducible/verifiable
  • Fault-tolerant (have multiple nodes computing & caching)

Q:

  • Is this just something on top of ipns? pub-sub? req-res? notifications?
@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Nov 5, 2015

Member

@ehd like Amazon lambda/WebTask for the network? That is a field that is very very exciting and something we keep going back, it is definitely a valid idea.

Now when it comes to npm on IPFS, what we want is to be able to install the modules from the network, which is far less ambitious that having a universal way to do computation in a P2P network (btw I did work towards that goal in my MSc thing that I got to share at DTN last May, we can talk more or open a issue about it :):)).

The main challenge was, without forcing users to adopt a new way to install their modules, how could we offer the same npm install module-name experience? Well, the answer to that was IPNS and mfs.

Quick update on state of things

  • got reginabox to use a blob-store driver for storage https://github.com/diasdavid/reginabox
  • read only mode (avoid forcing the download of all the modules, useful if ipfs-blob-store is being used and the modules are already in the network)
  • ipfs-blob-store
  • fat storage node ready to glob the entire npm
  • rebase ipfs 0.3.9 onto 0.4.0 so that the new js-ipfs-api works again (which is a dependency of ipfs-blob-store)
Member

diasdavid commented Nov 5, 2015

@ehd like Amazon lambda/WebTask for the network? That is a field that is very very exciting and something we keep going back, it is definitely a valid idea.

Now when it comes to npm on IPFS, what we want is to be able to install the modules from the network, which is far less ambitious that having a universal way to do computation in a P2P network (btw I did work towards that goal in my MSc thing that I got to share at DTN last May, we can talk more or open a issue about it :):)).

The main challenge was, without forcing users to adopt a new way to install their modules, how could we offer the same npm install module-name experience? Well, the answer to that was IPNS and mfs.

Quick update on state of things

  • got reginabox to use a blob-store driver for storage https://github.com/diasdavid/reginabox
  • read only mode (avoid forcing the download of all the modules, useful if ipfs-blob-store is being used and the modules are already in the network)
  • ipfs-blob-store
  • fat storage node ready to glob the entire npm
  • rebase ipfs 0.3.9 onto 0.4.0 so that the new js-ipfs-api works again (which is a dependency of ipfs-blob-store)
@seidtgeist

This comment has been minimized.

Show comment
Hide comment
@seidtgeist

seidtgeist Nov 5, 2015

@ehd like Amazon lambda/WebTask for the network? That is a field that is very very exciting and something we keep going back, it is definitely a valid idea.

Yeah, it's so cool! I'll try to visualize my idea on the weekend.

The main challenge was, without forcing users to adopt a new way to install their modules, how could we offer the same npm install module-name experience? Well, the answer to that was IPNS and mfs.

And that's awesome. It'll be really cool and useful to have the that for npm end users and also the idea above. It relies on some system serving all the archives and metadata, and using npm on ipfs instead of the registry brings those benefits with it.

seidtgeist commented Nov 5, 2015

@ehd like Amazon lambda/WebTask for the network? That is a field that is very very exciting and something we keep going back, it is definitely a valid idea.

Yeah, it's so cool! I'll try to visualize my idea on the weekend.

The main challenge was, without forcing users to adopt a new way to install their modules, how could we offer the same npm install module-name experience? Well, the answer to that was IPNS and mfs.

And that's awesome. It'll be really cool and useful to have the that for npm end users and also the idea above. It relies on some system serving all the archives and metadata, and using npm on ipfs instead of the registry brings those benefits with it.

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Nov 13, 2015

Member

The moment is near, you hear! :D

So, right now, using IPFS 0.4.0, with mfs, we can, using the new shiny js-ipfs-api and ipfs-blob-store, put the npm registry on IPFS and serve modules from there. To make this happen (and please replicate it so it works for everyone).

  1. git clone git@github.com:diasdavid/reginabox.git
  2. cd reginabox && npm i && npm link
  3. cd .. && mkdir custom-blob-store
  4. Create a index.js file with module.exports = require('ipfs-blob-store')().mfs && npm i ipfs-blob-store
  5. Install IPFS dev0.4.0 branch and ipfs daemon
  6. run reginabox mirror npm-registry <full-path-to-custom-blob-store>/index.js

This is to download the entire npm into an IPFS Node, once our fat Node has finished that, you will be able to do:

  1. everything till step 5
  2. run reginabox mirror npm-registry <full-path-to-custom-blob-store>/index.js true for fetch modules from the network instead of trying to pull everything down from npm first

@bengl, I wanted to make this setup more nice, without all of the installing steps, any suggestions? Would it be good to have all of this setting up be part of reginabox, so that devs only had to do npm i regianbox -g once and run the cli?

Member

diasdavid commented Nov 13, 2015

The moment is near, you hear! :D

So, right now, using IPFS 0.4.0, with mfs, we can, using the new shiny js-ipfs-api and ipfs-blob-store, put the npm registry on IPFS and serve modules from there. To make this happen (and please replicate it so it works for everyone).

  1. git clone git@github.com:diasdavid/reginabox.git
  2. cd reginabox && npm i && npm link
  3. cd .. && mkdir custom-blob-store
  4. Create a index.js file with module.exports = require('ipfs-blob-store')().mfs && npm i ipfs-blob-store
  5. Install IPFS dev0.4.0 branch and ipfs daemon
  6. run reginabox mirror npm-registry <full-path-to-custom-blob-store>/index.js

This is to download the entire npm into an IPFS Node, once our fat Node has finished that, you will be able to do:

  1. everything till step 5
  2. run reginabox mirror npm-registry <full-path-to-custom-blob-store>/index.js true for fetch modules from the network instead of trying to pull everything down from npm first

@bengl, I wanted to make this setup more nice, without all of the installing steps, any suggestions? Would it be good to have all of this setting up be part of reginabox, so that devs only had to do npm i regianbox -g once and run the cli?

@dignifiedquire

This comment has been minimized.

Show comment
Hide comment
@dignifiedquire

dignifiedquire Nov 13, 2015

Member

@diasdavid how about adding another module reginabox-cli which you can install globally and does all that work, without polluting the actual reginabox module?

Member

dignifiedquire commented Nov 13, 2015

@diasdavid how about adding another module reginabox-cli which you can install globally and does all that work, without polluting the actual reginabox module?

@dignifiedquire

This comment has been minimized.

Show comment
Hide comment
@dignifiedquire

dignifiedquire Nov 13, 2015

Member

So it would more like this:

$ npm i -g reginabox-cli
$ ipfs daemon 
$ reginabox mirror npm-registry
# internet connection lighting up because of all the traffic
Member

dignifiedquire commented Nov 13, 2015

So it would more like this:

$ npm i -g reginabox-cli
$ ipfs daemon 
$ reginabox mirror npm-registry
# internet connection lighting up because of all the traffic
@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Nov 13, 2015

Member

@dignifiedquire, regianbox is already a CLI :) so I guess we just need the changes on https://github.com/diasdavid/reginabox to go to https://github.com/yahoo/reginabox

Member

diasdavid commented Nov 13, 2015

@dignifiedquire, regianbox is already a CLI :) so I guess we just need the changes on https://github.com/diasdavid/reginabox to go to https://github.com/yahoo/reginabox

@dignifiedquire

This comment has been minimized.

Show comment
Hide comment
@dignifiedquire

dignifiedquire Nov 13, 2015

Member

hmmmmmmmm now that you say it like that cough ignore my comment, but it would still be nice to put all that manual work you mentioned in there or somewhere else to automate it more

Member

dignifiedquire commented Nov 13, 2015

hmmmmmmmm now that you say it like that cough ignore my comment, but it would still be nice to put all that manual work you mentioned in there or somewhere else to automate it more

@dignifiedquire

This comment has been minimized.

Show comment
Hide comment
@dignifiedquire

dignifiedquire Nov 13, 2015

Member

It has begun

/Volumes/npm/custom-blob-store
❯ ls npm-registry |wc -l                                                                     
     185
Member

dignifiedquire commented Nov 13, 2015

It has begun

/Volumes/npm/custom-blob-store
❯ ls npm-registry |wc -l                                                                     
     185
@travisperson

This comment has been minimized.

Show comment
Hide comment
@travisperson
Member

travisperson commented Nov 13, 2015

👍

@ion1

This comment has been minimized.

Show comment
Hide comment
@ion1

ion1 commented Nov 13, 2015

@dignifiedquire

It has begun

And so it begins

@bengl

This comment has been minimized.

Show comment
Hide comment
@bengl

bengl Nov 13, 2015

I'd suggest that a module that just implements the registry API by talking to IPFS would be sufficient, rather than having everyone use reginabox. reginabox is supposed to mean registry-in-a-box. Most users aren't going to need the whole registry mirrored, and thus have no need to run registry-static (the main component in reginabox, and probably the module this should have been based on in the first place).

That way users could do something like:

$ npm i -g ipfs-reg-mirror # or something like that

and then start that and use it as a registry in their .npmrc. I don't think we're likely to accept patches to reginabox that disable the actual mirroring when run with the mirror command, because that's its primary purpose.

reginabox also carries with it a bunch of code for discovering peers on your local network, which users likely don't need since that's kind of the point of having IPFS.

So, in summary, I'd say: a small module implementing only the http server that serves from ipfs-blob-store is all that an average user needs. Not reginabox.

Also, the fat server that's doing the following of the public registry doesn't need reginabox either, only registry-static.

bengl commented Nov 13, 2015

I'd suggest that a module that just implements the registry API by talking to IPFS would be sufficient, rather than having everyone use reginabox. reginabox is supposed to mean registry-in-a-box. Most users aren't going to need the whole registry mirrored, and thus have no need to run registry-static (the main component in reginabox, and probably the module this should have been based on in the first place).

That way users could do something like:

$ npm i -g ipfs-reg-mirror # or something like that

and then start that and use it as a registry in their .npmrc. I don't think we're likely to accept patches to reginabox that disable the actual mirroring when run with the mirror command, because that's its primary purpose.

reginabox also carries with it a bunch of code for discovering peers on your local network, which users likely don't need since that's kind of the point of having IPFS.

So, in summary, I'd say: a small module implementing only the http server that serves from ipfs-blob-store is all that an average user needs. Not reginabox.

Also, the fat server that's doing the following of the public registry doesn't need reginabox either, only registry-static.

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid
Member

diasdavid commented Nov 14, 2015

@dignifiedquire

This comment has been minimized.

Show comment
Hide comment
@dignifiedquire

dignifiedquire Nov 14, 2015

Member

@diasdavid @bengl registry-static is pretty slow :( not really using the full connection or cpu available. I'm only at 10% after 20h even though my connection could have downloaded 800gb in this time according to my calcs

Member

dignifiedquire commented Nov 14, 2015

@diasdavid @bengl registry-static is pretty slow :( not really using the full connection or cpu available. I'm only at 10% after 20h even though my connection could have downloaded 800gb in this time according to my calcs

@bengl

This comment has been minimized.

Show comment
Hide comment
@bengl

bengl Nov 14, 2015

@dignifiedquire No excuses, but here's an explanation of what's going on: registry-static uses the couchdb replication feed from skimdb.npmjs.com, and is limited by that. On each new piece of replication feed data, it goes through and processes all the metadata and tarballs for one package and all its versions, and then only upon completion moves on to the next piece of data from the replication feed. There are lots of tiny little files, and this has impacts on HTTP performance.

That being said, PRs to have more efficient pipelining of HTTP requests in registry-static would be considered. 😄

bengl commented Nov 14, 2015

@dignifiedquire No excuses, but here's an explanation of what's going on: registry-static uses the couchdb replication feed from skimdb.npmjs.com, and is limited by that. On each new piece of replication feed data, it goes through and processes all the metadata and tarballs for one package and all its versions, and then only upon completion moves on to the next piece of data from the replication feed. There are lots of tiny little files, and this has impacts on HTTP performance.

That being said, PRs to have more efficient pipelining of HTTP requests in registry-static would be considered. 😄

@dignifiedquire

This comment has been minimized.

Show comment
Hide comment
@dignifiedquire

dignifiedquire Nov 15, 2015

Member

@bengl created a PR to follow-registry to improve performance :)

Member

dignifiedquire commented Nov 15, 2015

@bengl created a PR to follow-registry to improve performance :)

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Nov 15, 2015

Member

Published the first version https://github.com/diasdavid/registry-mirror Wanna give it a try? Now it is only registry-mirror --ipfs --clone` and good to go :)

Member

diasdavid commented Nov 15, 2015

Published the first version https://github.com/diasdavid/registry-mirror Wanna give it a try? Now it is only registry-mirror --ipfs --clone` and good to go :)

@balupton

This comment has been minimized.

Show comment
Hide comment
@balupton

balupton Feb 27, 2016

I think replicating the entire registry into ipfs is the wrong approach here, as the popularity of npm will make the replication into another system a constant catchup.

The use case really seems to speed up npm in places of poor internet connectivity to the npm registry — australia, conferences, islands, etc.

The usage can be opted into. Essentially one only needs to replicate the files into ipfs that they are using. Essentially one could alias "npm install" to be "install from ipfs if there, otherwise install from npm then publish to ipfs" — one would need a ipns space to alias package names and versions to directory hashes, however I imagine the registry also has the hashes so would be easy enough to verify (is this npm install alias publishing the same version to hash link as to what the npm registry indicates). An alias could also be creates for "npm publish" to publish to ipfs after publishing to the registry.

While not comprehensive would allow moderare opting in abilities.

balupton commented Feb 27, 2016

I think replicating the entire registry into ipfs is the wrong approach here, as the popularity of npm will make the replication into another system a constant catchup.

The use case really seems to speed up npm in places of poor internet connectivity to the npm registry — australia, conferences, islands, etc.

The usage can be opted into. Essentially one only needs to replicate the files into ipfs that they are using. Essentially one could alias "npm install" to be "install from ipfs if there, otherwise install from npm then publish to ipfs" — one would need a ipns space to alias package names and versions to directory hashes, however I imagine the registry also has the hashes so would be easy enough to verify (is this npm install alias publishing the same version to hash link as to what the npm registry indicates). An alias could also be creates for "npm publish" to publish to ipfs after publishing to the registry.

While not comprehensive would allow moderare opting in abilities.

@perguth

This comment has been minimized.

Show comment
Hide comment
@perguth

perguth commented Mar 24, 2016

@balupton

This comment has been minimized.

Show comment
Hide comment
@balupton

balupton Mar 24, 2016

Perhaps @rstacruz from http://github.com/rstacruz/pnpm may be interested in this?

pnpm's architecture works really well for my ideal here, as pnpm I believe downloads each package to user directory, then symlinks the package directories to the relevant node_modules directory.

Combining this with IPFS, this user node_modules directory could be added to IPFS. Making all ipfs-pnpm users seeders of the packages they have installed - the ideal.

This works for hosting files, but doesn't work for hosting meta data and publishing. How is that planned to be addressed? Specifically publishing.

balupton commented Mar 24, 2016

Perhaps @rstacruz from http://github.com/rstacruz/pnpm may be interested in this?

pnpm's architecture works really well for my ideal here, as pnpm I believe downloads each package to user directory, then symlinks the package directories to the relevant node_modules directory.

Combining this with IPFS, this user node_modules directory could be added to IPFS. Making all ipfs-pnpm users seeders of the packages they have installed - the ideal.

This works for hosting files, but doesn't work for hosting meta data and publishing. How is that planned to be addressed? Specifically publishing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment