Node Foundation as stewards of packages

[scriptjs]

The circumstances of nodejs/node#3959 indicate that more can and should be done to lessen the dependence on NPM, to take back control of manifest standards, and to provide a more open approach to package distribution that is in alignment with other open source languages and initiatives including PyPi.

Package management has a direct relationship on the user experience of node. NPM, being a commercial entity with its own agenda, is too tightly coupled with node. Module developers have little choice where to publish due to the organic nature of how both node and npm have grown. Currently, mirrors also pull their data from NPM. This is a vulnerability as NPM asserts itself for its own benefit and not that of the node ecosystem.

We have already weathered disasters with NPM in the past, where performance interrupted thousands of businesses world wide. Today, the volume of modules being distributed daily is many times this and the level of centralization of the body of open source work under a sole commercial entity is unhealthy for open source. The web was founded on the basis of decentralization and availability. This needs to be practiced for the scale of node.

The Node Foundation is a logical choice to set standards and to foster solutions that provide the ecosystem with more choice, and the assurance of unrestricted access and availability of modules developers are providing for distribution. It is also in the nature of open source to diversify.

The way I see this unfolding in future working is for a CDN to be used for the distribution of modules.
The current collection of ~250,000 modules can be placed in placed in buckets of the host, with the package.json data as the metadata for each file. We already have semver and there is already a good body of code so developers can create resolvers that work together with the API of the CDN host to fetch dependencies. I am suggesting this starts as a pilot project.

PyPI is hosted by donation of Rackspace bandwidth and CDN. We simply let the node community build upon this infrastructure so more choices will be available in how modules are discovered and consumed.

At the base, the modules are published and hosted on infrastructure donated to the Node Foundation. NPM in future could use the same base of modules for its own services as anyone would have free and open access. The Node Foundation would primarily be involved in maintaining standards including those for the module manifest.

As a first step I am proposing that the Node Foundation seek donations for a CDN to host and distribute packages. PyPI is driven by Rackspace that has donated its bandwidth and CDN space. We have several healthy potential donors including IBM, Microsoft etc. From this first step, developers can create tools, search and sites that make use of the APIs of the CDN resource.

This would create a healthier environment for open source and eliminate the risks inherent in being controlled by sole commercial entity.

[mikeal]

A few followup comments.

• In terms of scale there are orders of magnitude differences between PyPI and npm's bandwidth needs. Last month there were 2.2 Billion package downloads from more than 2 million unique IP addresses.
• Package downloads are actually growing at an even faster rate than all other Node.js growth rates. While the user base grows at 100% a year the average dependency tree also grows which means that every month the average single package install ends up being even more individual package downloads. In aggregate this means that we have compounding rates of growth that all coalesce into outrageous download statistics.
• We already have a donated CDN and a lot of other donated computer resources managed by the Build WG. Cloudflare's CDN fronts all Node.js tarball downloads which I think are something like 2 million a month.
• The current registry is not open source so a new one supporting an identical API would have to be built for any of this to be possible.
• The npm client is not in the foundation so we would either have to convince npm to change the default, fork the client, or float a patch in the one we bundle.

The amount of work, support, and liability this involves should not be underestimated.

[scriptjs]

@mikeal I understand the magnitude of downloads. That said, the volume of packages is something very different. Approximately 20% of the registry being responsible for about 80% of the downloads. The total number of packages is still a relatively small number, not yet 250 K

Decentralizing the package registry is a choice for stability for the future. Currently what we have is unhealthy and having this continue unabated will only end in tears.

I am suggesting that the Node Foundation begin a pilot project by offering a CDN to allow the community to respond to this by building software and resources that can use it. Initially this can start by populating the CDN with modules from NPM. The CDN would be used as a resource to incubate, evaluate, and grow options for the future. It is never a great thing to have all your eggs in one basket.

Allow the community develop viable options using this resource, and there are options for the future. Start on a pilot scale and see where this goes.

On the APIs, in fact they do not in need to be identical to NPM. Projects like duo, jspm and others have different resolving mechanisms that can work with a variety of endpoints. Its not important that they be exactly the same as npm but to challenge the status quo. That is how we innovate. With support from the Node Foundation, the CDN resource can be used by developers and those testing the solutions and we can see what comes from this.

Other possibilities in the pipe include peer to peer solutions. This also makes much better sense for the scale of what we have here. Dat is getting close to a 1.0 and we can have thousands of systems as peers as opposed to a central registry in the future that is safer and distributed.

These are the answers for the future not a commercial entity that controls the ecosystem.

As you know, once modules are resolved and fetched, it is node's module API that resolves the software from node_modules, so the task is primarily a task of retrieving semantically versioned assets and recursively fetching package dependencies until all the packages arrive.

Btw, this goes for search capability as well, let people work against the CDN and come up with solutions. They do not need to be NPM, they can be what they imagine and let the community determine respond naturally to what they like.

[mikeal]

So, I don't have an opinion on this, I'm just trying to make sure the size and scope of this work is clear to everyone. As you'll see in my answers it's just very difficult to break off a part of this problem without taking on the whole problem.

Approximately 20% of the registry being responsible for about 80% of the downloads.

Sure, most systems like this follow an 80/20 rule, but I'm not quite sure why that matters it this context (although this is certainly relevant when considering a distributed alternative).

The total number of packages is still a relatively small number, not yet 250 K

Sure, but:

• This is still the largest open source package ecosystem in the world.
• The growth rate of total packages is still doubling every year.
• The storage requirements is much greater than 250K because each of those packages have N number of versions.

I am suggesting that the Node Foundation begin a pilot project by offering a CDN to allow the community to respond to this by building software and resources that work with it. Initially this can start by populating the CDN with modules from NPM. The CDN would be used as a resource to incubate, evaluate, and grow options for the future. It is never a great thing to have all your eggs in one basket.

We'd also need the publishing side of the registry which isn't just a CDN issue.

On the APIs, in fact they do not in need to be identical to NPM. Projects like duo, jspm and others have different resolving mechanisms that can work with a variety of endpoints.

Unless we fork or replace the client we need to have an identical HTTP API on the registry.

Other possibilities in the pipe include peer to peer solutions. This also makes much better sense for the scale of what we have here. Dat is getting close to a 1.0 and we can have thousands of systems as peers as opposed to a central registry in the future that is safer and distributed.

A Dat based npm replacement, or a distributed replacement, or any other replacement for that matter, is going to need to prove itself out a bit in the developer ecosystem before core would feel comfortable adopting it.

[scriptjs]

@mikeal publishing is a CDN issue. CDNs include APIs for putting content. Again, this is not about an immediate replacement, it is about incubating choices atm.

As I have said, we don't need a client that matches NPM. There are already clients that can resolve and fetch that do not have the same internals as NPM. They don't need to. The HTTP GET for the packages or metadata is not a difficult thing.

On your last comment, absolutely. That is the purpose of the Foundation providing the CDN. It is to foster solutions so there are choices in the future. There is a whole world of developers and a growing ecosystem, many of whom will be interested in these issues. Provide the resource for incubation and piloting and let the community determine solutions that look like winners. Leave the choice of what to do about NPM until there are viable alternatives – but grow them.

[isaacs]

I'd just like to point out that the npm public registry already uses a cdn (fastly) and that the outages mentioned were all from before the formation of npm, Inc. (That is, when the core team DID manage the registry, using donated infrastructure.)

Running a registry and writing a package manager are definitely not trivial tasks that are cheap or easy to do. Anyone who says otherwise hasn't done it. It is a worthwhile and educational activity that I encourage everyone to try at some point in their lives. You'll find it requires dedicated professionals to serve a community at this scale.

[scriptjs]

@issacs The title of the issue speaks for itself. Defending npm's Inc's position is not what this is about. Its about a broader dialog of what might be in the greater good of the community without this self interest. There are already efforts underway to work on alternatives and this is a welcome move.

Secondly, I am talking about decentralized approaches as well, this means no silos that makes the whole issue of scaling moot. I don't think anyone thought we'd get to this place. npm's role in general is feeling increasingly out of place and intrusive.

[jasnell]

Closing since there hasn't been any further discussion in over a year