Decentralized Module Resolving w/ proof of concept

[formula1]

Last conversation here - #26
I believe I was part of the problem and I'd like to get the thread started on the right foot again.

I'm basing most of this off the TC39 process which has been an effective means of getting features included into the w3c standards. Link here -https://tc39.github.io/process-document/

Purpose

• Problem - modules are currently centralized to one location. With this has brought up problems such as
  • Naming taken by deprecated, abandoned, un-tested or purposefully empty repositories
  • Attempting to duplicate or reserve the data is an extreme task making the one server extremely important developing. I believe it was this year that one of the servers at npm went down causing fires and panic. I cried
• Potential challengers
  • Mutability and Source of Truth - If the repo changes but the code doesn't this should be reflected. If there is a mutability issue, it can be resolved.
  • Maintaing Existing APIs - cli commands and possibly ability to require it as a module
  • Enabling servers to get started quickly - Ensuring someone that wants to be involved can get into it quickly
  • Standardization and Adapting - Ensuring changes in the software ecosystem can also be reflected here while maintaining backwards compatability and solid standards for internal use (Semver, package.json, etc)
  • Trust - Authentication, signing of packages, handing rougue machines. Something that likely others can speak more about.
• Opportunities
  • Even a newbie can help node - Peer to Peer distribution allows for a newbie to give back to the community even if they aren't perfect coders (which few of us are)
  • People can see first hand of what node and the node community is capable of - Its one thing to 'hear' about all the awesome projects on hacker news but to see and be apart of one is a step further.
  • Individuals can create their own registries - Whether adding statistics, having a rigid universal testing suite or other, they don't have to depend on npm accepting their pull request. They can build their own.

Current Proposed Solution

This is a culmination of efforts from @joepie91 @ChALkeR @mbostock @scriptjs @formula1

For More Detailed Explaination

Overview

• Enabling a Client to have Multiple Registries
  • Registries can advertise Responsibility, Statistics or etc - May choose to host a package or not by their own standards, May choose to offer developers statistics about downloads and queries. Ideas can come into fruition and possibly getting clients excited about developing for it.
  • Easy(ier?) Duplication gives backups to a client - By ensuring the delivery mechanism is not the package resolving mechanism, disk space will not be a heavy requirement. This allows for backups to be readilly available.
  • Rogue Registries can be removed - Likelyhood is minimal that npm would do anything absurd but there is a cost to trust. But allowing it to be removed easily, there is no issue.
• Enabling a Client to have Multiple Distribution Mechanisms
  • Circumvents the need to upload to two locations - Client can receive direct from their wherever the author is writing to
  • Distributed amoungst Peers - Tends to be faster and also allows for individuals to give back to the community if they choose
  • Allow trusted download - They may choose to only download from one place they trust. This may be a mirror of another location
• Enabling any Client to get even deeper into node
  • Peer to Peer -Clients can contribute to the peer to peer network giving back to the community even.
  • Inspiritaional - By seeing this sort of mechinism, it may inspire individuals to look and think differently about node and javascript.
  • Operating their own registry - Client seeing how simple a registry can be, may choose to start their own at minimal cost.

Psuedo Code

Installation

• Client gives Semver to Registry and gets a Distribution Handle
• Client figures out how to use the Distribution handle then uses it to download the package
• Client Unpacks the download and moves it to the appropriate location

Publishing

• Client gives handle or a tar to Registry
• Registry figures out what it needs to do with the handle and Downloads it
• Registry Unpacks it, Validates it, Indexes it then Allows their distribution method(s) to handle it
  • This may involve notifying distributors, which may or may not choose to mirror

Enabling a Client to Have Multiple Registries

• Registries can advertise Responsibility, Statistics or etc - May choose to host a package or not by their own standards, May choose to offer developers statistics about downloads and queries. Ideas can come into fruition and possibly getting clients excited about developing for it.
• Easy(ier?) Duplication gives backups to a client - By ensuring the delivery mechanism is not the package resolving mechanism, disk space will not be a heavy requirement. This allows for backups to be readilly available.
• Rogue Registries can be added and removed - Likelyhood is minimal that npm would do anything absurd but there is a cost to trust.

Technologies To take inspiration from

• https://github.com/npm/npm-registry-client - Is the current standard as a client
• https://github.com/npm/npm-registry-couchapp - Was the current standard as a server
• https://github.com/alexanderGugel/ied - Speed, Flatness keeps it the npm syntax only does it better
• https://github.com/mbostock/crom - Handles multiple results errors, Implements custom resolver, implements git based downloads
• https://github.com/feross/webtorrent - Peer to peer sharing
• https://github.com/ipfs/ipfs - Attempting to maintain the exact same files across all nodes. This is very close but a registry like npm has the ability to take down a project if they choose to which I believe is worth enabling because an individuals package may contain harmful materials.

Demo

https://github.com/formula1/decentralized-package-resolving-example

git clone https://github.com/formula1/decentralized-package-resolving-example
npm install
npm test

Its unoptimized, much of it is synchronous. But hey, I'd like to believe I'm paying my time and effort forward so that if this thing gets through I will feel like I was part of the solution. Every journey begins with a step I suppose.

PInging those who seemed interested: @scriptjs, @mikeal, @Martii, @ChALkeR, @joshmanders, @jasnell, @ashleygwilliams, @Qard, @bnoordhuis

[jasnell]

@formula1 ... thank you for engaging with a concrete proposal. This is exactly the kind of thing I was hoping for. It might take me a couple days to dig into the details but I promise I'll take a look.

[scriptjs]

@formula1 I'll take a look. I have not have the time to write anything formal after #26 was closed yesterday (after this discussion went south a second time) but had made a commitment to @jasnell yesterday to do so. I have a prototype for resolving and fetching to endpoints at the moment with providers and resolvers using semver. I am studying ied's caching atm which is robust.

[ChALkeR]

@scriptjs @formula1

If you are proposing any new schemes, note that they should support moderation. Trust does not work like this, even if you trust someone, if malware infects his computer or his accounts somehow become compromised (ref: my post), that could be used to replace some packages with malware.

Copy-pasting myself from #26 (comment):

Note that moderation of the registry is currenly needed, because there could be harmful packages. Also there could be (theoretical) situations when the whole registry must be stopped for moderation (I could describe such a situation a bit later), and that should be achievable. I am not saying that this restriction must be absolute, though.

So, please either make sure that your proposals support moderation (not necessary by one party, but by a limited amount of parties all of which have adequate support timings) or show me (and everyone else) how that would not be an issue.

[Qard]

@othiym23 and I discussed a bit last night, the idea of a stripped down fetch + tar tool that wouldn't include any repository connection or dependency management. It'd just fetch a tar and unpack it, maybe running some very basic lifecycle scripts, like node-gyp, but probably without the configurability you have with npm. If you want more full-featured package management, you could just use the tool to fetch a more complete package manager.

https://twitter.com/stephenbelanger/status/691046744780599296

[jasnell]

Yep, I've been stewing in the same direction. It could be very minimal
without the more advanced layout and dependency algorithms, and it wouldn't
need much in the way of package.json smarts. Higher level package managers
can be hooked in somehow to provide the higher level functions. I like it.
On Jan 24, 2016 7:50 AM, "Stephen Belanger" notifications@github.com
wrote:

@othiym23 https://github.com/othiym23 and I discussed a bit last night,
the idea of a stripped down fetch + tar tool that wouldn't include any
repository connection or dependency management. It'd just fetch a tar and
unpack it, maybe running some very basic lifecycle scripts, like node-gyp,
but probably without the configurability you have with npm. If you want
more full-featured package management, you could just use the tool to fetch
a more complete package manager.

https://twitter.com/stephenbelanger/status/691046744780599296

—
Reply to this email directly or view it on GitHub
#29 (comment).

[scriptjs]

I am in favour of this approach as well.

[joepie91]

I've been contemplating a more decentralized implementation for NPM for the past few days. While I'd planned to keep thinking it over for a little while, it seems that now that the discussion is underway, I should probably share my ideas and how I believe they address some of the technical (and 'political') challenges that are posed by the aforementioned proposals.

There are still some 'gaps' in my proposal, given that I've not had enough time to think it over yet. Perhaps those can be filled in here.

The primary requirements

• Deterministic package resolution, primarily due to sub-dependencies. If I publish a package foobar on a registry, and it relies on package baz, then I must be able to trust that any one person installing foobar will get the same (or a compatible) copy of baz that I have been developing against.
• Tolerance for moving of the 'source location' - GitHub is not going to be around forever, and people do move around the primary location of their source code every once in a while. This makes hardcoding locations impractical.
• Bandwidth-efficiency. By making it expensive to run a registry, this will self-select for commercial organizations who might have incentives that do not align with the best interests of the registry. It should be viable for non-commercial registry servers to exist, for example.
• Resistance to 'corruption' - it should not be possible (or be very hard/expensive) for any one person or entity to 'subvert' the registry in some way that is not in the best interest of the registry from a technical perspective, for their own benefit - whether commercial, political, or otherwise.
• Private registries, for internal/proprietary modules. I'm not a fan of proprietary code at all, but realistically, this is going to be a requirement.

How NPM solves these problems right now

• Deterministic package resolution: One central registry that is 'authorative' where it concerns package names. NPM acts as the gatekeeper.
• Tolerance for moving of the 'source location': The repository URL can be changed and updated in package.json. This does not affect the name in the registry. Additionally, deprecation notices can be used to inform users of name changes.
• Bandwidth-efficiency: NPM does not solve this. They simply pay for the bandwidth used, and charge their (private) users.
• Resistance to 'corruption': NPM does not currently solve this.
• Private registries: NPM only solves this partially, by providing a (commercial) hosted private registry service, and the ability to configure the registry URL for other cases.

Where alternative solutions fail

• Change to a different registry provider with the existing software: There is a considerable cost burden to running a registry, which makes this a very narrow pool of possible operators, all of which have commercial incentives that may not necessarily align with the best interests of the ecosystem. It also does not fix the fundamental monopoly issue, no matter who is 'in charge'.
• Install from GitHub/BitBucket/etc.: Not viable. Software moves over time, and this makes it near impossible to specify subdependencies and be able to trust that they will remain working in the future. This also complicates matters when end users are on a restrictive connection (eg. behind the Great Firewall), and may not be able to access all of these sources.
• Decentralized repository: The usual 'lack of a trust path' issues apply. Additionally, implementing moderation is hard. This is likely to take considerably more time to design and implement reliably than can be afforded in this case.
• Ask multiple authorative servers: No deterministic package resolution.

My proposal

Core to my proposal is the splitting up of tasks:

• Read servers: Read-only registry servers that accept requests for the serving of existing packages, and sometimes also serving tarballs. This is similar to how NPM mirrors work now, but I will get into details about the differences later.
• Write server(s): One or more servers, controlled by a single entity, on a fully open-source software stack. These are the servers that actually authenticate users, accept package uploads, and decide what the registry looks like.
• Distribution nodes: Nodes that serve the tarballs. These can be arbitrary systems, including clients, which can do tarball delivery through a P2P distribution mechanism like BitTorrent + DHT.

The distribution of tarballs could use a mechanism like webseeds, pointing at one or more registry servers that serve tarballs (or even generate such a webseed list on the fly), to decrease the latency for downloading tarballs while still offloading part of the distribution load to other peers.

All metadata and tarballs are cryptographically signed by the 'write servers'. I want to emphasize that UX is very important here, and it cannot become any significantly harder for end users to use NPM as a result of architectural changes.

Furthermore, for the absolute worst-case scenario, it should be possible for individual end users to install from a Git repository using semantic versioning specifications, referring to the tags on the repository to find compatible versions.

Threat models

• Write server operator goes 'rogue': If a consensus is reached about this by read server operators (or the ecosystem at large), they can simply point their servers at a write server operated by a different entity.

  End users will transparently receive their metadata and tarballs from a new source, and could explicitly blacklist any sources if they so desire (since the source can be identified by the signature).

  This makes it very expensive for a write server operator to go rogue, and relatively cheap for the ecosystem to move away from an operator that has done so. It doesn't prevent this from occurring, it just changes the incentives so that it becomes unattractive for an operator to do so.

  If no consensus can be reached that the write server operator has gone rogue, but a subset of the users has issues with the moderation policy of the write server, they can fall back to using Git repositories with tags as absolute last measure. This has downsides (eg. no ability to move the sources), but is better than nothing.

• Write server blocks end users from downloading: This has no effect. Package distribution is not handled by the write server. This removes a very big issue with current NPM, where critical Node infrastructure can be arbitrarily denied to any given user.

• Write server blocks end users from uploading: See above 'goes rogue' section.

• Write server goes down: Metadata and packages remain being distributed by read servers and distribution nodes as normal.

• Read server operator goes rogue: The 'source' of the signed data would change. There are a number of possible solutions for resolving this:

  • Do not validate the source by default. This is probably a bad idea.
  • 'Pin' the source by default, and warn the user and ask for confirmation when it changes, explaining the risks.

  In all of these cases, a decision would have to be made between letting people configure their own read servers, or shipping with a set of rotated read servers. I have not yet weighed the pros and cons of this.

• Read server goes down: Fallback to another read server. As each server just operates as a more or less 'stateless' cache, this should not be a concern.

How my proposal addresses the requirements

• Deterministic package resolution: There is still an authorative registry, but that authorative registry can be more easily changed if absolutely necessary.
• Tolerance for moving of the 'source location': This remains unchanged from current NPM. The registry metadata points at the authorative location.
• Bandwidth-efficiency: P2P distribution, with webseeds as a fallback.
• Resistance to 'corruption': Expensive to go rogue, cheap to mitigate. See 'threat models' section above.
• Private registries: Tools like Sinopia can be used as normal. I have not taken this into account in the design yet, but can't see a reason why it wouldn't work. I have not yet considered whether better solutions exist.

Unsolved problems

• How is authentication data transferred between write servers?

  When migrating to a different write server, that write server would need some way to transfer authentication data, but without exposing users. A possible approach for this would be keypair authentication (as the public key can be freely exposed), but this introduces a problem of access recovery.

  How would a write server decide to do a 'password reset' for somebody, when keypair cryptography is involved? Conversely, how can the new write server trust the old write server to be supplying the correct authentication data, rather than eg. 'backdoored' authentication data? Audits can be done (as the public keys are publicly exposed), but this can still miss things.

[scriptjs]

From what I see, there are few topics to look at this holistically:

Minimal viable client in node

I think if we were to agree on the notion @jasnell came up with yesterday and @Qard reiterated here today, we can have something tangible here soon. Along the lines of solving the broader issue of distributed package management the rest can fall into place. Someone can start a repo. Most of this code is already available in some form. This is essentially what bower was for the most part. This resolves the optics of NPMs bundling in node, NPM as an upstream, and opens node up to developer choice. We should agree on what it will handle but it should be as little as possible. Enough to bring in a client and build it if necessary.

Full featured clients

Let developers choose their full featured client based on its capabilities and features. NPM is currently what most use today but there are viable clients up and coming that are showing promise and will be competitive this year. We don't need to solve this here, let the community do its thing and build software that offers these choices.

Vetting/moderation of modules for inclusion in a registry

This is a process. I think a community process could be established in conjunction with nodesecurity.io. This can be phased in with a process of working through existing packages. Currently, there is a lot of cruft and a large percentage of the current registry is never accessed.

A public module publishing process might work where a module could move through a community vetting process. Vetted modules could be held in a central registry by the node foundation. This central registry is not one used for everyone to fetch modules, but used as a source of updates to public registries. Something suitable like Rackspace Cloud or S3 could be used with an API by the node foundation to host the central registry.

Public registries

Public registries should adhere to some basic standards set out by the node foundation for trust. Such as they will only host vetted modules and must maintain their sources with updates by syncing with the central registry. A public registry should not need to be more than a source of static files, whether that is a CDN or peer that persists the data to seed it. It is possible for public registry to provide a static endpoint and peer simultaneously. Dat as an example of a peer to peer system requires a key for the data to access the content.

Let those interested in hosting registries freely host if they can meet these basic standards. The node foundations can create an icon or something to verify their status as a registry host.

Private registries

Here again let the community do its thing. Whether you want to use your own git repos, sinopia, commercial provider, etc.

Module discovery

Currently there is npm for searching modules, but I would encourage any company to use the source of modules to create innovative solutions for search and discovery. The volume of metadata is also not large for todays search software like elasticsearch etc to build something interesting.

package.json

The package.json is something that needs to be under community control for minimum standardized metadata properties. Formal proposals should need to come to make changes to this standard that are in the best interests of the community.

[ChALkeR]

@scriptjs Sorry, but your post again does not look like a technical proposal to me.

I doubt that manual verification of all modules before publishing them will work — who is going to do that? I doubt that «trust» and «standards» would work that way you suppose they would: either you should «trust» a lot of public registries, or your proposal wouldn't be distributed enough — you should, in fact, not trust them but sign the packages and make sure that those public registries could not meddle into packages content. I do not see how fast content moderation of already published modules would work in your system, if there would be a lot of those public registries. I also do not see how the emergency switch (aka «turn the whole thing down») would work in your model. I also don't like the idea of recommending multiple clients or giving the user an early choice there.

@joepie91 proposal (with splitting that into three groups — a central/replicated auth, a few read servers and various distribution nodes would work better, I think. Perhaps it would be good to add a possibility of using other methods of delivery as «distribution nodes» — e.g. tarballs from GitHub (given that those are signed by the auth «write» server), etc. The bad thing about directly installing unpublished versions from GitHub is that the tags on GitHub are not immutable, so auth server should store the information about which versions (tarballs) are signed.

Emergency switch could be introduced on the client-side, so that it would check the flag with auth/read servers (@joepie91 has a bit more to say about that), perhaps with an opt-out on the client, but that opt-out should give the user some grave warnings and require some non-trivial actions.

As for the usability — the auth server could offer GitHub auth (as one of the possible login methods) and get public keys directly from the user account on GitHub.

I also think that replacing the bundled npm with a «minimal viable client» would not be a great solution atm — it could introduce more problems than it solves. If we replace it with something, let's make sure that the new solution is superior.

Private registries and private registries hostings would be on their own, and that is fine enough.

[scriptjs]

@ChALkeR This is not a technical proposal, you are correct. It is a high level view of the elements of a system beginning with what is packaged with node. It frames responsibilities for a system so we can continue a technical discussion on the same page as to how we see this as a whole and who might be responsible for what.

I think it is well understood that registries need to be read only and that signed packages are a prerequisite. What we have today, however, is also a number of packages and cruft in NPM that should have never reached a public registry in the first place, yet is there and persisted. There needs to be some gatekeeping.

Here, there is the notion of a central authority for the modules. Publishing a module for first time could place it into a vetting/review queue. Vetted/trusted modules enter central registry that is operated by the Node Foundation. The function of the central registry is only for public registries to sync their content. Public registries offer read only access to the signed tarballs. Every public registry serves the same content.

[ChALkeR]

@scriptjs Are you saying that someone out there should manually review all the diffs between all the versions of all modules in that sparkling registry? That just sounds as an ideal approach, but there is no such amount of reviewers time available to achieve that.

[joepie91]

Regarding the emergency switch, the sanest implementation of that would probably be for the write server to propagate a (signed) 'shutdown signal' to the read servers, and have the clients simply rely on the read servers to tell them whether they can install packages.

This shutdown signal could be used in cases where some kind of wide-spread security issue were to make package installations unsafe. Ideally, it'd never be needed.

• Why not let the clients talk to the write server? Because that makes the write server a single point of failure.

• What if somebody attacks the write server to prevent propagation of the shutdown signal? This signal would be sent once, or upon connection of a read server. As read servers are expected to be running continuously, it is reasonable for the read server daemon to refuse to start if the write server is unavailable.

• Why does it matter whether the shutdown signal gets propagated?

  • If we assume that "no write server available" means all is fine, an attacker can intentionally prevent clients from being notified of a dangerous situation, by attacking the write server.
  • If we assume that "no write server is available" means that something is wrong and shut down installations accordingly, this makes the write server a single point of failure, and an attack towards it would effectively take down the entire infrastructure. This (partially) defeats the point of having multiple read servers, by giving up redundancy.

  In this context, "one read server less" is the least impactful failure mode for the Node.js ecosystem.

I should note that the above is a result of @ChALkeR explaining to me the need to have a 'killswitch' - perhaps he could elaborate on the kind of scenario he is envisioning.

---

As for mutability of Git tags - I'm afraid that this is inevitable, and given that Git tags are only meant as a last-resort installation method, this could be an acceptable tradeoff.

Whereas I initially thought that adding a hash to the installation URI would resolve the matter, this would remove the possibility of using semantic versioning ranges, which I personally consider to be an essential part of NPM's packaging model, and necessary to make Git tags an 'equivalent approach' to regular dependency specifications.

Enterprises that need an absolute guarantee of immutability (ie. not able to trust the registry host either), already need to check in node_modules or use something like Sinopia inbetween, so I'd imagine that the loss of immutability guarantees on a small number of packages would not be a big problem.

---

@scriptjs I don't feel that a 'walled garden' would be a viable solution. Part of the reason why the Node.js ecosystem is so useful, is because everybody can publish modules. Reviewing all modules would introduce a significant delay, as well as manpower requirements (further increasing the dependency on the registry operator, which is precisely what we don't want).

A better solution would be to have two after-the-fact review queues - one that every package goes through to filter out the obviously malicious stuff (this may already exist), and one queue that people can request their modules to be placed into, for more in-depth review that can cover matters like security and code quality (as time is available). Neither of these queues need to be visible to the public, nor do they need to happen before package publication.

[scriptjs]

@ChALkeR No. I am not speaking of a form of detailed review here but something that would prevent cruft or malicious code from entering the registry. Currently there is zero barrier to publishing anything even if done in error.

I think at the very least there could be a scan of the package by an automated tool or an approach that would generate an issue where we let the community examine new modules that will entering the registry over some days.

This could work using a graduated program so that new authors will be slowed while trust is built. A module passes, the author/publisher earns some level of trust. We let trusted authors publish freely.

@joepie91 This is not meant to deter publishing by anyone at all. Only to be proactive rather than reactive.

[joepie91]

This is not meant to deter publishing by anyone at all. Only to be proactive rather than reactive.

Regardless of whether it's meant to do that, it will do that. The core reason why NPM grows as quickly as it does, is that there are absolutely no barriers to publishing things. A delay in publishing modules will discourage people from publishing anything at all.