Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Investigate alternatives to NPM #291
tldr; Edit 8 Apr 2017
Pulled out of NPM. An alternative is written at prettydiff/biddle and it will become a dependency of Pretty Diff very soon.
The intention is to find a distribution solution so that Pretty Diff can leave NPM.
These two quotes from NPM co-founder Laurie Voss (CTO) are the straw that broken the camel's back:
Conflict to Open Source
I fundamentally disagree. The needs of a package owner completely and absolutely outweigh everybody else. This is a fundamental issue of open source software. The commercial resolution to this condition exists in various forms of warranties, contracts, and documented obligations. Where such documents are neither expressed nor implied the open source software exists for use entirely at risk to the consuming dependent. This the reality of open source software and is stated as such in the most popular open source software licenses.
NPM forms their position because they are a revenue generating business who sell infrastructure. The most important quality to their business is availability (like any infrastructure model), which, in this case, puts their business into conflict with basic principles of open source software. This is an exceedingly rare edge case even if absolutely fundamental. When availability is in question the value of their service is called into question and their brand is severely harmed.
The reason why this conflict likely exists (speculation) is because NPM traffic growth is largely driven by a dependency model. Use of dependencies requires a choice between freely available work (and assuming all risks of that work and its availability) versus doing that work yourself locally.
Personal Thoughts and Convictions of Liberty
The quoted statement from NPM is the opposite of liberty. I hold the concepts of liberty and leadership above all other sociopolitical establishments and is the primary reason I continue to serve in the US Army after 19 years.
If the subject at hand were of gay rights or racial distinctions, instead of software, people would not be so generally silent on the subject. There would be loud angry boycotts and protests. While the community of NPM users and leaders may distinguish these subjects differently liberty does not. The logical conclusions then are that:
Regardless of whether the thoughts and actions opposing liberty are willful, as an adherent to the practice of supporting and upholding liberty as a paramount virtue I must remove myself from that community and no longer contribute works toward it. I believe a willful opposition to liberty on principle alone is perhaps the greatest of evils and all that it takes for evil to thrive is for good persons to do nothing.
For the definition of liberty I subscribe to the preeminent treatise On Liberty by John Stuart Mill. I have included a few select quotes from the second chapter, the first of which is the thesis of the essay.
NPM can arbitrarily reassign a package name without owner consent or awareness. I understand the occasional need for this and that such power is used exceedingly rarely. NPM can violate a package maintainers' intentions to unpublish by arbitrarily publishing the same code under the same name for continuity. When these two concerns coincide namespace assignment and management of all associated code are fundamentally lost to the package maintainers. This presents immediate risks to product management (for distribution) and thus secondary consequences of associated branding and product health decisions are lost to maintainers. The third level consequences apply to disruptions of contractual obligations that arise from loss of control of maintained packages.
Weak Software and a Community of Dependency Hell
The problem that precipitated the NPM action and quotes above is due to a catastrophic break from the removal a single tiny artifact. This could be characterized as the removal of a grain a straw that instantly destroyed all the buildings in a grand city upon its plucking. The problem there is not the result of an NPM directed action or the malicious convictions of a single evil agent. The problem is a community entirely reliant upon a system of dependencies, without regard for integrity or health, with NPM directly serving as the primary enabler.
This community problem exists because weak software developers value convenience above other more necessary considerations. Audits are not performed (not regularly, not ever) over included dependencies and the dependencies of the primary dependencies are certainly not thought of. These dependencies are frequently included without constraint or safety of any kind. Emerging software developers who are new to software development take this weakness as standard practice and so the weakness is the new baseline. The consequence is that discussions and thoughts of architecture are avoided out-right thereby avoiding ownership of responsibility, so therefore it becomes the user's fault when software breaks due to the dependency madness.
This problem is observed in numerous posted issues to the Atom-Beautify project. Atom-Beautify is a large project containing many dependencies. Even after the project itself is well-written and does everything it can to reinforce concepts of health and integrity it still suffers from dependency hell due to the madness that comes in from certain grandchild dependencies. When a micro nth level dependency fails to come across the line there is only one solution: We tell the user to kindly reinstall. Clearly, this is a user error (due to no fault of the user) that we are powerless to fix.
Additional and unrelated frustrations that break Pretty Diff:
It is hard to qualify NPM download numbers. This is absolutely not related to any project health or security concern, so it is absolutely minor. NPM download counts appear to be total downloads. This includes counts for specifically requested packages, packages downloaded as any dependency type, packages further down the dependency chain, and various versions of the same package on the same dependent and/or dependency tree. This generates some level of confusion in that a package could get 30 (hypothetical) download counts when a single unrelated package is downloaded.
See also: #250
It would be nice to have more qualified numbers. In the reality of open source the numbers matter very little. In the rest of the world it is a validation. The rest of the world includes:
In order to prevent potential malicious entities from taking the NPM names reserved by user austincheney I will not unpublish from NPM. Instead Pretty Diff will upgrade to a new major version number when publication at an alternate venue is announced. Pretty Diff will continue to retain version 1.x.x at NPM, but it will no longer be maintained. This plan also minimizes any disruptions associated with this move.
Custom (self-hosted) package management
Another solution I have been thinking about is to simple write a new node application that does the job of NPM, but without a registry. You don't need a centrally managed registry, because the web has already solved this problem and applied the solution with great success in the past.
URI to the rescue
URI is always unique, is universal, and can be applied with varying degrees of specificity. Its universal nature solves the NPM namespace collision problem while simultaneously ignoring the friction of public/private or commercial/free.
Listing of basic requirements for package management
Example of an applied solution
The Pretty Diff project has its own domain at http://prettydiff.com/ so I will use this as a potential demonstration. Let's presume there is a http://prettydiff.com/package which contains an index file that points to the default version via symlink so you when go to that address a tarball is returned. Let's also presume there is a http://prettydiff.com/package/versions directory, so for example http://prettydiff.com/packages/versions/1.16.37
When applied in this way the version format is completely custom so long as there is an address that points to tarball of matching version, such as http://prettydiff.com/packages/versions/1.16.37--alpha3. If you want the package to be private then move it to a web-server behind a firewall. Even better would be to move it to an internally available address/domain scheme. Since this idea is reliant upon URI the privacy of a given package is vested in the ability to resolve an address.
The great thing about this is that its an application and not a service. The only required service is a HTTP server (a simple one comes prepackaged with Node). The application must run in response to a command that creates a new package tarball, puts that tarball in the proper location (a version directory on the web server), updates the default package symlink, and optionally beacons that a new version is published.
This, like everything else in this idea, is completely open to custom definition. It would just have to be defined in the package's data file so that the format and point of access are known to the distant end where the package is used.
A client application
Unfortunately a client application would also be required to, at least, unpack the tarball. Some optional (ideal) qualities for the client-side app:
This was referenced
Mar 30, 2016
Sad to see npm/npm#10686 mentioned (and still not fixed). I actually bisected the thing down to a single commit (a botched refactoring/variable renaming attempt). To my knowledge, no-one ever looked at that bad commit as a source of the problem. I think my comment simply got drowned out in the incredible noise from people pasting entire npm logs while trying to prove that they had "the same problem"...
The one that really REALLY did it for me was #5082.
The problem for me isn't that there are defects in software or even the severity of the defects. All software has defects of which some are hard to solve even by the people best positioned to provide the solutions. My problem is that the defects are so incredibly hard to solve that they remain unsolved for so long. Normally in open source software defects will remain open for a long time due to the limited time of volunteers. NPM has a paid staff, hundreds of contributors, thousands of participants, and millions of users. Limited man power isn't an argument for NPM when problems are so dangerously critical.
This is a problem for me, because it is symptomatic of bigger problems. Why would such foundational problems be so hard to solve for or so challenging to detect? Is it because of how the software is organized, managed, or executed? This line of thinking led me to believe the primary problem of NPM is its over-reliance on modularity.
When I was looking into issue 5082. I had trouble following the code. The application was divided into many different modules written by NPM staff. The NPM code style is really sloppy which opens the code to some ambiguity and the tar component was binary heavy with great use of typed arrays. Each of these qualities compounded the problem and likely prevented entry from many people willing to solve this problem. Had the various components been centralized into a single application the problem would have been easier to detect. Even though the code would still be sloppy the problem would be confined to a single known location, so at the very least you would know where to start and could test your way through until test criteria is firmly defined.
Although I identified a technical hurdle limiting understandability of the code it wasn't until I heard the responses to the left-pad problem that I realized the problem isn't technical at all. It is purely a management problem of the code. This is a design problem more so than an execution problem. This is the thinking that led me to design an alternative. I knew I could do this not because of confidence in my programming skills, but because they identified a unique problem to solve that they are less willing or capable of solving themselves.
referenced this issue
Nov 23, 2016
referenced this issue
Dec 2, 2016
As a maintainer of Yarn there are a couple things developers need from distribution to be successful.
Yarn has checksums, so this one is done.
I should have the ability to publish my application in a variety of different flavors simultaneously. If somebody wanted to tinker with my application they should be able to request the entire thing unit tests and all (4mb zip). If they want only command line operation (170kb zip) then they shouldn't have download the other 4mb of nonsense. biddle does this and Yarn could too.
Because NPM and Yarn don't have this many developers publish one version to their git server for storage, and different version to NPM that is only good for use in a limited context. Sometimes, if the application is popular enough and well staffed, they will publish multiple variations to NPM to manually work around this limitation. All of this can be automated.
Dependency management sucks. It's hard. Fine, I got it. Either you are taking that pain as a product owner or you are forcing that stupidity upon your users. It makes for horribly fragile software that breaks all the time. This problem is magnified for Chinese users behind the Great Firewall. They really hate this madness.
This NPM style dependency madness should be optional and disabled by default. Its great for software developers who are more consumers than publishers and really need that level of convenience, but I wouldn't trust or endorse any application in production like this.
By default dependencies should be rolled up into the published artifact. Consider, for instance, when you download something from Steam. Are you download one large file or a million snowflakes? The installation step should be automatically managed by the distribution app according to instructions provided by the application's author. There is no dependency madness and limited configuration hell. The user just requests their application and achieves immediate satisfaction.
Nice to haves
Alternate Repositories of the Author's Choosing
I am a big believer in liberty. It is ultimately why I gave up on NPM. Regardless of all the best wishes, intentions, and desires a central repository is ultimately a beacon of censorship by two means. On one hand if there is a single universal standard repo then anything they don't like they could forbid, which is censorship by omission. Secondly, they could silence or eliminate something that is published, which is a purge. Let's not forget the primary cause of the Left-Pad failure was a trademark conflict.
The cure for this form of censorship to allow a package publisher the means to publish to any number of locations of their choosing.
When a package author has the freedom to choose where they publish to they get privacy for free. It is simply a matter of moving the publication point from the public network to the internal network. Privacy for free.
I work for the military where confidentiality is important. If the means to distribute software requires beacons to a centralized third party the military will never adopt it. If, on the other hand, there is no third party network traffic and confidentiality labels can be automatically applied to software at the publication point and are read before unpacking the zip the military would be all over it. This isn't just useful to the military, but is also useful to the corporate enterprise where legal documents and financial documents exist at a higher level of classification.
Variations: If a single package is this flexible, you should be breaking it apart into separate packages. Many tiny packages are controversial, but ultimately the right way to share code.
In regards to file size, Yarn has
Packaged Dependencies: I disagree, npm's best innovation was the design of the
Reliability and performance are separate matters that can be addressed. I have friends in Asia-Pacific who quite like the performance and reliability improvements of Yarn.
Alternative Repositories: You've always been able to publish and install packages to other locations with both npm and Yarn. npm has done nothing to lock you into their platform.
Also the primary cause of left-pad was a whiny brat.
Privacy: (see "Alternative Repositories")
Confidentiality: (see "Alternative Repositories")
I am not centering this position solely upon NPM. It is an anti-authority/decentralization move.
In my opinion the greatest technological democracies are BGP and DNS. They are decentralized. When one item in those systems falls off the interest isn't harmed, due to redundancy, though it might get a little slower. Those technologies provide no means to enact of enforce censorship. All they do is share routes or name resolution. Even if they wanted it or were so ordered they can only be made to harm their internal users. The network at large remains free and unrestrained, because a change to one item in those systems is by no means a change to multiple items in those systems.
The problem with central authorities is two fold. If everybody is reliant upon the central authority for content or distribution the central authority can enact censorship by blocking publication from users it wishes. There is likely a very good reason to block certain content whether it be obscene material, disgusting and hateful material, or simply an intellectual property violation. Regardless of the reason and no matter how noble the intention this is still censorship. Where the availability of certain content becomes a legal violation it is best to let the legal authorities impose the necessary corrections according to the law, which can be challenged upon appeal.
The other side of that coin is that now the system is a bit less diverse. Most of us don't like hate speech and wish it would just go away, but this is a slippery slope. Eventually matters of political discourse quickly become hateful and minority opinions are better suppressed. It is often necessary for the health and well being of the system to prevent trolling. The challenge is that some opinions are naturally upsetting and challenge the common order. Whether these opinions challenge commonly held assumptions, or the majority opinion, does not necessarily mean they aren't worth having. If you don't enjoy a particular sort of content you don't have to consume it, which isn't the same thing as preventing other people from access by blocking it from the central system.
When there is redundancy and no central system the content remains alive and available long after it is removed from a single location. In a centralized authority there is no redundancy. Sure NPM has mirrors and CDNs so if a file cannot be retrieved from one known address it can be fetched from another address. But, in order to be fetched it is resolved against a name that is centrally managed. For instance I am essentially squatting on the NPM name prettydiff. I cannot stop squatting on this name or allow it to be replaced with something else because then the 75,000 users who download this every month will be harmed. A publisher wishing to use that name for something that is actively maintained in the system cannot, because I am squatting there. For fear of harming the system, and its users, I cannot even give this name away if I wanted to. This is a huge limitation.
Can I publish to Yarn's repository independent of NPM name resolution. I have been told no. Therefore Yarn is even less free than NPM, because it is subject to the authoritarianism on names at NPM as well as imposing a similar authority of its own. The argument in defense is to imagine the horror if somebody registered jquery on Yarn and publishing something different than what is on NPM? My argument is that they would get different software. That isn't bad. It is just different.
Half the problem has long since been solved. The bottle neck is the simplicity and convenience of a single name without a context. This is a problem that has long been solved with conventions such as URI and IP. It has mostly been solved with hashes, aside from the rare hash collision. If the application resolution were by address, opposed to name, then any user could publish an application to Yarn completely independent of NPM, because they are identified differently, such as different domains for resolution.
The other half the problem is to prevent a single authority to own, limit, or regulate what can be published or accessed via a ubiquitous system. This is solved by encouraging duplication and republication while providing rights for those who republish above and beyond those who merely consume the application. Trust and integrity are verifiable with hash comparisons regardless of where an application is served from.
NPM is popular because it is convenient for developers. It isn't convenient for application consumers. If you had to choose between (fill in the blank) and a more democratic approach what would you choose? Historically speaking there is always valid cause to disagree, but the means to allow that disagreement is itself an independent decentralized opinion.
If you want to throw a tantrum about npm following the law and taking down a package after a dispute: Fine, at least your heart seems to be in the right place.
But you aren't punishing npm at all, you're punishing developers like me who just want to write code and not deal with 8 separate package managers.
I'm not adding Biddle as a dependency, I don't trust you after this incident– nor should anyone.
Perhaps. This software retains a copyright and license, so it isn't in the public domain, which means it is regulated to some extent. As the project owner, for better or worse, I control that direction.
My intention is not to punish anybody. If anybody is being punished for this decision it is primarily this project and software reliant upon it. I have lost considerable traffic, and as a result external credibility. In January of 2016 I had just shy of a million NPM downloads almost entirely from outside the NPM system.
In conclusion this is free software. I have invested my time and money in this work without ever asking for payment or labor. I provide it to you and everybody else as a courtesy, because community feedback makes identifying defects and suggesting enhancements far easier. We are all better for it. I am not forcing you to use or consume this software. If you choose to not use this software I don't really care, because countless others will. There are many alternatives available of which most are far more popular and on NPM/Yarn. Go use one of those.
I will leave you with this: http://www.constitution.org/jsm/liberty.htm
I have gotten by with git submodules.
I also write a lot of free software and am working to get more people paid for the free software that they create. I don't go punishing my users like you do though.
Well there's intention and then there's effect. I don't really care what your intentions are.
So you admit you're punishing users? And you think that's okay because...
I don't actually use this library for anything. I'm just here because you're being a jerk.
PS: I don't think you have a good understanding of "liberty", I think you should read some more.
Also I'm not really worried, your new user-hostile license contradicts its primary function so you can't really make an argument against what I'm doing.
I'm not telling you why though, because if you're gonna go writing your own license you should talk to a lawyer instead of pouring your frustrations over a keyboard.