Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
"scoped" packages #5239
The biggest request from enterprise users is a way to manage private modules with the same level of simplicity and ease as open source modules in the public npm registry. (A close second is group/role based management, but let's leave that for a subsequent discussion.)
Current State of the Art: Private Registry Replicas, Private Git, Proxies
Since the registry is (mostly) a Couch App, and we expose a public CouchDB interface to it, you can replicate the metadata into a database on your network, and optionally also use the npm-fullfat-registry module to fetch binaries into your copy of the db as attachments.
However, this is suboptimal in the following ways:
PayPal wrote kappa to deal with this, which doesn't require using a CouchDB full of attachments. There are several other similar proxy implementations that store the metadata and binaries to disk, but this is the one that I hear of the most often. This is a bit better than managing a Couch App, and has pluggable interfaces for caching, whitelists, etc.
Another option is to use private git repositories, either managed in-house or on GitHub. The npm client has some snazzy shorthands for GitHub repos, so you can do
If you use basically any private system, you'll end up with name conflicts. Let's say our company creates a module called
We can get around this by implementing some rather convoluted logic to manage multiple registries, and keeping track of where on the list of registries we started in the lookup process, and always go down the list rather than up.
However, this means that we can't do things like fork the markdown parser package named
What's worse, the logic for knowing how to resolve conflicts gets really confusing to reason about, and when you're looking at code, it's not clear what
So, we have to be very careful to not use names that are publicly used, or squat on public names, which disrupts the OSS community flow, and so is generally bad behavior.
It'd be nice if we could scope modules to a specific user or organization. If such scoped modules were able to be made private, and if they could be unambiguously recognized by the client, then it would know that it might have to do a "give me access to this private thing" authorization dance in those cases.
I've discussed this with the people here at npm, Inc., and with a few of the most prolific npm contributors and publishers, and users at several large companies. The scheme that seems to be the most intuitive, which also fits within the rather narrow constraints is
We can then also scope auth info to each registry, and voila, we have multiple registry support that works, doesn't conflict with anything that already exists, doesn't result in fetching the right thing from the wrong place.
And since that's the actual package name, you'll do
As a way of implementation details, the
This does mean that the registry will probably not be strictly just a Couch App for much longer. In practice, it already isn't, but since CouchDB chokes on doc names that have a
We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic. Basically, npm-registry-couchapp will be just a part of the registry implementation, instead of effectively the entire thing. In addition, there'll be a minimal "front door" bit that proxies to the Couch App, manages roles, restricts GETs, etc., and that's where you'll find hooks for auth magic. That also means we can avoid putting the attachments into CouchDB in the first place, since all we do is then immediately take them out of CouchDB, without breaking old clients, and opens the door to doing other creative stuff, like more efficient views, relational reporting stuff, and putting download counts and other analytics right into the data itself. Handwavey vaporware, etc.
Would you use this? What would you like it to do? Anything about it that you think would be weird or confusing? Is there an approach you'd like better that meets all the constraints?
First class citizen questions
The existing story around private git links has some usability problems.
We should think of all the areas where git+ssh links are a second class citizen in terms of the features of the npm cli and ensure that
This will help address
By making @org modules just as easy & simple as open source ones.
Effectively, orgs are users. You've already "reserved"
Names will be handled similar to the "disputes" policies that we already have. If your company name is
No, because you're not in the
Don't do that :)
You do bring up a reasonable issue, in that this doesn't solve all namespace problems, but "merely" adds a single additional dimension, so there's "only" one less order of namespace collision magnitude.
Yes. Shrinkwrap will know how to deal with
Probably it will throw a "module not found" flavor of ENOENT.
Tell me about it! They're kind of awful.
Agreed. We're starting with the experience of non-org-scoped modules ("global" modules, in our internal parlance), and going from there. Rather than using Git as the model, since it is really klunky to work into npm's way of doing things.
That is the plan :)
"Global" modules (ie, unscoped, what we have now) will not be allowed to depend on scoped modules. This is important.
Otherwise, the community as it is now would be unduly harmed, because we'd very quickly end up with global packages that can't be installed by old versions of npm.
This "namespaced maybe-private org-scoped" thing is only going to work with newer npm clients of the future, but there's no reason to invite ruin on older npm clients that are currently working, unless it's actually necessary for some reason.
I want people to upgrade to the latest and greatest npm, but it should be much more carrot than stick.
just for clarification, if an
if it were the latter, more existing setups may be supported, and if you have a private registry that supports the new
@hayes That's interesting. Our assumption had been that it'd be
@izs thanks for opening a place to discuss this.
This is of course inevitable, which is why the proxy I wrote with @jcrugzz has a policy format which understands both blacklists and what we refer to as "known private modules". This also solves the question from @Raynos
Because you could simply blacklist your
I agree in principle that we need something to hang users off of. Discussions around "groups" have happened on and off for years, which is essentially the same thing as what is being referred to "orgs" now. What I don't explicitly see the value in is the
Knowing that private packages will in principle cause a large swath of breaking changes for users why not just roll prefix-less orgs into
That's a pretty bold statement, but we of course have our own pieces that are private at Nodejitsu so I can't call the kettle black. The biggest request I would make it make importing and exporting private modules out of the private thing easy because both of our customers will want that.
+1 to moving more complex logic out of the CouchApp as jcrugzz outlined in his post back in February. With that in mind, if you are overhauling the auth system we should really consider implementing OAuth2 as a proper way to grant authorization to other apps. It's not the best, but OAuth2 has it's fingers in enough things that it's a sane first choice.
If you move the main proxy logic out of the CouchApp (i.e. by removing the rewrites) then the
Aside: Package signing
If we are really going to say "send all requests to
@Raynos An argument to disallow publishing packages that depend on
To clarify some confusion, where does the config mapping between
And does a missing
@chrisdickinson The resolution of an @org to a given registry happens in the client, although the responsibility for validating permissions will probably be scattered throughout the infrastructure. All configuration will be handled using the configuration mechanisms that exist today, although if / once support for multiple / per-project configuration files lands, it might make sense to migrate some of this there. The implication is that @org can map to different registries for different users, depending on their configuration. We have discussed how to consolidate these mappings site-wide for large installations, although we haven't arrived at a final decision on how that's going to work.
Something useful to keep in mind is that the mechanics of resolving packages exists independently from the permissions governing how the server determines whether a given publish is permissible (to speak to @Raynos's point). I'll let Isaac speak to the specifics of how that will work.
@othiym23 Thanks! The points I was somewhat foggy about were as follows:
As I understand it, the
Please see the list of namespace constraints in the OP.
This probably will not use the
Can you elaborate on which breaking changes are planned? I don't see any impact on existing users. They may be left behind if the bulk of development activities move toward namespaced modules, but their existing workflows will keep working.
Sorry, I'm not sure which is the bold statement there? Most of what we do at npm, Inc. is developed in the open, and I don't see any reason to change that.
Xylophone. Implementing an OAuth2 provider for npmjs.org is an idea worth exploring, but it's well outside the scope of this discussion.
Xylophone. Package signing is important, but outside the scope of this discussion.
@othiym23 asks: "What is 'xylophone'?"
On Github, the top-level namespace is the same for both users and organizations. Same deal here. This concept is not exclusive to private registries.
Given the fact that private packages can depend on public packages, but not vice versa, this is not necessarily the future. As an aside, I find it easiest to think about @orgs as segmenting one namespace, rather than creating multiple top-level namespaces.
If you mean, can package
This is correct.
We only switched to
The community wasn't ruined, but it was also a much smaller change than this addition, and in retrospect, I would have liked very much to have put more thought into avoiding that issue. In this case, the rule that "global modules can't depend on scoped modules" avoids the problems rather nicely.
Still working out the best way to accomplish that, but yes, you'll add a user or a group to the package, and then there can be a way to add a user to a specific group.
Part of the purpose of our on-prem beta program is to give us a little petri dish in which to experiment with a few different ways to manage auth and groups. Most likely, there'll be group management by the end of the year in the public registry, and we can have another one of these to discuss that. Until then: xylophone.
Scoped modules will be either public or private, at the whim of the owner. I will not expect that they will ever be migrated to the global namespace. This thus satisfies another common request: having simpler/shorter names for packages, scoped to a specific owner, so that you can have
I think you're understanding it correctly. Deps can go from scoped -> scoped, scoped -> global, or global -> global, but never global -> scoped.
One downside of the @-sign that I didn't realize until posting this discussion here: we've been spamming Olly Groves (https://github.com/org) quite a lot. I apologize for that, Olly. You may want to ignore this thread entirely, I suspect it will continue.
To amplify this, this makes it easier for people to fork and experiment with existing packages to scratch particular itches without the original package needing to change or grow unnecessarily.
Can namespaces mutate their privateness?
... to expand:
EDIT: I misunderstood. @othiym23 straightened me out:
Since an organization can be depended on by other organizations, and because local configuration can change where @org resolves, we'll have to register an npmjs.org account for the organization to ensure we don't accidentally shadow some other organization.
If @org is directed to some other registry, but there are some @org packages that are public, the other registry would need to have both the private and the public packages, as the client won't fallback to the public registry.
Previous discussion within the community seemed to have concluded on scoping packages by registry, not by organization. I want to make sure I'm entirely clear of the impact here.
Turned into a bit of an essay, but please bear with me. I am not simply disagreeing for the sake of being difficult.
The private mechanism is important here
So if the TL;DR here is "org ns as step towards private modules" it is reasonable to ask what the mechanism for making things private is going to be.
I (wrongly) assumed that the plan was to repurpose
Why so in-favor of strict backwards compatibility?
Right now there are 225 public packages which use this syntax in their
It would require formal deprecation of the Github syntax, but it precludes a new syntax. Further, we could use the new
Based on the anecdotal evidence I've seen organizations would be used 1-2 orders of magnitude more frequently than the existing github shorthand -- most of them use the
Why are we complicating things?
If scoped modules are public by default as @chrisdickinson and @othiym23 said, why the restriction? There is a widely used workflow at companies using Node.js (and even client-side JS) which implicitly conflicts with it:
Xylophone or just not on this issue?
Why is this a xylophone? It is hard (if not impossible) to completely assess the intrinsic value of the proposed organizations feature without fully understanding how the publish and user management mechanics will work. e.g. our customers love user sync across their team.
The same goes for OAuth2 and package signing. All of these features are intertwined in nuanced ways: the implementation details matter here. I can move that part of the discussion to another issue if you'd prefer.
Can we count on this
Just to clarify, people want to change
If we do use scope in require I would avoid using a specialized string in require. Keeping it separated ala
Either way, I would need more info on what is actually being proposed to be put onto disk to say much more. This would not interfere with archives or bundling most likely since that just reuses the
referenced this issue
Sep 13, 2014
With the publishing of
Thank you to everyone here for your contributions!