"scoped" packages #5239

Closed
isaacs opened this Issue May 8, 2014 · 116 comments

Comments

Projects
None yet
Owner

isaacs commented May 8, 2014

The biggest request from enterprise users is a way to manage private modules with the same level of simplicity and ease as open source modules in the public npm registry. (A close second is group/role based management, but let's leave that for a subsequent discussion.)

Current State of the Art: Private Registry Replicas, Private Git, Proxies

Since the registry is (mostly) a Couch App, and we expose a public CouchDB interface to it, you can replicate the metadata into a database on your network, and optionally also use the npm-fullfat-registry module to fetch binaries into your copy of the db as attachments.

However, this is suboptimal in the following ways:

  1. You end up replicating hundreds of GB of stuff that you don't need.
  2. CouchDB with half a million binary attachments is not an easily maintained beast.

PayPal wrote kappa to deal with this, which doesn't require using a CouchDB full of attachments. There are several other similar proxy implementations that store the metadata and binaries to disk, but this is the one that I hear of the most often. This is a bit better than managing a Couch App, and has pluggable interfaces for caching, whitelists, etc.

Another option is to use private git repositories, either managed in-house or on GitHub. The npm client has some snazzy shorthands for GitHub repos, so you can do "foo":"org/node-foo" in your package.json dependencies, or npm install org/node-foo on the cli. However, there is a bit of an impedance mismatch and occasional friction in using git repos as npm dependencies. It's ok in a pinch, or if you have really well-established practices, but it's generally always going to be a bit of a mismatch, simply because git and npm make different assumptions about the world.

Name Conflicts

If you use basically any private system, you'll end up with name conflicts. Let's say our company creates a module called marked, to track places that we've marked in our location tracker app. Then we decide to use some public module that depends on @chjj's marked module, and it ends up installing our location tracker doohickey as a dep, because the registry is set to npm.my-company.com.

We can get around this by implementing some rather convoluted logic to manage multiple registries, and keeping track of where on the list of registries we started in the lookup process, and always go down the list rather than up.

However, this means that we can't do things like fork the markdown parser package named marked to fix a bug, and have our dependency lookups find that instead. There is no way to solve both problems, because they contradict one another.

What's worse, the logic for knowing how to resolve conflicts gets really confusing to reason about, and when you're looking at code, it's not clear what require("marked") refers to.

So, we have to be very careful to not use names that are publicly used, or squat on public names, which disrupts the OSS community flow, and so is generally bad behavior.

Namespace Constraints

It'd be nice if we could scope modules to a specific user or organization. If such scoped modules were able to be made private, and if they could be unambiguously recognized by the client, then it would know that it might have to do a "give me access to this private thing" authorization dance in those cases.

Some constraints:

  1. There must be a simple and unambiguous way to say "install mycompany's marked module".
  2. It must not conflict with any currently allowed names. (Ie, it should be invalid in the current paradigm. Not hard to do, since names are very restrictive.)
  3. It should not conflict with the user/repo shorthand used for GitHub. (Oops. That would've been a really nice way to go. Oh well, backwards compatibility. It sucks, but not as much as it would suck to break users.)
  4. If you do npm install <scoped module> then you should be able to do require("<scoped module>") in your JavaScript, and view the data at https://registry.npmjs.org/<scoped module> in a web browser.
  5. Since (3) implies we have some kind of magic character or syntax, (4) means that any such magic character/prefix/syntax must not use characters that are magical in URLs, file systems, or shells. (For example, npm install ~org/foo or npm install $org/foo would be fine in the fs and url, but the shell will expand ~org to the home of user org and $org to the value of the org var. + and # and % all have similar problems in other places.)

I've discussed this with the people here at npm, Inc., and with a few of the most prolific npm contributors and publishers, and users at several large companies. The scheme that seems to be the most intuitive, which also fits within the rather narrow constraints is @org/pkgname.

Multiple Registries

Each @org will map to a specific registry in the config (or the default --registry config, which defaults to registry.npmjs.org, if not specified).

We can then also scope auth info to each registry, and voila, we have multiple registry support that works, doesn't conflict with anything that already exists, doesn't result in fetching the right thing from the wrong place.

@org/pkg modules will be dropped onto the filesystem at node_modules/@org/foo, so require("@org/pkg") will work.

And since that's the actual package name, you'll do "dependencies":{"@org/pkg":"1.x"} in your package.json file.

As a way of implementation details, the @org/pkg thing will map to npm://{registry}/@org/pkg, where {registry} is the registry that is assigned to the @org or the default --registry config. npm:// urls will be fetched via https, and always send authentication information, and treat 404's as a potential "maybe you don't have permission" response.

This does mean that the registry will probably not be strictly just a Couch App for much longer. In practice, it already isn't, but since CouchDB chokes on doc names that have a / in them means that we'll have to %2F it, and that's just a bit confusing. Maybe we could use - instead of /, but the niceness of / as a filesystem separator is really awesome.

We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic. Basically, npm-registry-couchapp will be just a part of the registry implementation, instead of effectively the entire thing. In addition, there'll be a minimal "front door" bit that proxies to the Couch App, manages roles, restricts GETs, etc., and that's where you'll find hooks for auth magic. That also means we can avoid putting the attachments into CouchDB in the first place, since all we do is then immediately take them out of CouchDB, without breaking old clients, and opens the door to doing other creative stuff, like more efficient views, relational reporting stuff, and putting download counts and other analytics right into the data itself. Handwavey vaporware, etc.

Feedback please

Would you use this? What would you like it to do? Anything about it that you think would be weird or confusing? Is there an approach you'd like better that meets all the constraints?

Contributor

thomblake commented May 8, 2014

+1

Contributor

Raynos commented May 8, 2014

Questions

  • is the "name" in the package.json "@org/foo" ?
  • how do we deal with org names in the public registry ? How do we reserve them ? For example can I publish "@npm/config" ? What if I have an internal "@foo/bar" and want to use a public module that's also called "@foo/bar"
  • are we updating shrinkwrap to know how to deal with "@org/name" ? as far as shrinkwrap cares there is just a folder called "@org" in node_modules with no package.json.
  • what's the user experience when you accidentally require("@org")
  • will the node_modules/@org folder need a package.json ? What will blow up if the folder doesn't have a package.json
Contributor

Raynos commented May 8, 2014

First class citizen questions

The existing story around private git links has some usability problems.

  • will npm docs @org/foo work, it doesn't for git links
  • will npm info @org/foo work
  • will npm be able to dedupe these modules properly, it can't dedupe git links
  • will npm update @org/foo work ? It just installs the latest for git links
  • will npm i @org/foo@{range} work ? it doesn't for git links
  • will npm link @org/foo work ? it doesn't for git links.

We should think of all the areas where git+ssh links are a second class citizen in terms of the features of the npm cli and ensure that @org/foo does not suffer similar problems.

This will help address

manage private modules with the same level of simplicity and ease as open source modules

By making @org modules just as easy & simple as open source ones.

Contributor

Raynos commented May 8, 2014

btw this sounds great :) 👍 on this, let's make it really really smooth & easy to work with.

Owner

isaacs commented May 8, 2014

is the "name" in the package.json "" ?

Yes.

how do we deal with org names in the public registry ? How do we reserve them ?

Effectively, orgs are users. You've already "reserved" @raynos, for example.

Names will be handled similar to the "disputes" policies that we already have. If your company name is foo, and some jerkwad is squatting on @foo and not using it, we may be glad to evict them, but that's a "have the humans use their human brains for moderation" kind of issue.

For example can I publish "@npm/config" ?

No, because you're not in the @npm org.

What if I have an internal "" and want to use a public module that's also called ""

Don't do that :)

You do bring up a reasonable issue, in that this doesn't solve all namespace problems, but "merely" adds a single additional dimension, so there's "only" one less order of namespace collision magnitude.

are we updating shrinkwrap to know how to deal with "" ? as far as shrinkwrap cares there is just a folder called "@org" in node_modules with no package.json.

Yes. Shrinkwrap will know how to deal with @org/pkg as the name of a package.

what's the user experience when you accidentally require("@org")

Probably it will throw a "module not found" flavor of ENOENT.

will the node_modules/@org folder need a package.json ? What will blow up if the folder doesn't have a package.json

Nope. node_modules/@org will probably just have folders in it. If we do throw some kind of helpful metadata in there, it'll probably just be for the npm client's use.

The existing story around private git links has some usability problems.

Tell me about it! They're kind of awful.

will npm docs work, it doesn't for git links

Yes.

will npm info work

Yes.

will npm be able to dedupe these modules properly, it can't dedupe git links

Yes.

will npm update work ? It just installs the latest for git links

Yes.

will npm i @{range} work ? it doesn't for git links

Yes.

will npm link work ? it doesn't for git links.

Yes.

We should think of all the areas where git+ssh links are a second class citizen in terms of the features of the npm cli and ensure that does not suffer similar problems.

Agreed. We're starting with the experience of non-org-scoped modules ("global" modules, in our internal parlance), and going from there. Rather than using Git as the model, since it is really klunky to work into npm's way of doing things.

This will help address

manage private modules with the same level of simplicity and ease as open source modules
By making @org modules just as easy & simple as open source ones.

That is the plan :)

Owner

isaacs commented May 8, 2014

Another thing:

"Global" modules (ie, unscoped, what we have now) will not be allowed to depend on scoped modules. This is important.

Otherwise, the community as it is now would be unduly harmed, because we'd very quickly end up with global packages that can't be installed by old versions of npm.

This "namespaced maybe-private org-scoped" thing is only going to work with newer npm clients of the future, but there's no reason to invite ruin on older npm clients that are currently working, unless it's actually necessary for some reason.

I want people to upgrade to the latest and greatest npm, but it should be much more carrot than stick.

Contributor

hayes commented May 8, 2014

just for clarification, if an @my-org is scoped to a private registry https://registry.my-org.com will packages be fetched at https://registry.my-org.com/@my-org/my-module or just https://registry.my-org.com/my-module

if it were the latter, more existing setups may be supported, and if you have a private registry that supports the new @org pattern, you could set the url for for your orgs modules to https://registry.my-org.com/@my-org

Owner

isaacs commented May 8, 2014

@hayes That's interesting. Our assumption had been that it'd be https://registry.my-org.com/@my-org/my-module, so that the npm client could just Do The Same Thing no matter where it's fetching stuff from. But, I can definitely see the benefits there. Maybe "send the org name" could be configurable in the same way that the registry is, for maximum flexibility?

Contributor

hayes commented May 8, 2014

I am a big fan of flexibility. I think that could be a very useful options. but after thinking about it for a minute I realized it couldn't be the default

Contributor

indexzero commented May 8, 2014

@izs thanks for opening a place to discuss this.

Name Conflicts

This is of course inevitable, which is why the proxy I wrote with @jcrugzz has a policy format which understands both blacklists and what we refer to as "known private modules". This also solves the question from @Raynos

What if I have an internal "" and want to use a public module that's also called ""

Because you could simply blacklist your @foo/bar. The default for all publishes is to proxy them to your private registry, which leads to the one caveat of our private npm product: existing public modules proxy pass through just fine, but that new public modules have to explicitly be published directly to the public registry. This gotcha does come up every now and again, I've seen support tickets heard this from 4-5 customers (about 1-3%).

Namespace Constraints

I agree in principle that we need something to hang users off of. Discussions around "groups" have happened on and off for years, which is essentially the same thing as what is being referred to "orgs" now. What I don't explicitly see the value in is the @ scoping. If you are implementing private packages using the existing "private" then a large breaking change is already being introduced since up until now "private" has meant "never publish this thing to any registry ever".

Knowing that private packages will in principle cause a large swath of breaking changes for users why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now? It should be pretty easy to run a view to see how many packages are using that now. Whichever thing has lower impact from data should be the answer. I'm separately of the opinion that backwards compatibility shouldn't be preserved just because reasons, but then again seeing @ry this week might've just made me nostalgic :)

Multiple registries

We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic.

That's a pretty bold statement, but we of course have our own pieces that are private at Nodejitsu so I can't call the kettle black. The biggest request I would make it make importing and exporting private modules out of the private thing easy because both of our customers will want that.

+1 to moving more complex logic out of the CouchApp as jcrugzz outlined in his post back in February. With that in mind, if you are overhauling the auth system we should really consider implementing OAuth2 as a proper way to grant authorization to other apps. It's not the best, but OAuth2 has it's fingers in enough things that it's a sane first choice.

If you move the main proxy logic out of the CouchApp (i.e. by removing the rewrites) then the %2Fs aren't too bad. I've done this a lot for other CouchDB based applications to get vanity URLs and just taking advantage of the kv + map-reduce qualities of CouchDB. Clearly I have other opinions about where binary data should be, but that's outside the scope of this discussion.

Aside: Package signing

If we are really going to say "send all requests to @org/* packages to http://etc-etc-etc" we still have not prevented the specter of MITM attacks because by introducting N places to get packages we simply have N more places that could be vulnerable. I think that "orgs", "teams" (i.e. group/role management), and package signing are very much intertwined and should be thought of as a single unit to avoid painting ourselves into a corner as we unfortunately have done here with Github syntax.

OK but what does this have to do with twitter?

Contributor

Raynos commented May 8, 2014

@isaacs

but there's no reason to invite ruin on older npm clients that are currently working

This argument is void. The introduction of "foo": "^1.0.0" already broke old npm clients. The community was not ruined by this change as far as I can tell.

Contributor

hayes commented May 8, 2014

will the public registry host packages that specify a specific @org in its dependency to be a different registry? (are those settings in the package.json, or are they npm client settings)

Contributor

thomblake commented May 8, 2014

What's the use case for this? Is it just specifically for a particular company to host their secret sauce and never give anyone else access to it? Will they migrate them to the global namespace if they ever want to make those modules public?

Contributor

chrisdickinson commented May 8, 2014

@Raynos An argument to disallow publishing packages that depend on @org packages to the public registry is that, since the binding between @org -> url is local to each machine, packages published with @org deps would be unresolvable by clients who did not share .npmrcs, no matter what version of the npm client they had installed (if I'm understanding this correctly).

Contributor

Raynos commented May 8, 2014

How will publish rights to a module at "@org/foo" work ?

Currently I can just npm owner add mattesch @org/foo ? will the user mattesch then be able to publish to a module at ?

Contributor

chrisdickinson commented May 8, 2014

To clarify some confusion, where does the config mapping between @org and registry.my-org.com live?

  1. The public registry?
  2. A client's .npmrc?
  3. A scoped client .npmrc? (i.e., ~/.npm/@foo/npmrc?)
  4. In the package.json?

And does a missing @org cause an error, or default to the public registry? (from re-reading, I assume "it defaults to public registry", but want to be clear)

Contributor

othiym23 commented May 8, 2014

@chrisdickinson The resolution of an @org to a given registry happens in the client, although the responsibility for validating permissions will probably be scattered throughout the infrastructure. All configuration will be handled using the configuration mechanisms that exist today, although if / once support for multiple / per-project configuration files lands, it might make sense to migrate some of this there. The implication is that @org can map to different registries for different users, depending on their configuration. We have discussed how to consolidate these mappings site-wide for large installations, although we haven't arrived at a final decision on how that's going to work.

Something useful to keep in mind is that the mechanics of resolving packages exists independently from the permissions governing how the server determines whether a given publish is permissible (to speak to @Raynos's point). I'll let Isaac speak to the specifics of how that will work.

Contributor

chrisdickinson commented May 8, 2014

@othiym23 Thanks! The points I was somewhat foggy about were as follows:

  1. Whether @org would exist as an entity outside of private registries, based on @isaacs's comment, "Effectively, orgs are users. You've already "reserved" @Raynos, for example."
  2. If the above is true, whether they would become "the future" of packages published on npm -- i.e., would it become passé to public an unscoped package?
  3. Can dependencies flow from public npm into @org/packages (looks like "no, due to client compatibility", but there might be other concerns there).
  4. Can dependencies flow from @a/x to @b/y (one package in one org depending on another package from another org)?

As I understand it, the @org syntax is kind of like the Host header combined with /etc/hosts -- it gives clients an opportunity to use a different registry, but if not given, it will request whatever the mainline --registry is, while letting that registry know that it wants to request from a certain @org. Given that, my assumption would be that "unscoped modules" would remain the norm, "unscoped public modules" could not depend on @org modules (due to the risk of depending on a private/unresolvable dependency), and @orgs could freely depend on public packages, packages from their own @org, and between different @orgs. Is this correct?

Owner

isaacs commented May 8, 2014

What I don't explicitly see the value in is the @ scoping.

Please see the list of namespace constraints in the OP.

If you are implementing private packages using the existing "private" then a large breaking change is already being introduced since up until now "private" has meant "never publish this thing to any registry ever".

This probably will not use the "private" field in package.json. If it does, it'll only be for new packages, since existing packages can never have a name like @org/name.

Knowing that private packages will in principle cause a large swath of breaking changes for users

Can you elaborate on which breaking changes are planned? I don't see any impact on existing users. They may be left behind if the bulk of development activities move toward namespaced modules, but their existing workflows will keep working.

why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now?

Backwards compatibility.

I'm separately of the opinion that backwards compatibility shouldn't be preserved just because reasons, but then again seeing @ry this week might've just made me nostalgic :)

Noted :)

That's a pretty bold statement, but we of course have our own pieces that are private at Nodejitsu so I can't call the kettle black.

Sorry, I'm not sure which is the bold statement there? Most of what we do at npm, Inc. is developed in the open, and I don't see any reason to change that.

With that in mind, if you are overhauling the auth system we should really consider implementing OAuth2 as a proper way to grant authorization to other apps.

Xylophone. Implementing an OAuth2 provider for npmjs.org is an idea worth exploring, but it's well outside the scope of this discussion.

Aside: Package signing

Xylophone. Package signing is important, but outside the scope of this discussion.

Contributor

othiym23 commented May 8, 2014

Whether @org would exist as an entity outside of private registries, based on @isaacs's comment, "Effectively, orgs are users. You've already "reserved" @Raynos, for example."

On Github, the top-level namespace is the same for both users and organizations. Same deal here. This concept is not exclusive to private registries.

If the above is true, whether they would become "the future" of packages published on npm -- i.e., would it become passé to public an unscoped package?

Given the fact that private packages can depend on public packages, but not vice versa, this is not necessarily the future. As an aside, I find it easiest to think about @orgs as segmenting one namespace, rather than creating multiple top-level namespaces.

Can dependencies flow from public npm into (looks like "no, due to client compatibility", but there might be other concerns there).

If you mean, can package @aoaioxxysz/euterpe depend on request, most assuredly. The other way, no.

Can dependencies flow from to (one package in one org depending on another package from another org)?

Yes.

As I understand it, the @org syntax is kind of like the Host header combined with /etc/hosts -- it gives clients an opportunity to use a different registry, but if not given, it will request whatever the mainline --registry is, while letting that registry know that it wants to request from a certain @org. Given that, my assumption would be that "unscoped modules" would remain the norm, "unscoped public modules" could not depend on @org modules (due to the risk of depending on a private/unresolvable dependency), and @orgs could freely depend on public packages, packages from their own @org, and between different @orgs. Is this correct?

This is correct.

Owner

isaacs commented May 8, 2014

@Raynos

The introduction of "foo": "^1.0.0" already broke old npm clients. The community was not ruined by this change as far as I can tell.

We only switched to ^ as a default once it had been supported in stable releases of Node/npm for almost a year, iirc, and in fact, there was quite a bit of wailing and gnashing of teeth, leading eventually to the recent addition of the --save-prefix config.

The community wasn't ruined, but it was also a much smaller change than this addition, and in retrospect, I would have liked very much to have put more thought into avoiding that issue. In this case, the rule that "global modules can't depend on scoped modules" avoids the problems rather nicely.

How will publish rights to a module at "" work ?

Still working out the best way to accomplish that, but yes, you'll add a user or a group to the package, and then there can be a way to add a user to a specific group.

Part of the purpose of our on-prem beta program is to give us a little petri dish in which to experiment with a few different ways to manage auth and groups. Most likely, there'll be group management by the end of the year in the public registry, and we can have another one of these to discuss that. Until then: xylophone.

@thomblake

What's the use case for this? Is it just specifically for a particular company to host their secret sauce and never give anyone else access to it? Will they migrate them to the global namespace if they ever want to make those modules public?

Scoped modules will be either public or private, at the whim of the owner. I will not expect that they will ever be migrated to the global namespace. This thus satisfies another common request: having simpler/shorter names for packages, scoped to a specific owner, so that you can have require("@company/foo") instead of require("company-foo-but-not-the-foo-for-bar-the-other-for-the-one-for-baz-you-know-for-kids").

@chrisdickinson

I think you're understanding it correctly. Deps can go from scoped -> scoped, scoped -> global, or global -> global, but never global -> scoped.


One downside of the @-sign that I didn't realize until posting this discussion here: we've been spamming Olly Groves (https://github.com/org) quite a lot. I apologize for that, Olly. You may want to ignore this thread entirely, I suspect it will continue.

Contributor

thomblake commented May 8, 2014

Yes writing @foo everywhere might spam lots of people on lots of different platforms (Sorry @foo but you were asking for it)

Contributor

othiym23 commented May 8, 2014

This thus satisfies another common request: having simpler/shorter names for packages, scoped to a specific owner, so that you can have require("") instead of require("company-foo-but-not-the-foo-for-bar-the-other-for-the-one-for-baz-you-know-for-kids").

To amplify this, this makes it easier for people to fork and experiment with existing packages to scratch particular itches without the original package needing to change or grow unnecessarily.

Contributor

chrisdickinson commented May 8, 2014

Can namespaces mutate their privateness?

... to expand:

[16:57:11]  <chrisdickinson>     the other problem is the mutability of organization private-ness
[16:57:50]  <othiym23>   chrisdickinson: care to expand?
[16:58:16]  <chrisdickinson>     I have @chrisdickinson on npm. Folks start do depend on me from their @scoped packages
[16:58:40]  <chrisdickinson>     I make @chrisdickinson private
[16:58:48]  <chrisdickinson>     I've broken all of the @scoped packages.

EDIT: I misunderstood. @othiym23 straightened me out: @org/packages publicly available on registry.npmjs.org will always be public (modulo unpublish); going private means you've (individually) spun up a private registry for @org.

Member

terinjokes commented May 9, 2014

Since an organization can be depended on by other organizations, and because local configuration can change where @org resolves, we'll have to register an npmjs.org account for the organization to ensure we don't accidentally shadow some other organization.

If @org is directed to some other registry, but there are some @org packages that are public, the other registry would need to have both the private and the public packages, as the client won't fallback to the public registry.

Previous discussion within the community seemed to have concluded on scoping packages by registry, not by organization. I want to make sure I'm entirely clear of the impact here.

Contributor

indexzero commented May 9, 2014

Turned into a bit of an essay, but please bear with me. I am not simply disagreeing for the sake of being difficult.

The private mechanism is important here

@izs

Can you elaborate on which breaking changes are planned? I don't see any impact on existing users. They may be left behind if the bulk of development activities move toward namespaced modules, but their existing workflows will keep working.

So if the TL;DR here is "org ns as step towards private modules" it is reasonable to ask what the mechanism for making things private is going to be.

  • Is it going to be a flag in the package.json?
  • Is it going to be done through the npm CLI?
  • Is it going to be done through npm-www?

I (wrongly) assumed that the plan was to repurpose { "private": true } in package.json files which was the breaking change I was referring to.

Why so in-favor of strict backwards compatibility?

@izs

why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now?

Backwards compatibility.

Right now there are 225 public packages which use this syntax in their dependencies (did not check *Depedencies like devDependencies), or roughly 0.320% of all packages (source). So that is not a lot, but it is enough to want to find a backwards compatible way to fix this. That mechanism doesn't have to be as strict as to introduce a new syntax that every new node developer is going to have to learn. What about this flow:

LOAD_AS_ORG(X,Y)
1. Check to see if `X/Y` exists in npm.
2. If not, assume Github repo with `X/Y` and warn the user of deprecation.

It would require formal deprecation of the Github syntax, but it precludes a new syntax. Further, we could use the new @ syntax to replace the existing Github syntax so people can switch when they see the deprecation notice. This is clearly not the list of all consumers of this feature, but it is representatively small. Any interest in surveying users on their existing use of this feature? Have you already?

Based on the anecdotal evidence I've seen organizations would be used 1-2 orders of magnitude more frequently than the existing github shorthand -- most of them use the git:// URL syntax explicitly. So if we could deprecate and replace that seems like a win, not a loss.

Why are we complicating things?

@othiym23 @chrisdickinson @izs

I think you're understanding it correctly. Deps can go from scoped -> scoped, scoped -> global, or global -> global, but never global -> scoped.

I misunderstood. @othiym23 straightened me out: publicly available on registry.npmjs.org will always be public (modulo unpublish); going private means you've (individually) spun up a private registry for @org.

If scoped modules are public by default as @chrisdickinson and @othiym23 said, why the restriction? There is a widely used workflow at companies using Node.js (and even client-side JS) which implicitly conflicts with it:

  1. Company X makes a module, some-thing that they intend to Open Source.
  2. Company X publishes that thing to npm as a private package @company/some-thing
  3. A year later Company X wants to release some-thing as Open Source, but the global some-thing is taken.
  4. Company X is now discouraged from Open Sourcing because of additional breaking config changes in package.json files if they change the name to some-other-thing and making the scoped @company/some-thing Open Source can't be used by global modules.

Xylophone or just not on this issue?

@izs @Raynos

How will publish rights to a module at "" work ?

Still working out the best way to accomplish that, but yes, you'll add a user or a group to the package, and then there can be a way to add a user to a specific group.

Why is this a xylophone? It is hard (if not impossible) to completely assess the intrinsic value of the proposed organizations feature without fully understanding how the publish and user management mechanics will work. e.g. our customers love user sync across their team.

The same goes for OAuth2 and package signing. All of these features are intertwined in nuanced ways: the implementation details matter here. I can move that part of the discussion to another issue if you'd prefer.

Can we count on this module.js "hack"?

@izs @tjfontaine @bmeck

So this thing is a nit, but it seems we got lucky with the @org syntax, because according to the docs it should not exist since the @org directory does not need a package.json file.

LOAD_AS_DIRECTORY(X)
1. If X/package.json is a file,
   a. Parse X/package.json, and look for "main" field.
   b. let M = X + (json main field)
   c. LOAD_AS_FILE(M)
2. If X/index.js is a file, load X/index.js as JavaScript text.  STOP
3. If X/index.node is a file, load X/index.node as binary addon.  STOP
//
// ... ? We are here, in undefined territory.
//

module.js in core hasn't changed in almost six months, but I know that bundling into a single executable was on @tjfontaine's radar for node@0.14 which will require changes to the module loader. Summoning him and @bmeck here to see if that has any bearing on this discussion.

What is a "reference implementation"?

@izs

We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic.

Sorry, I'm not sure which is the bold statement there? Most of what we do at npm, Inc. is developed in the open, and I don't see any reason to change that.

What's bold is that you're saying the OSS npm registry is going to become a "reference implementation", which won't be run in production by npm Inc. Please forgive me if I am misinterpreting this, but I've always read the phrase "reference implementation" in the past as "a toy for hobbyists". Is that the intention here?

Part of the purpose of our on-prem beta program is to give us a little petri dish in which to experiment with a few different ways to manage auth and groups. Most likely, there'll be group management by the end of the year in the public registry, and we can have another one of these to discuss that.

Could you clarify the above? Is there an implementation of these features somewhere that is not yet public? Sorry for prying, but your initial segmentation about what would be private and what would be public was rather nebulous at best and I know I'm not the only one who would like a clearer line drawn in the sand.

Contributor

bmeck commented May 9, 2014

Just to clarify, people want to change require to include a scope mechanism as well? This would be pretty major since it could break node_modules as a folder if you do have 2 modules with the same name but different scopes.

If we do use scope in require I would avoid using a specialized string in require. Keeping it separated ala require.scope(org)(module_name) or require.scoped(org, module_name) (idc syntax, just separated) since collisions can still occur if only module_name is present unless we want to continuously add new string syntax to require.

Either way, I would need more info on what is actually being proposed to be put onto disk to say much more. This would not interfere with archives or bundling most likely since that just reuses the require mechanisms.

Contributor

Raynos commented May 9, 2014

Based on the anecdotal evidence I've seen organizations would be used 1-2 orders of magnitude more frequently than the existing github shorthand -- most of them use the git:// URL syntax explicitly. So if we could deprecate and replace that seems like a win, not a loss.

Remember that npm i flatiron/winston --save expands the github uri to a git link and stores the git link in the package.json.

I believe npm install is far more common then manually writing git links in package.json

I don't know actual numbers though

Contributor

indexzero commented May 9, 2014

@Raynos

Based on the anecdotal evidence I've seen organizations would be used 1-2 orders of magnitude more frequently than the existing github shorthand -- most of them use the git:// URL syntax explicitly. So if we could deprecate and replace that seems like a win, not a loss.

Remember that npm i flatiron/winston --save expands the github uri to a git link and stores the git link in the package.json.

I believe npm install is far more common then manually writing git links in package.json

This I did not know. Thank you for the clarification. Digging into that further the most important reason for backwards compatibility should be changing the behavior of packages already published to the npm registry.

The flow I suggested falling back to git:// URLs works even better here because there is nothing to be backwards compatible with. We simply need to make consumers of the CLI aware of the change. Crunching the numbers for usage of git:// URLs in packages published would be less representative since they could have been added explicitly without the CLI syntax you mentioned.

Contributor

indexzero commented May 9, 2014

@bmeck the syntax being proposed here works as is today which was a bit of a head turner for me. e.g. consider:

module/
  index.js
  node_modules/
    @org/
      winston
  package.json

where your index.js is:

var winston = require('@org/winston');
winston.info('wat');

The above will just work. I had to verify it to believe it since it isn't documented officially. Any bundling should consider these folders or use Module._resolveFilename and/or Module._resolveLookupPaths when traversing require statements. Probably xylophone though to stay on the meat of the discussion here.

Contributor

chrisdickinson commented May 9, 2014

re: complicating things.

If scoped modules are public by default as @chrisdickinson and @othiym23 said, why the restriction?

My primary issue with allowing pkgA depends-on @org/pkgB is the potential for the client (given this scheme) to be talking about a different @org/pkgB than is available on the target default registry -- or referring to one that is not present on the target default registry. This opens up the public namespace to pollution with packages that either cannot successfully install or do not work correctly upon installation. Disallowing reentrance strengthens the assurance that the packages in the public namespace are installable (disregarding existing git:// links to private repos / broken http://tar.gz deps).

There is a widely used workflow at companies using Node.js (and even client-side JS) which implicitly conflicts with it: [...]

It also puts up guard rails keeping users from bad practices in this regard: if A depends-on B depends-on C, open source C, then B, then A -- not B first. Put another way, it keeps users from accidentally making public a package which cannot be installed by any user outside of their company.

re: the privacy mechanism:

So if the TL;DR here is "org ns as step towards private modules" it is reasonable to ask what the mechanism for making things private is going to be.

Is it going to be a flag in the package.json?
Is it going to be done through the npm CLI?
Is it going to be done through npm-www?

As I understand it (and to recap and make sure we're on the same page):

  1. Namespaces are a flex point for the npm client to override what registry it is referring to.
  2. Dependencies can only flow into them and between them -- non namespaced packages can never depend on a namespaced package.
  3. Clients may override the mapping between namespace & registry url. Find $REGISTRY:
    1. If the registry url is overridden locally [*]:
      1. ... Use the overridden registry url.
    2. Otherwise,
      1. Use --registry=<url>, if present.
      2. Use NPM_REGISTRY env var, if present.
      3. Use .npmrc's registry setting, if present.
      4. ... Eventually fall back to registry.npmjs.org.
  4. Packages are resolved at $REGISTRY using name/version, where name is the full name of the dep (including the @org/). Allow $REGISTRY to do any special resolution based on @org/.

This leads me to believe that the privacy setting is the overridden @org -> registry.url mapping on the client (which might increase the risk of accidental publishes?)

Alternatively, it's the combination of the client's org/url mapping and the default registry's ACLs for that @org -- i.e., does user X have access to publish package Y on @org Z? Since packages are always requested from the server with @org/ included, the server has a chance to make its own determinations there.

Personally, I'd be much more comfortable with the alternative situation -- being able to flip a switch on registry.npmjs.org and say "nope, no public packages from @urbanairship please" rather than the (error-prone) process of making sure every developer always has their npmrc correctly configured to prevent accidental publishing.

[*]: Leaving this vague as I'm not sure if it's going to be .npm/@org/npmrc or NPM_REGISTRY_org or ... really at all how that mapping will be stored.

What if we want to add a shortcut for Bitbucket-repos?

What about this Syntax:

npm install github user/repo
npm install bitbucket user/repo
npm install organization user/module

Then throw them in like this:

node_modules/
    module1
    module2
    @github/
        user/
            repo
    @bitbucket/
        user/
            repo
    @organization/
        user/
            module

maxogden commented May 9, 2014

if you are interested in providing feedback on this issue you may also be interested in feedback on https://normalize.github.io/ (https://github.com/normalize/discussions/issues/3)

Member

domenic commented May 9, 2014

I like @org/package much more than org/package, independent of any backward-compat issues. The @ sign is a nice differentiator. E.g. require("@kriskowal/q") versus require("q/queue"): the former is Kris's Q package; the latter is the queue module inside the global q package.

So I'm a little surprised you're spending so much time arguing for org/name, @indexzero. It doesn't really have any upsides besides one less character to type.

Contributor

indexzero commented May 9, 2014

re: complicating things

@chrisdickinson

Your technical argument has merits, but you did not respond to the cultural or incentive argument. The community goal has always been (as I understood it) "don't slow down growth". Adding extra steps to the publish ceremony water dance that developers must do just because they work at a company and want take advantage of this feature will discourage them from using it.

The fallback mechanism that allows for global packages to depend on scoped packages is actually quite simple:

INSTALL_SCOPED_MODULE(org, module)
1. Does REGISTRY configuration exist for "org"?
  - YES: Attempt to fetch "org/module" from that registry
    - 200 OK. STOP. Module installed.
    - 404 Not found. Attempt to fetch "org/module" from "default" registry or registry.npmjs.org 
      - 200 OK.        STOP. Module installed.
      - 404 Not found. STOP. Module not found.
  - NO: Attempt to fetch "org/module" from "default" registry or registry.npmjs.org 
    - 200 OK.        STOP. Module installed.
    - 404 Not found. STOP. Module not found.    

It also puts up guard rails keeping users from bad practices in this regard: if A depends-on B depends-on C, open source C, then B, then A -- not B first. Put another way, it keeps users from accidentally making public a package which cannot be installed by any user outside of their company.

This is where I'm getting really confused, especially w.r.t. to multiple registries. Consider this scenario:

  1. I am a complete n00b
  2. I think up a name for an org, like @imanawesomenoob
  3. I run
cd /iamanawesomenoob
cat package.json | grep name
"name": "@iamanawesomenoob/some-module"
npm publish

Where does it live? And more importantly: where does it live it I decide to start making it private? Does it only exist privately if I run my own private registry? Or is there a place where it is hosted?

re: privacy mechanism

@chrisdickinson

Personally, I'd be much more comfortable with the alternative situation -- being able to flip a switch on registry.npmjs.org and say "nope, no public packages from @urbanairship please" rather than the (error-prone) process of making sure every developer always has their npmrc correctly configured to prevent accidental publishing.

Yes. I agree with this as well, which is why our private npm product default all publishes to private to avoid this behavior. As wonderful and amazing and fulfilling as writing Open Source is, I believe that the number of private modules dwarfs the number public modules by at least one order of magnitude. This anecdotally makes sense when you consider that every application is, in fact, a module. So in the "paid and corporate-y" world, the scenario of accidentally publishing a private module public is more likely than accidentally publishing a public module private. Optimizing for this fact would cause much rejoicing of future developers.

re: scoped modules vs. sub-modules

@domenic

I like @org/package much more than org/package, independent of any backward-compat issues. The @ sign is a nice differentiator. E.g. require("@kriskowal/q") versus require("q/queue"): the former is Kris's Q package; the latter is the queue module inside the global q package.

That is a very good point that I simply had not thought about. So good it should be in the list of "Namespace constraints" in the OP:

  1. It should be easily differentiable from "submodule syntax". That is, require("org/module") should be easy to understand when compared to require("global-module/sub-module-path")

My original argument was counter-pointing the argument for backwards compatibility of a feature that did not appear to be used heavily. Your point makes that moot because disambiguation of what's being required is much more important than backwards compatibility.

Owner

isaacs commented May 9, 2014

Re "Reference Implementation" vs "Toy for Hobbyists"

A reference implementation is not necessarily low-quality. In fact, being a low-quality implementation can reduce its value as a reference!

Both production and reference implementations benefit by being highly performant and easy to use. However, for a reference implementation, "ease of use" is prioritized over "high performance", whereas in a production implementation, the opposite is the case.

The public npm registry has not been a single couchapp exposed to the world since fall of 2013. Today, it is a write-master, several read-only replicas, a CDN, and a static file server, with a bunch of various workers and rewrites and automation keeping it all together. The npm-registry-couchapp piece is important, but it's not as if the production implementation and reference implementation necessarily need to be identical.

A reference implementation is a specification that can answer questions you didn't think of ahead of time. I believe that empowering innovation has massive long-term upsides, even if some of that innovation might happen in the heads of potential competitors. So, we have decided to do this development in the open, including making the standalone reference implementation include more than just a couchapp. This means it might be a bit harder to set up and use, but it will also be a better reference. Tradeoffs.

My primary issue with allowing pkgA depends-on is the potential for the client (given this scheme) to be talking about a different than is available on the target default registry

While you're right that this is still a problem, it's a much smaller problem than the one we have today. An organization will likely have only one name, but many modules. Requiring that ALL of their modules don't conflict with anything public is much more hazardous than requiring that their organization name doesn't conflict with anything public. How many "paypal"s or "yahoo"s are there in the world?

Re require.scope("@org") or some such:

We won't be modifying node's module system. But, it might be possible to do something like this in a userland module. Node's module system is actually quite well documented, and everything proposed here works today. Try it!

I believe npm install is far more common then manually writing git links in package.json

Indeed, most people using this for private modules are not publishing them to the registry. Your website depends on company/foo and company/bar which are both private. You don't publish that to the npm registry.

I've seen a lot of stuff that depends on joyent/private-thing when I worked at Joyent. None of it was published to the registry, though.

Several people have brought up the problems with deprecating and re-purposing the GitHub shorthand syntax. That's not going to happen.

Re "path to dev private, open later":

The sequence will not be to work on @org/foo in private, and then publish it as foo.

The sequence will be to work on @org/foo as a private module, and then make @org/foo public.

Being scoped does not mean that a package is necessarily private. The organization owner will have the option to make packages public or restricted.

This has been repeated several times in this thread.

Many of the mechanisms for this are still TBD, and there may well be competing implementations.

The thing about "open development" is that you get to see stuff before all the questions are answered. One might say that's the only thing about open development, in fact! Expect some partly-baked experimentation.

In the interest of keeping this conversation useful for newcomers, and not them scaring away with an even more impenetrable wall of text, any further messages that reintroduce already-answered questions may be deleted or edited at the discretion of the npm project maintainers. Thanks.

Owner

isaacs commented May 9, 2014

The fallback mechanism that allows for global packages to depend on scoped packages is actually quite simple:

We're not going to send 404-ing requests to the main registry. Doing that would be an information disclosure vulnerability.

Where does it live? And more importantly: where does it live it I decide to start making it private? Does it only exist privately if I run my own private registry? Or is there a place where it is hosted?

It'd be published on the public npm registry. As mentioned above, the specific mechanisms for how that'll be handled is still TBD.

Scoped modules are "a step towards private modules and multiple registry management". They're not "a private module solution".

It may be that (org)name would work as well @org/name, and avoid discussions of those modules on github getting emailed to random users.

I think I missed how this will work with private (probably temporary) forks of a public module. Say you find a bug in the module wonderbar, so you github fork it to your organization, patch it, PR the author, and while waiting for the fix to land on npmjs.org, you publish it to your private registry, npm.mycompany.com.

You don't change its name, and don't want to (you have dozens of modules depending on wonderbar, and don't want to go through and hack up all their package.json, and then unhack them once the PR is accepted and wonderbar is republished, and anyhow, if you were going to do that you could just hack them to use github refs, like somecompany/wonderbar, but github refs suck for many reasons already discussed).

Currently, to make this work, you point your default registry at your private registry (registry.mycompany.com), and probably arrange to have the private reg forward requests for unknown modules to npmjs.org (there's a handful of ways and registry implementations that do this)

With the proposed change... you still have to do this, right? The "@" syntax may be mapped in your local .npmrc to say "use registry npm.mycompany.com for packages like @mycompany/someprivatething", but that doesn't help for internal forks of public modules.

It's possible that the above use case is out of scope for scoped modules, but do I miss something?

Scoped modules help people to have modules that are entirely private, and not risk namespace collision, and configure a private registry (if they want) to host packages with a specific scope?

But scoped modules don't deal with temporary forks of public modules, that still requires a private FORWARDING registry to be set up? (or github refs)

Contributor

davglass commented May 9, 2014

I'm interested in this as well and I like the @org/foo syntax. It looks clean and it's easy to grok.

My only issue with the @ & the / would be to test it on the CLI, especially on windows, and from shell scripts to ensure that the character is escaped properly and it doesn't break if you are installing CLI tools via: npm -g install @org/foo@1.2.2.

Looking at that last command makes it look a little funny now that I typed it.

Just to put a little perspective on it from my side, we've had our internal registry for a few years now. We have over 700 internal modules in our registry and we only have 4 that have name collisions. And that's simply because those modules were created before the open source modules were. We also don't allow developers to publish directly to the registry, our CI/CD system handles that.

I'm not saying that we wouldn't use it, I like it, I'm just saying that it's not been that big of an issue for us. In fact, I've actually used the single naming in the past to publish an internal module that fixes a broken or mis-configured external module so my devs could continue to work.

We've also seen the issues of working off of patched forks while we wait for the developer to accept our PR and publish a new version. Doing that with @org/ prefixes will be a PITA.

Not to mention that while typing this, I almost included several people in this thread due to the @ sign, so there's that too :)

Contributor

chrisdickinson commented May 9, 2014

@indexzero: You'll have to forgive me, I'm answering out of order here:

Where does it live?

If you don't have any custom mappings locally, it will live as a public package on the public registry.

Does it only exist privately if I run my own private registry? Or is there a place where it is hosted?

The behavior of the public npm registry w/r/t to privacy/package visibility isn't fully defined yet (outside of support for publicly visible @org packages).

That means that to make a package private, a client may add a mapping for @org to a private registry in their .npmrc. There may be support on whatever default registry (npmjs, nodejitsu) eventually for setting visibility of an @org, but that's not part of this issue (as I understand it!)

And more importantly: where does it live it I decide to start making it private?

Anything you published to public registry already will remain public. Anything published after the client has added the mapping between @org and a private registry url will be private (or at least, only available at that other registry).

Member

terinjokes commented May 9, 2014

The sequence will not be to work on in private, and then publish it as foo.

The sequence will be to work on as a private module, and then make public.

If we're assuming that the majority of open source plugins are going to be in the global space—which so far we have—this seems very wrong to me. We're effectively saying that @org/foo, even if open source and public, shall have no value to other open source modules.

Echoing @davglass, while we don't have 700 modules—we're quickly approaching 100—we haven't had any issues with naming conflicts between our private modules and public ones, excepting in the case where we explicitly want to fix modules somewhere deep in our tree while we wait on the maintainer to fix. This is a very real non-temporary issue for us.

Contributor

thomblake commented May 9, 2014

If we're assuming that the majority of open source plugins are going to be in the global space—which so far we have—this seems very wrong to me. We're effectively saying that , even if open source and public, shall have no value to other open source modules.

That's a good point. Either global modules should be allowed to depend on @org/foo, or making new global modules should be deprecated going forward so that new modules can freely depend on new open-source namespaced modules.

sbisbee commented May 9, 2014

I'm +1 on deprecating global namespaces and requiring that all packages live under a top level name space. That top level name space could be used as a vendor/org, username, etc. Generally this is a requirement or best practice in other languages that have used name spacing. Strictly speaking the NPM implementation doesn't have to care whether it's a user or an organization.

This also removes the above sub-module reference problem because sub-module references would become, for example, require('kriskowal/q/queue'); Any reference to require('q/queue'); would reference the q organization/user's queue package.

Users who wish to maintain authorship in the name space can continue to do so. Those who wish authorship to remain in the copyright or contributors file, instead conveying ownership/publisher in the namespace, can do so as well.

Lastly, it makes policy writing dead simple for anyone writing a NPM server, proxy, cache, client, etc. who wants to clearly dictate the mobility of an entire organization's namespace.

Cheers.

Contributor

othiym23 commented May 9, 2014

Either global modules should be allowed to depend on , or making new global modules should be deprecated going forward so that new modules can freely depend on new open-source namespaced modules.

I think this is a problem that will solve itself over time. It allows the current module system to remain forward-compatible, and creates a cowpath by which new modules will gradually adopt the new system. Assuming the registries and clients all enforce the rules, it requires very little else be done by developers, which makes it feel like a pretty natural evolution of the system.

I'm curious about the implications this will have on the eventuality of a module being abandoned by its original author and subsequently adopted by another. Maybe this has already been discussed, but I'm too lazy to dig through the bikeshedding.

What will the process of transferring authorship look like in the future of namespaced modules?

Contributor

othiym23 commented May 9, 2014

@nexxy I don't think much has changed (yet?). Ownership transfer of modules without org prefixes will continue the way it always has. If somebody wants to take over development of a prefixed module, they could just publish it under their organization. There's still only the one namespace, it's just segmented differently.

That makes sense for 'non-prefixed' modules.

I'm more curious about what happens when 'org authorship' changes. Does become upon authorship transfer? Will there be some sort of 'symlink'? How will org-module consumers know when this happens?

Maybe this kind of thing won't happen that often, but it sounds like it could be confusing and potentially cause sudden breakage.

Owner

isaacs commented May 9, 2014

@sam-github

$ npm install (org)name
bash: syntax error near unexpected token `('

No dice, sorry.

@thomblake

Either global modules should be allowed to depend on , or making new global modules should be deprecated going forward so that new modules can freely depend on new open-source namespaced modules.

Yeah, we can re-evaluate that a while down the road, say in a few years or so. I think we may end up opening the dependency graph to go in both directions, and/or perhaps closing the door on new global modules. But neither thing will happen soon, and by the time it does, we'll have lots more information at our disposal.

@davglass

Yes, this does not address the "override xyz module for our bugfix" issue. However, it also doesn't make it any harder to do that, if you have a private registry. It will prevent those rare collision cases from happening, even if you use very generic names like logrotate or markdown or querystring, without having to publish via a CI.

My hope is that we finally put an end once and for all to the "all the good names are taken!" complaints :)

Also, keep in mind, if you're running a private registry already, you might not need this feature, or it might basically change nothing for your use case. But for a lot of other users who want namespaced modules, and especially those who may want private modules without running an entirely separate registry, or want to run a private registry for a few modules but not for all of them, we hope this will be a useful tool.

@nexxy

Good questions! We hadn't thought about that!

Does become upon authorship transfer?

Yes.

Will there be some sort of 'symlink'?

301 Redirect, yes.

How will org-module consumers know when this happens?

Maybe the client can print out a warning when this happens?

Contributor

othiym23 commented May 9, 2014

Does become upon authorship transfer?

  1. Person at @thisonefancyorg expresses an intent to transfer module to @someotherfancyorg.
  2. Admins of @someotherfancyorg receive a message that the transfer has been requested.
  3. @someotherfancyorg admins signal their intent to accept the transfer.
  4. Hej presto! The module belongs to @someotherfancyorg!

Will there be some sort of 'symlink'?

This should work at least as smoothly as GitHub does it today. Requests to the module at its original location are redirected to the module at its new location.

How will org-module consumers know when this happens?

¯\(º_o)/¯ iono! Probably some mixture of the npm CLI silently rewriting to package.json when --save is used and printing out warnings on install.

I'm also not sure I fully understand the benefits of the additional complexity.

If orgs will be maintaining their own registry anyway (I assume?); what does this schema provide over the approach of simply mapping 'overrides' to global modules in the rare instance that it's necessary? Private modules are already available via private org registry without any namespace.

The only missing component I can identify is the ability to pull from multiple registries. That seems like something adequately solved by proxying requests for unavailable modules to a 'global' registry (one might even cache versions; faster on subsequent installs, and saving bandwidth).

Maybe I just don't understand the nature of enterprise-y needs well enough (strong possibility), but the more I read about the proposed solutions; the more I feel like there is a simpler way...

Unless this is a lead-up to the ACL/roles stuff mentioned on the original post; in which case I understand the need for these changes.

@othiym23 @isaacs thanks. these things make sense within the context of the proposed changes.

I also notice you mention orgs that are already running their own registry, which makes me wonder if I assumed the right things.

Is this to say that orgs which want their own private modules, but don't already host their own registry will be able to do so via an NPM-provided facility? That would make sense, I suppose! I guess this is where the mapping of @org -> org.something.like.whatever.npm-registry.com comes in?

Contributor

othiym23 commented May 10, 2014

Is this to say that orgs which want their own private modules, but don't already host their own registry will be able to do so via an NPM-provided facility?

Yes. The first full implementation of this will be an on-premise registry appliance-type thing.

I guess this is where the mapping of @someorg -> org.something.like.whatever.npm-registry.com comes in?

Yes. The mapping from @someorgorg.something.registry.whatever.org will be set in client configuration (perhaps in a site-wide way for larger installations).

Yes. The first full implementation of this will be an on-premise registry appliance-type thing.

very fancy indeed. I support this endeavor; particularly if you make them look like little toasters.

carry on then!

Contributor

thomblake commented May 10, 2014

+1 to toasters
On May 9, 2014 5:37 PM, "Emily Rose" notifications@github.com wrote:

Yes. The first full implementation of this will be an on-premise registry
appliance-type thing.

very fancy indeed. I support this endeavor; particularly if you make them
look like little toasters.

carry on then!


Reply to this email directly or view it on GitHubhttps://github.com/npm/npm/issues/5239#issuecomment-42726525
.

Owner

isaacs commented May 10, 2014

If orgs will be maintaining their own registry anyway (I assume?)

You assume incorrectly! Many orgs will not be maintaining their own registry.

We spent a few months talking to a lot of companies that use Node about how they're using Node. There is a pretty sharp split between "We will never host our code in your registry" and "Please don't make us spin up another thing, just shut up and take our money". (Similar to the companies that use GH Enterprise vs those that use GH orgs and private repos.)

Every company we talked to both (a) fell into one of those buckets, and (b) didn't think it was reasonable to use the other. And the split was about 50/50. I suspect that smaller shops and individuals will be even more strongly on the "I will never run a private registry" side of the divide.

The only missing component I can identify is the ability to pull from multiple registries. That seems like something adequately solved by proxying requests for unavailable modules to a 'global' registry (one might even cache versions; faster on subsequent installs, and saving bandwidth).

You still have a single registry in that case, just one that gets some of its data from registry.npmjs.org.

Unless you mean that the client sends 404-ing requests to the public registry, but that's an information disclosure risk, and also much more complexity in practice than the "additional complexity" you're referring to.

This is indeed a lead-up to the ACL/roles stuff, as well as a way for organizations and users to have simple names that are namespaced to their owner, which has been a very frequent request over the years.

Owner

isaacs commented May 10, 2014

I keep getting scooped by @othiym23 who's answering questions very well. I will stop multitasking and focus on the other things I should be doing :)

@isaacs all of that sounds quite reasonable. thanks for taking the time to explain!

I will let you get back to the toasters.

Owner

isaacs commented May 13, 2014

Also, devs at this NSA company will be able to put a npmrc file in the root of all their projects that defines the internal registry location, or even pre-configure their devs' computers with this config pre-installed.

"Misconfigured client" is a very solvable problem. "Typoed the package name" is not. If the client directs 404s to the public registry, then that's terrible.

If you're currently doing this in your product, I'd suggest that's a pretty significant bug you ought to disclose to your customers.

Contributor

mantoni commented May 13, 2014

From within the NSA, would I set the package name to "" and global npm module dependencies would be "mochify"?

Or would I set the package name to "secret-project", have my default registry set to the internal one, and name my global dependencies ""?

I would find it slightly confusing if I name my project "secret-project", npm publish it under that name, but have to add a dependency named "". That is likely to go wrong and lead to the described information disclosure problem.

Contributor

indexzero commented May 13, 2014

Also, devs at this NSA company will be able to put a npmrc file in the root of all their projects that defines the internal registry location, or even pre-configure their devs' computers with this config pre-installed.

+1 to this and deprecating publishConfig. Solving config problem(s) using a dotfile makes much more sense.

"Misconfigured client" is a very solvable problem. "Typoed the package name" is not. If the client directs 404s to the public registry, then that's terrible.

To be clear there is no information disclosure vulnerability in a properly configured npm client using the mechanism suggested. In this perfect world the private registry config for @org is always set properly and the requests to the public registry on 404 are intended, not an unintended disclosure of any kind.

It's tradeoffs. Perhaps hundreds of bad 404s would be made. Hundreds out of billions of total requests. Yes, the information is there, but it's a needle in a haystack and the information itself is not that sensitive. It is not code: it is knowing that @org has a module named foo. Not even where the real private registry that holds @org/foo is. This of course presumes that the npm client drops the Authorization headers on your GET requests to the public registry (which is a must here regardless imho).

Beyond that: this information disclosure will happen one way or another. Even the best of the best Node.js developers make fat fingered mistakes pretty frequently. publishConfig helps, but makes projects rigid when you switch registries.

The upside of this calculated risk is that scoped modules and existing global modules can live side-by-side peacefully depending on one-another and generally not confusing people. It's a risk that is highly worth it. What are the thoughts on syntax that explicitly says a module is private or public? e.g. git+ssh:// already tells the npm client with some certainty that "hey, this thing is secured." What about either of these:

{
  "dependencies": {
    "@NSA/prism": "private ^1.2.3",
    "@NSA/prism": "p ^1.2.3"
  }
} 

Making an explicit distinction between "scoped public modules" and "scoped private modules" may be just the thing here.

Owner

isaacs commented May 13, 2014

In this perfect world the private registry config for @org is always set properly and the requests to the public registry on 404 are intended, not an unintended disclosure of any kind.

I'm bound by NDA to not say too much. But, just to be 100% clear on this: I have personally spoken with technical decision makers at real companies who have told me that they would be uncomfortable if the client "fell back" to a public registry for 404s. If you say "@org/..." modules should be hosted in your private registry, and a user types @org/foo instead of @org/fop, then it should simply fail, rather than tell npm, Inc. that @org exists, does not have a module named foo, and probably does have a module named something close to foo.

So, we'd have to provide a clear way to opt-out, as well as disclose it. And frankly, building something that adds complexity and isn't desired by most users is really not something I'm too keen on anyway.

Your suggestion has been heard, the problems elaborated by me and others. If you don't have new information to provide, let's close that thread and move on.

Owner

isaacs commented May 13, 2014

@mantoni

From within the NSA, would I set the package name to "" and global npm module dependencies would be "mochify"?

Yes, scoped modules can depend on global modules.

Or would I set the package name to "secret-project", have my default registry set to the internal one, and name my global dependencies ""?

No, I think introducing a "magic" namespace called "@global" is unnecessary complexity.

I would find it slightly confusing if I name my project "secret-project", npm publish it under that name, but have to add a dependency named "". That is likely to go wrong and lead to the described information disclosure problem.

In the proposed scheme, a non-scoped module like "secret-project" woudln't be able to depend on a scoped dependency like "@nsa/secret-project"

@isaacs I sympathize with the desire to not query npm, inc's registries with "" when it isn't found on the private reg configured to be used for "@nsa", we hear similar things from clients. But if the desire is to not leak info to Npm.inc on accidental mis-config/mis-type, I don't think you've achieved that, you only plugged one hole (misnamed modules).

If I forget to configure "@nsa" to go to nsa's registry (unlikely on a dev machine, more likely on vms), and npm install an app that uses "@nsa" scoped modules, it will send every single dependency's name to Npm,inc. Oops.

Obvious way to fix this is to not send any queries to a registry unless it is configured to serve that scope.

This would conflict with making it easy to publish/subscribe scoped modules to npmjs.org.

Seems like a requirements conflict to me.

Owner

isaacs commented May 13, 2014

@sam-github It's plugging the hole we CAN plug, which incidentally also the much more difficult hole for the client to fix on their end. Human fingers will do the wrong things repeatedly. But dev environments and package folders can be deterministically configured with a high degree of reliability.

And for the truly paranoid, they would be able to use npm://nsa-registry.local/@nsa/fop to refer to the package, which is what @nsa/fop would desugar to anyhow. So, as long as the client doesn't try to be overly clever in that case, there's zero risk.

Contributor

mantoni commented May 13, 2014

Thanks for the explanation @isaacs.

From my perspective from within an "enterprise" environment (and a potential npm customer), I would prefer it to be the other way around. Let me explain in more detail.

The @org feature, as discussed so far, is used for two things:

  • namespacing
  • registry routing

I think it's going to cause quite some confusion if @namespace and @repo are used side by side in the dependencies. One pointing to a namespace in the central repo, the other one being private.

If we take a step back and think about only doing the routing, we could have a much simpler and clearer solution:

  • "my-project" refers to whatever my default repository is
  • "" refers to whatever "repo" is configured to be

Looking at our projects, we have 90% private modules, and they must stay within the company network. It would feel natural to only mark the "external" dependencies. If there is a need for another department within the company to have their own repository, that would work as well.

We already want to have two synchronized repositories for performance, deployed in different locations. They would have different URLs, but the contents are shared. We could solve that by having different default repo configs per region and make @npm point to the central.

Does that make any sense? Or am I getting it all wrong?

Contributor

glenjamin commented May 13, 2014

This seems like an interesting slant on the problem, I think it's worth noting again at this point that we have two viable syntax choices, perhaps using one each for these two features might be a good approach?

Unsure if that makes things more complicated, or actually does manage to decomplect the concepts of private registries and namespaces.

Contributor

indexzero commented May 14, 2014

I'm bound by NDA to not say too much. But, just to be 100% clear on this: I have personally spoken with technical decision makers at real companies who have told me that they would be uncomfortable if the client "fell back" to a public registry for 404s.

So, let me see if I follow you. Companies that willfully disclosed to you and npm Inc. that they are using node and npm are concerned about disclosing this information to you again in the future? I hope you can see how two dimes and a nickel aren't making a quarter on this here.

So, we'd have to provide a clear way to opt-out, as well as disclose it. And frankly, building something that adds complexity and isn't desired by most users is really not something I'm too keen on anyway.

Philosophically "the needs of the many outweigh the needs of the few" is easy to agree on, but are we sure that this is the need of the many? There are only ~20 people on this comment thread, many of whom (like myself) do desire this. How do we know "global can depend on scoped modules" is not desired by most users? I'm happy to help get the word out and try to distill a public opinion on this. This survey only got 13 responses, but that was only a single tweet looking for answers.

Search for omnipresence aside, making this opt-in could meet mutual goals here. Module consumers who wish to use global modules that depend on scoped modules would be forced to enable it, so it would be up to the module authors (as it always has been) to decide whether or not this is a worthwhile feature to them.

pmuellr commented May 15, 2014

Rather than try to overload package names, has there been any consideration to re-archicture of package.json dependencies and dev/optional/bundled/peer/whatsNext?-Dependencies??

as a off-the-cuff example of changing the shape ...

{
  "repos": {
    "myCo": "http://npmjs.myco.wtf",
  },

  "deps": {
    "runtime": {
      "foo": "0.1.x",
      "bar": "myCo:0.2.x"
    },
    "dev": {
      "blortz": "0.3.x",
      "pinky": "myCo:0.4.x"
    }
  }
}

Seems like it would be straight-forward-ish to prototype this with an install script in the package.json

Owner

isaacs commented May 15, 2014

@mantoni

The @org feature, as discussed so far, is used for two things:

namespacing
registry routing

I think it's going to cause quite some confusion if @namespace and @repo are used side by side in the dependencies. One pointing to a namespace in the central repo, the other one being private.

I disagree. @org/pkg is for namespacing only. A property of a namespace is that you can specify which registry it lives in, but multiple namespaces can live in a single registry. The top-level --registry is then effectively specifying just the home for the global modules, and namespaces that do not have some other home specified.

Remember, @org/pkg doesn't expand to https://registry.my-org.com/pkg, but rather https://registry.my-org.com/@org/pkg. So, you can still have multiple different organizational namespaces within a single internal registry.

Looking at our projects, we have 90% private modules, and they must stay within the company network. It would feel natural to only mark the "external" dependencies.

What you may find is that non-namespaced modules start to stand out after a while. Already, I have a ton of npm-this and npm-that. I'm starting to come around to @visionmedia's pov that it'd be nice to just have @npm/this and @npm/that and then be able to manage and look at them more easily as a single group of things.

@glenjamin Using one syntax for namespacing, and another for registry location, does indeed increase complexity. There will already be a syntax for specifying the registry location, using npm://registry.my-org.com/@foo/bar, if you want to use that.

Owner

isaacs commented May 15, 2014

@indexzero

So, let me see if I follow you. Companies that willfully disclosed to you and npm Inc. that they are using node and npm are concerned about disclosing this information to you again in the future?

You are not following me.

Companies that willfully disclosed to me, consciously and with careful foresight and executive involvement, while I was under NDA that they are using Node and npm. They are concerned about disclosing additional information about the details of their private use of Node, via automated and uncontrolled mechanisms, without executive oversight or legal protection.

That subject is closed. Thanks for your suggestion. It's been explored, the problems elaborated, and it has been rejected on the basis of those problems.

If you want to explore doing that kind of behavior, you can build your own registry implementation which does exactly what you are describing (either via a 302 response, or a proxy), without changing the client at all. In fact, I believe you already have. So, I'm not sure what there is to even discuss. You asked a question, and got an answer.

Regarding the second suggestion, that we repurpose foo/bar instead of @foo/bar for the use of scoped modules:

The issue isn't just backwards compatibility, either. We benefit by making sure that the new thing works and fails crisply when it can't work, rather than doing a different thing, and encountering hazards when users of new-npm (where foo/bar is a scoped module) try to consume package.json files created by users of old-npm (where foo/bar is a github repo). That's just a recipe for disaster.

We may break backwards compatibility if there are clear benefits, but in this case, the benefits are much less than the costs. In fact, I don't even really see what the benefits are, except for typing the @ sign. So, no, not going to happen.

At this point, continuing to restate settled issues is approaching trolling. Your points have been made. Please make new points, or just let it rest.

Owner

isaacs commented May 15, 2014

@pmuellr

Rather than try to overload package names, has there been any consideration to re-archicture of package.json dependencies and dev/optional/bundled/peer/whatsNext?-Dependencies??

We did discuss that internally, but it hasn't been brought up yet in this thread.

There are some advantages to adding to the package name namespace rather than expanding package.json shape.

  1. We don't have to teach people as much new stuff. The existing pattern of "you put X in package.json dependencies, you do require(X), and npm install X installs it" continues to hold. That's a pattern that has proven successful.
  2. It's a lot less surgery in npm, the registry data validation scripts, etc.

That's not to say that there isn't a use case for re-thinking some of the things in the package.json data structures, but moving that stuff around is hazardous and in this case, not the simplest way to achieve the desired goal.

Owner

isaacs commented May 15, 2014

@pmuellr Another advantage of putting this data in the cli and require() statements explicitly is that it's hinted at a much higher "level" in a development process. Rather than edit package.json to add a second repo, and then do npm install foo, and know that foo is actually in the org repo, and manage two (potentially conflicting) require("foo") things in my JavaScript, you get all that info hinted from the very start, and throughout the process. So, you do npm install @org/foo and require("@org/foo"), and conflicts are significantly less of a problem.

Contributor

mantoni commented May 15, 2014

@isaacs Thanks for taking the time to explain. I think it's going to work for me as suggested.

I tried to follow this discussion and try to make my point here now, apologies for bad language.

I don't feel quite comfortable with this whole thing. I don't like the fact that @SomeCompany should be a namespace. if i want to namespace some packages i can do it, today like myns-mypackage. namespacing is all about prefixing of names. this is a non-issue imho.

private packages is an issue and here i would go with the prefixes notation, but for repository resolution only. so if i put @somens/somepackage as a dependancy, then i either define that @somens points to a specific repository, or i let npm resolve this name to an url from a repository registry. this would imply. This has some benefits:

  1. If you want to keep your package private, you can either host your own repo (fits @isaacs most paranoid customers), or buy a hosted one as i.e. nodejitsu is offering (fits the shut-up fraction). then just add a repository to your local client and go on.
  2. if you want to publish you private package later, you can just do it on main npm.
  3. every repository can provide public access to a subset of packages. if they do it so, they can register themselves as public repos in main npm, so npm can resolve @namespace to repos
  4. no need for 404 disclosure in this case: 1. look at the private repo, 2. look if there is a public repo for this namespace. 3. if there is one, load it from there. if not, spit out an error to the client, because in this case something isn't configured right

i hope i made my point clear here. :)

Owner

isaacs commented May 23, 2014

@greelgorke You are essentially describing how the feature is designed to work :)

@isaacs : I think it would be nice if;

var foo = require('foo');
// registry.npmjs.org/foo
// No changes

var foo = require('foo', 'username');
// registry.npmjs.org/username/foo
// Public packages

var foo = require('foo', 'username@namespace');
// registry.npmjs.org/username/namespace/foo
// Public and private packages depends on package settings on the registery
// Also audit, ENV, etc. can be control by remote registry

var foo = require('foo', {registry: 'private-registery.com'});
// private-registery.com/foo

var foo = require('foo', {namespace: 'me', registry: 'private-registery.com'});
// private-registery.com/me/foo

var foo = require('foo', {namespace: 'me@project', registry: 'private-registery.com'});
// private-registery.com/me/project/foo

// Example;

// Public express module
var express = require('express');
// User's cookie parser - public
var cookieParser = require('cookie-parser', 'me');
// User's private logger module
var dbLogger = require('db-logger', 'me@underdevelopment');
npm install foo
npm install foo -ns username
npm install foo -ns username@namespace
npm install foo -ns username@namespace -registery url
"dependencies": {
  "express": "4.1.x",
  "cookie-parser": {"version": "1.0.x", "namespace": "me"},
  "dbLogger": {"version": "1.x", "namespace": "me@underdevelopment"}
},
"devDependencies": {
  "mocha": "1.x",
  "chai": "1.x",
  "fakeDatabase": {
    "version": "1.x", 
    "namespace": "org@private", 
    "registery": "registery.org.com"
  }
}
Member

edef1c commented May 25, 2014

@cmfatih
Anything changing the behaviour of require is off the table, really. the require algorithm is a frozen part of node core. npm can't change it, and changing it in node core is off the table because it's frozen.

@isaacs i didn't had that impression after all the chatter here

@isaacs @greelgorke perhaps a better notation than @org/name would be @registry/package to avoid any confusion

👍

Contributor

fengmk2 commented Jul 18, 2014

Any news from "scoped" packages? When "scoped" packages publish on registry.npmjs.org ?

Contributor

othiym23 commented Jul 18, 2014

@fengmk2 scoped package support is in the v1.5rc versions of the client (install the latest with npm -g install npm@v1.5rc if you want to play with what's there so far). The server-side support will be out later this year, but we don't have any specific dates yet.

Contributor

fengmk2 commented Jul 21, 2014

@othiym23 Thanks a lot! We use npm v1.5rc on our private registry now, and it support the scoped packages by https://github.com/cnpm/cnpmjs.org.

Contributor

othiym23 commented Sep 16, 2014

With the publishing of npm@2.0.0, this feature is now complete in the npm CLI. Scoped packages are currently supported by npm Enterprise (and at least one other private registry product, cnpmjs), and will be coming to the primary npm, Inc. registry in the not-too-distant future.

Thank you to everyone here for your contributions!

@othiym23 othiym23 closed this Sep 16, 2014

@othiym23 Will scoped packages be made available in the public version of npmjs.org?

Contributor

othiym23 commented Sep 29, 2014

@bkochendorfer Yes, absolutely. Current estimate is that it will be rolled out before the end of the year. Look for a blog post about this soon.

@othiym23 👍 Thank you

kflorence added a commit to solarmosaic/gulp-tasks that referenced this issue Dec 4, 2014

Update package name
Looks like company scoped packages aren't supported in the public
registry yet, see: npm/npm#5239

What's the status of getting scoped packages into the public registry at this point?

Contributor

othiym23 commented Feb 19, 2015

I think the best word for it would be "imminent". No fixed date, but very soon indeed.

taoqf commented Jun 13, 2016

@isaacs i have an exception:
i use webpack for packing my packages , error thorws when i ignore some packages becuase @ could not be a variable name.
cnpm/cnpmjs.org#955

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment