"scoped" packages

[isaacs]

The biggest request from enterprise users is a way to manage private modules with the same level of simplicity and ease as open source modules in the public npm registry. (A close second is group/role based management, but let's leave that for a subsequent discussion.)

Current State of the Art: Private Registry Replicas, Private Git, Proxies

Since the registry is (mostly) a Couch App, and we expose a public CouchDB interface to it, you can replicate the metadata into a database on your network, and optionally also use the npm-fullfat-registry module to fetch binaries into your copy of the db as attachments.

However, this is suboptimal in the following ways:

1. You end up replicating hundreds of GB of stuff that you don't need.
2. CouchDB with half a million binary attachments is not an easily maintained beast.

PayPal wrote kappa to deal with this, which doesn't require using a CouchDB full of attachments. There are several other similar proxy implementations that store the metadata and binaries to disk, but this is the one that I hear of the most often. This is a bit better than managing a Couch App, and has pluggable interfaces for caching, whitelists, etc.

Another option is to use private git repositories, either managed in-house or on GitHub. The npm client has some snazzy shorthands for GitHub repos, so you can do "foo":"org/node-foo" in your package.json dependencies, or npm install org/node-foo on the cli. However, there is a bit of an impedance mismatch and occasional friction in using git repos as npm dependencies. It's ok in a pinch, or if you have really well-established practices, but it's generally always going to be a bit of a mismatch, simply because git and npm make different assumptions about the world.

Name Conflicts

If you use basically any private system, you'll end up with name conflicts. Let's say our company creates a module called marked, to track places that we've marked in our location tracker app. Then we decide to use some public module that depends on @chjj's marked module, and it ends up installing our location tracker doohickey as a dep, because the registry is set to npm.my-company.com.

We can get around this by implementing some rather convoluted logic to manage multiple registries, and keeping track of where on the list of registries we started in the lookup process, and always go down the list rather than up.

However, this means that we can't do things like fork the markdown parser package named marked to fix a bug, and have our dependency lookups find that instead. There is no way to solve both problems, because they contradict one another.

What's worse, the logic for knowing how to resolve conflicts gets really confusing to reason about, and when you're looking at code, it's not clear what require("marked") refers to.

So, we have to be very careful to not use names that are publicly used, or squat on public names, which disrupts the OSS community flow, and so is generally bad behavior.

Namespace Constraints

It'd be nice if we could scope modules to a specific user or organization. If such scoped modules were able to be made private, and if they could be unambiguously recognized by the client, then it would know that it might have to do a "give me access to this private thing" authorization dance in those cases.

Some constraints:

1. There must be a simple and unambiguous way to say "install mycompany's marked module".
2. It must not conflict with any currently allowed names. (Ie, it should be invalid in the current paradigm. Not hard to do, since names are very restrictive.)
3. It should not conflict with the user/repo shorthand used for GitHub. (Oops. That would've been a really nice way to go. Oh well, backwards compatibility. It sucks, but not as much as it would suck to break users.)
4. If you do npm install <scoped module> then you should be able to do require("<scoped module>") in your JavaScript, and view the data at https://registry.npmjs.org/<scoped module> in a web browser.
5. Since (3) implies we have some kind of magic character or syntax, (4) means that any such magic character/prefix/syntax must not use characters that are magical in URLs, file systems, or shells. (For example, npm install ~org/foo or npm install $org/foo would be fine in the fs and url, but the shell will expand ~org to the home of user org and $org to the value of the org var. + and # and % all have similar problems in other places.)

I've discussed this with the people here at npm, Inc., and with a few of the most prolific npm contributors and publishers, and users at several large companies. The scheme that seems to be the most intuitive, which also fits within the rather narrow constraints is @org/pkgname.

Multiple Registries

Each @org will map to a specific registry in the config (or the default --registry config, which defaults to registry.npmjs.org, if not specified).

We can then also scope auth info to each registry, and voila, we have multiple registry support that works, doesn't conflict with anything that already exists, doesn't result in fetching the right thing from the wrong place.

@org/pkg modules will be dropped onto the filesystem at node_modules/@org/foo, so require("@org/pkg") will work.

And since that's the actual package name, you'll do "dependencies":{"@org/pkg":"1.x"} in your package.json file.

As a way of implementation details, the @org/pkg thing will map to npm://{registry}/@org/pkg, where {registry} is the registry that is assigned to the @org or the default --registry config. npm:// urls will be fetched via https, and always send authentication information, and treat 404's as a potential "maybe you don't have permission" response.

This does mean that the registry will probably not be strictly just a Couch App for much longer. In practice, it already isn't, but since CouchDB chokes on doc names that have a / in them means that we'll have to %2F it, and that's just a bit confusing. Maybe we could use - instead of /, but the niceness of / as a filesystem separator is really awesome.

We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic. Basically, npm-registry-couchapp will be just a part of the registry implementation, instead of effectively the entire thing. In addition, there'll be a minimal "front door" bit that proxies to the Couch App, manages roles, restricts GETs, etc., and that's where you'll find hooks for auth magic. That also means we can avoid putting the attachments into CouchDB in the first place, since all we do is then immediately take them out of CouchDB, without breaking old clients, and opens the door to doing other creative stuff, like more efficient views, relational reporting stuff, and putting download counts and other analytics right into the data itself. Handwavey vaporware, etc.

Feedback please

Would you use this? What would you like it to do? Anything about it that you think would be weird or confusing? Is there an approach you'd like better that meets all the constraints?

[tamzinblake]

+1

[Raynos]

Questions

• is the "name" in the package.json "@org/foo" ?
• how do we deal with org names in the public registry ? How do we reserve them ? For example can I publish "@npm/config" ? What if I have an internal "@foo/bar" and want to use a public module that's also called "@foo/bar"
• are we updating shrinkwrap to know how to deal with "@org/name" ? as far as shrinkwrap cares there is just a folder called "@org" in node_modules with no package.json.
• what's the user experience when you accidentally require("@org")
• will the node_modules/@org folder need a package.json ? What will blow up if the folder doesn't have a package.json

[Raynos]

First class citizen questions

The existing story around private git links has some usability problems.

• will npm docs @org/foo work, it doesn't for git links
• will npm info @org/foo work
• will npm be able to dedupe these @org/foo modules properly, it can't dedupe git links
• will npm update @org/foo work ? It just installs the latest for git links
• will npm i @org/foo@{range} work ? it doesn't for git links
• will npm link @org/foo work ? it doesn't for git links.

We should think of all the areas where git+ssh links are a second class citizen in terms of the features of the npm cli and ensure that @org/foo does not suffer similar problems.

This will help address

manage private modules with the same level of simplicity and ease as open source modules

By making @org modules just as easy & simple as open source ones.

[Raynos]

btw this sounds great :) 👍 on this, let's make it really really smooth & easy to work with.

[isaacs]

is the "name" in the package.json "@org/foo" ?

Yes.

how do we deal with org names in the public registry ? How do we reserve them ?

Effectively, orgs are users. You've already "reserved" @raynos, for example.

Names will be handled similar to the "disputes" policies that we already have. If your company name is foo, and some jerkwad is squatting on @foo and not using it, we may be glad to evict them, but that's a "have the humans use their human brains for moderation" kind of issue.

For example can I publish "@npm/config" ?

No, because you're not in the @npm org.

What if I have an internal "@foo/bar" and want to use a public module that's also called "@foo/bar"

Don't do that :)

You do bring up a reasonable issue, in that this doesn't solve all namespace problems, but "merely" adds a single additional dimension, so there's "only" one less order of namespace collision magnitude.

are we updating shrinkwrap to know how to deal with "@org/name" ? as far as shrinkwrap cares there is just a folder called "@org" in node_modules with no package.json.

Yes. Shrinkwrap will know how to deal with @org/pkg as the name of a package.

what's the user experience when you accidentally require("@org")

Probably it will throw a "module not found" flavor of ENOENT.

will the node_modules/@org folder need a package.json ? What will blow up if the folder doesn't have a package.json

Nope. node_modules/@org will probably just have folders in it. If we do throw some kind of helpful metadata in there, it'll probably just be for the npm client's use.

The existing story around private git links has some usability problems.

Tell me about it! They're kind of awful.

will npm docs @org/foo work, it doesn't for git links

Yes.

will npm info @org/foo work

Yes.

will npm be able to dedupe these @org/foo modules properly, it can't dedupe git links

Yes.

will npm update @org/foo work ? It just installs the latest for git links

Yes.

will npm i @org/foo@{range} work ? it doesn't for git links

Yes.

will npm link @org/foo work ? it doesn't for git links.

Yes.

We should think of all the areas where git+ssh links are a second class citizen in terms of the features of the npm cli and ensure that @org/foo does not suffer similar problems.

Agreed. We're starting with the experience of non-org-scoped modules ("global" modules, in our internal parlance), and going from there. Rather than using Git as the model, since it is really klunky to work into npm's way of doing things.

This will help address

manage private modules with the same level of simplicity and ease as open source modules
By making @org modules just as easy & simple as open source ones.

That is the plan :)

[isaacs]

Another thing:

"Global" modules (ie, unscoped, what we have now) will not be allowed to depend on scoped modules. This is important.

Otherwise, the community as it is now would be unduly harmed, because we'd very quickly end up with global packages that can't be installed by old versions of npm.

This "namespaced maybe-private org-scoped" thing is only going to work with newer npm clients of the future, but there's no reason to invite ruin on older npm clients that are currently working, unless it's actually necessary for some reason.

I want people to upgrade to the latest and greatest npm, but it should be much more carrot than stick.

[hayes]

just for clarification, if an @my-org is scoped to a private registry https://registry.my-org.com will packages be fetched at https://registry.my-org.com/@my-org/my-module or just https://registry.my-org.com/my-module

if it were the latter, more existing setups may be supported, and if you have a private registry that supports the new @org pattern, you could set the url for for your orgs modules to https://registry.my-org.com/@my-org

[isaacs]

@hayes That's interesting. Our assumption had been that it'd be https://registry.my-org.com/@my-org/my-module, so that the npm client could just Do The Same Thing no matter where it's fetching stuff from. But, I can definitely see the benefits there. Maybe "send the org name" could be configurable in the same way that the registry is, for maximum flexibility?

[hayes]

I am a big fan of flexibility. I think that could be a very useful options. but after thinking about it for a minute I realized it couldn't be the default

[indexzero]

@izs thanks for opening a place to discuss this.

Name Conflicts

This is of course inevitable, which is why the proxy I wrote with @jcrugzz has a policy format which understands both blacklists and what we refer to as "known private modules". This also solves the question from @Raynos

What if I have an internal "@foo/bar" and want to use a public module that's also called "@foo/bar"

Because you could simply blacklist your @foo/bar. The default for all publishes is to proxy them to your private registry, which leads to the one caveat of our private npm product: existing public modules proxy pass through just fine, but that new public modules have to explicitly be published directly to the public registry. This gotcha does come up every now and again, I've seen support tickets heard this from 4-5 customers (about 1-3%).

Namespace Constraints

I agree in principle that we need something to hang users off of. Discussions around "groups" have happened on and off for years, which is essentially the same thing as what is being referred to "orgs" now. What I don't explicitly see the value in is the @ scoping. If you are implementing private packages using the existing "private" then a large breaking change is already being introduced since up until now "private" has meant "never publish this thing to any registry ever".

Knowing that private packages will in principle cause a large swath of breaking changes for users why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now? It should be pretty easy to run a view to see how many packages are using that now. Whichever thing has lower impact from data should be the answer. I'm separately of the opinion that backwards compatibility shouldn't be preserved just because reasons, but then again seeing @ry this week might've just made me nostalgic :)

Multiple registries

We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic.

That's a pretty bold statement, but we of course have our own pieces that are private at Nodejitsu so I can't call the kettle black. The biggest request I would make it make importing and exporting private modules out of the private thing easy because both of our customers will want that.

+1 to moving more complex logic out of the CouchApp as jcrugzz outlined in his post back in February. With that in mind, if you are overhauling the auth system we should really consider implementing OAuth2 as a proper way to grant authorization to other apps. It's not the best, but OAuth2 has it's fingers in enough things that it's a sane first choice.

If you move the main proxy logic out of the CouchApp (i.e. by removing the rewrites) then the %2Fs aren't too bad. I've done this a lot for other CouchDB based applications to get vanity URLs and just taking advantage of the kv + map-reduce qualities of CouchDB. Clearly I have other opinions about where binary data should be, but that's outside the scope of this discussion.

Aside: Package signing

If we are really going to say "send all requests to @org/* packages to http://etc-etc-etc" we still have not prevented the specter of MITM attacks because by introducting N places to get packages we simply have N more places that could be vulnerable. I think that "orgs", "teams" (i.e. group/role management), and package signing are very much intertwined and should be thought of as a single unit to avoid painting ourselves into a corner as we unfortunately have done here with Github syntax.

[UnidentifiedContributor]

OK but what does this have to do with twitter?

[Raynos]

@isaacs

but there's no reason to invite ruin on older npm clients that are currently working

This argument is void. The introduction of "foo": "^1.0.0" already broke old npm clients. The community was not ruined by this change as far as I can tell.

[hayes]

will the public registry host packages that specify a specific @org in its dependency to be a different registry? (are those settings in the package.json, or are they npm client settings)

[tamzinblake]

What's the use case for this? Is it just specifically for a particular company to host their secret sauce and never give anyone else access to it? Will they migrate them to the global namespace if they ever want to make those modules public?

[chrisdickinson]

@Raynos An argument to disallow publishing packages that depend on @org packages to the public registry is that, since the binding between @org -> url is local to each machine, packages published with @org deps would be unresolvable by clients who did not share .npmrcs, no matter what version of the npm client they had installed (if I'm understanding this correctly).

[Raynos]

How will publish rights to a module at "@org/foo" work ?

Currently I can just npm owner add mattesch @org/foo ? will the user mattesch then be able to publish to a module at @org/foo ?

[chrisdickinson]

To clarify some confusion, where does the config mapping between @org and registry.my-org.com live?

1. The public registry?
2. A client's .npmrc?
3. A scoped client .npmrc? (i.e., ~/.npm/@foo/npmrc?)
4. In the package.json?

And does a missing @org cause an error, or default to the public registry? (from re-reading, I assume "it defaults to public registry", but want to be clear)

[othiym23]

@chrisdickinson The resolution of an @org to a given registry happens in the client, although the responsibility for validating permissions will probably be scattered throughout the infrastructure. All configuration will be handled using the configuration mechanisms that exist today, although if / once support for multiple / per-project configuration files lands, it might make sense to migrate some of this there. The implication is that @org can map to different registries for different users, depending on their configuration. We have discussed how to consolidate these mappings site-wide for large installations, although we haven't arrived at a final decision on how that's going to work.

Something useful to keep in mind is that the mechanics of resolving packages exists independently from the permissions governing how the server determines whether a given publish is permissible (to speak to @Raynos's point). I'll let Isaac speak to the specifics of how that will work.

[chrisdickinson]

@othiym23 Thanks! The points I was somewhat foggy about were as follows:

1. Whether @org would exist as an entity outside of private registries, based on @isaacs's comment, "Effectively, orgs are users. You've already "reserved" @Raynos, for example."
2. If the above is true, whether they would become "the future" of packages published on npm -- i.e., would it become passé to public an unscoped package?
3. Can dependencies flow from public npm into @org/packages (looks like "no, due to client compatibility", but there might be other concerns there).
4. Can dependencies flow from @a/x to @b/y (one package in one org depending on another package from another org)?

As I understand it, the @org syntax is kind of like the Host header combined with /etc/hosts -- it gives clients an opportunity to use a different registry, but if not given, it will request whatever the mainline --registry is, while letting that registry know that it wants to request from a certain @org. Given that, my assumption would be that "unscoped modules" would remain the norm, "unscoped public modules" could not depend on @org modules (due to the risk of depending on a private/unresolvable dependency), and @orgs could freely depend on public packages, packages from their own @org, and between different @orgs. Is this correct?

[isaacs]

What I don't explicitly see the value in is the @ scoping.

Please see the list of namespace constraints in the OP.

If you are implementing private packages using the existing "private" then a large breaking change is already being introduced since up until now "private" has meant "never publish this thing to any registry ever".

This probably will not use the "private" field in package.json. If it does, it'll only be for new packages, since existing packages can never have a name like @org/name.

Knowing that private packages will in principle cause a large swath of breaking changes for users

Can you elaborate on which breaking changes are planned? I don't see any impact on existing users. They may be left behind if the bulk of development activities move toward namespaced modules, but their existing workflows will keep working.

why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now?

Backwards compatibility.

I'm separately of the opinion that backwards compatibility shouldn't be preserved just because reasons, but then again seeing @ry this week might've just made me nostalgic :)

Noted :)

That's a pretty bold statement, but we of course have our own pieces that are private at Nodejitsu so I can't call the kettle black.

Sorry, I'm not sure which is the bold statement there? Most of what we do at npm, Inc. is developed in the open, and I don't see any reason to change that.

With that in mind, if you are overhauling the auth system we should really consider implementing OAuth2 as a proper way to grant authorization to other apps.

Xylophone. Implementing an OAuth2 provider for npmjs.org is an idea worth exploring, but it's well outside the scope of this discussion.

Aside: Package signing

Xylophone. Package signing is important, but outside the scope of this discussion.

[isaacs]

@othiym23 asks: "What is 'xylophone'?"

Answer: http://mail-archives.apache.org/mod_mbox/couchdb-dev/201303.mbox/%3CCAN-3CBJixAKbtK+76C+h_r+vDSm+rR++cL4Xj2oe8iOJ+xDFGQ@mail.gmail.com%3E

[othiym23]

Whether @org would exist as an entity outside of private registries, based on @isaacs's comment, "Effectively, orgs are users. You've already "reserved" @Raynos, for example."

On Github, the top-level namespace is the same for both users and organizations. Same deal here. This concept is not exclusive to private registries.

If the above is true, whether they would become "the future" of packages published on npm -- i.e., would it become passé to public an unscoped package?

Given the fact that private packages can depend on public packages, but not vice versa, this is not necessarily the future. As an aside, I find it easiest to think about @orgs as segmenting one namespace, rather than creating multiple top-level namespaces.

Can dependencies flow from public npm into @org/packages (looks like "no, due to client compatibility", but there might be other concerns there).

If you mean, can package @aoaioxxysz/euterpe depend on request, most assuredly. The other way, no.

Can dependencies flow from @a/x to @b/y (one package in one org depending on another package from another org)?

Yes.

As I understand it, the @org syntax is kind of like the Host header combined with /etc/hosts -- it gives clients an opportunity to use a different registry, but if not given, it will request whatever the mainline --registry is, while letting that registry know that it wants to request from a certain @org. Given that, my assumption would be that "unscoped modules" would remain the norm, "unscoped public modules" could not depend on @org modules (due to the risk of depending on a private/unresolvable dependency), and @orgs could freely depend on public packages, packages from their own @org, and between different @orgs. Is this correct?

This is correct.

[isaacs]

@Raynos

The introduction of "foo": "^1.0.0" already broke old npm clients. The community was not ruined by this change as far as I can tell.

We only switched to ^ as a default once it had been supported in stable releases of Node/npm for almost a year, iirc, and in fact, there was quite a bit of wailing and gnashing of teeth, leading eventually to the recent addition of the --save-prefix config.

The community wasn't ruined, but it was also a much smaller change than this addition, and in retrospect, I would have liked very much to have put more thought into avoiding that issue. In this case, the rule that "global modules can't depend on scoped modules" avoids the problems rather nicely.

How will publish rights to a module at "@org/foo" work ?

Still working out the best way to accomplish that, but yes, you'll add a user or a group to the package, and then there can be a way to add a user to a specific group.

Part of the purpose of our on-prem beta program is to give us a little petri dish in which to experiment with a few different ways to manage auth and groups. Most likely, there'll be group management by the end of the year in the public registry, and we can have another one of these to discuss that. Until then: xylophone.

@thomblake

What's the use case for this? Is it just specifically for a particular company to host their secret sauce and never give anyone else access to it? Will they migrate them to the global namespace if they ever want to make those modules public?

Scoped modules will be either public or private, at the whim of the owner. I will not expect that they will ever be migrated to the global namespace. This thus satisfies another common request: having simpler/shorter names for packages, scoped to a specific owner, so that you can have require("@company/foo") instead of require("company-foo-but-not-the-foo-for-bar-the-other-for-the-one-for-baz-you-know-for-kids").

@chrisdickinson

I think you're understanding it correctly. Deps can go from scoped -> scoped, scoped -> global, or global -> global, but never global -> scoped.

---

One downside of the @-sign that I didn't realize until posting this discussion here: we've been spamming Olly Groves (https://github.com/org) quite a lot. I apologize for that, Olly. You may want to ignore this thread entirely, I suspect it will continue.

[tamzinblake]

Yes writing @foo everywhere might spam lots of people on lots of different platforms (Sorry @foo but you were asking for it)

[othiym23]

This thus satisfies another common request: having simpler/shorter names for packages, scoped to a specific owner, so that you can have require("@company/foo") instead of require("company-foo-but-not-the-foo-for-bar-the-other-for-the-one-for-baz-you-know-for-kids").

To amplify this, this makes it easier for people to fork and experiment with existing packages to scratch particular itches without the original package needing to change or grow unnecessarily.

[chrisdickinson]

Can namespaces mutate their privateness?

... to expand:

[16:57:11]  <chrisdickinson>     the other problem is the mutability of organization private-ness
[16:57:50]  <othiym23>   chrisdickinson: care to expand?
[16:58:16]  <chrisdickinson>     I have @chrisdickinson on npm. Folks start do depend on me from their @scoped packages
[16:58:40]  <chrisdickinson>     I make @chrisdickinson private
[16:58:48]  <chrisdickinson>     I've broken all of the @scoped packages.

EDIT: I misunderstood. @othiym23 straightened me out: @org/packages publicly available on registry.npmjs.org will always be public (modulo unpublish); going private means you've (individually) spun up a private registry for @org.

[terinjokes]

Since an organization can be depended on by other organizations, and because local configuration can change where @org resolves, we'll have to register an npmjs.org account for the organization to ensure we don't accidentally shadow some other organization.

If @org is directed to some other registry, but there are some @org packages that are public, the other registry would need to have both the private and the public packages, as the client won't fallback to the public registry.

Previous discussion within the community seemed to have concluded on scoping packages by registry, not by organization. I want to make sure I'm entirely clear of the impact here.

[indexzero]

Turned into a bit of an essay, but please bear with me. I am not simply disagreeing for the sake of being difficult.

The private mechanism is important here

@izs

Can you elaborate on which breaking changes are planned? I don't see any impact on existing users. They may be left behind if the bulk of development activities move toward namespaced modules, but their existing workflows will keep working.

So if the TL;DR here is "org ns as step towards private modules" it is reasonable to ask what the mechanism for making things private is going to be.

• Is it going to be a flag in the package.json?
• Is it going to be done through the npm CLI?
• Is it going to be done through npm-www?

I (wrongly) assumed that the plan was to repurpose { "private": true } in package.json files which was the breaking change I was referring to.

Why so in-favor of strict backwards compatibility?

@izs

why not just roll prefix-less orgs into npm@2.0.0 and start deprecating the Github syntax now?

Backwards compatibility.

Right now there are 225 public packages which use this syntax in their dependencies (did not check *Depedencies like devDependencies), or roughly 0.320% of all packages (source). So that is not a lot, but it is enough to want to find a backwards compatible way to fix this. That mechanism doesn't have to be as strict as to introduce a new syntax that every new node developer is going to have to learn. What about this flow:

LOAD_AS_ORG(X,Y)
1. Check to see if `X/Y` exists in npm.
2. If not, assume Github repo with `X/Y` and warn the user of deprecation.

It would require formal deprecation of the Github syntax, but it precludes a new syntax. Further, we could use the new @ syntax to replace the existing Github syntax so people can switch when they see the deprecation notice. This is clearly not the list of all consumers of this feature, but it is representatively small. Any interest in surveying users on their existing use of this feature? Have you already?

Based on the anecdotal evidence I've seen organizations would be used 1-2 orders of magnitude more frequently than the existing github shorthand -- most of them use the git:// URL syntax explicitly. So if we could deprecate and replace that seems like a win, not a loss.

Why are we complicating things?

@othiym23 @chrisdickinson @izs

I think you're understanding it correctly. Deps can go from scoped -> scoped, scoped -> global, or global -> global, but never global -> scoped.

I misunderstood. @othiym23 straightened me out: @org/packages publicly available on registry.npmjs.org will always be public (modulo unpublish); going private means you've (individually) spun up a private registry for @org.

If scoped modules are public by default as @chrisdickinson and @othiym23 said, why the restriction? There is a widely used workflow at companies using Node.js (and even client-side JS) which implicitly conflicts with it:

1. Company X makes a module, some-thing that they intend to Open Source.
2. Company X publishes that thing to npm as a private package @company/some-thing
3. A year later Company X wants to release some-thing as Open Source, but the global some-thing is taken.
4. Company X is now discouraged from Open Sourcing because of additional breaking config changes in package.json files if they change the name to some-other-thing and making the scoped @company/some-thing Open Source can't be used by global modules.

Xylophone or just not on this issue?

@izs @Raynos

How will publish rights to a module at "@org/foo" work ?

Still working out the best way to accomplish that, but yes, you'll add a user or a group to the package, and then there can be a way to add a user to a specific group.

Why is this a xylophone? It is hard (if not impossible) to completely assess the intrinsic value of the proposed organizations feature without fully understanding how the publish and user management mechanics will work. e.g. our customers love user sync across their team.

The same goes for OAuth2 and package signing. All of these features are intertwined in nuanced ways: the implementation details matter here. I can move that part of the discussion to another issue if you'd prefer.

Can we count on this module.js "hack"?

@izs @tjfontaine @bmeck

So this thing is a nit, but it seems we got lucky with the @org syntax, because according to the docs it should not exist since the @org directory does not need a package.json file.

LOAD_AS_DIRECTORY(X)
1. If X/package.json is a file,
   a. Parse X/package.json, and look for "main" field.
   b. let M = X + (json main field)
   c. LOAD_AS_FILE(M)
2. If X/index.js is a file, load X/index.js as JavaScript text.  STOP
3. If X/index.node is a file, load X/index.node as binary addon.  STOP
//
// ... ? We are here, in undefined territory.
//

module.js in core hasn't changed in almost six months, but I know that bundling into a single executable was on @tjfontaine's radar for node@0.14 which will require changes to the module loader. Summoning him and @bmeck here to see if that has any bearing on this discussion.

What is a "reference implementation"?

@izs

We'll still keep shipping an OSS reference implementation, but it probably won't do private modules, since that usually is going to require some kind of custom logic.

Sorry, I'm not sure which is the bold statement there? Most of what we do at npm, Inc. is developed in the open, and I don't see any reason to change that.

What's bold is that you're saying the OSS npm registry is going to become a "reference implementation", which won't be run in production by npm Inc. Please forgive me if I am misinterpreting this, but I've always read the phrase "reference implementation" in the past as "a toy for hobbyists". Is that the intention here?

Part of the purpose of our on-prem beta program is to give us a little petri dish in which to experiment with a few different ways to manage auth and groups. Most likely, there'll be group management by the end of the year in the public registry, and we can have another one of these to discuss that.

Could you clarify the above? Is there an implementation of these features somewhere that is not yet public? Sorry for prying, but your initial segmentation about what would be private and what would be public was rather nebulous at best and I know I'm not the only one who would like a clearer line drawn in the sand.

[bmeck]

Just to clarify, people want to change require to include a scope mechanism as well? This would be pretty major since it could break node_modules as a folder if you do have 2 modules with the same name but different scopes.

If we do use scope in require I would avoid using a specialized string in require. Keeping it separated ala require.scope(org)(module_name) or require.scoped(org, module_name) (idc syntax, just separated) since collisions can still occur if only module_name is present unless we want to continuously add new string syntax to require.

Either way, I would need more info on what is actually being proposed to be put onto disk to say much more. This would not interfere with archives or bundling most likely since that just reuses the require mechanisms.

[fengmk2]

@othiym23 Thanks a lot! We use npm v1.5rc on our private registry now, and it support the scoped packages by https://github.com/cnpm/cnpmjs.org.

[othiym23]

With the publishing of npm@2.0.0, this feature is now complete in the npm CLI. Scoped packages are currently supported by npm Enterprise (and at least one other private registry product, cnpmjs), and will be coming to the primary npm, Inc. registry in the not-too-distant future.

Thank you to everyone here for your contributions!

[bkochendorfer]

@othiym23 Will scoped packages be made available in the public version of npmjs.org?

[othiym23]

@bkochendorfer Yes, absolutely. Current estimate is that it will be rolled out before the end of the year. Look for a blog post about this soon.

[bkochendorfer]

@othiym23 👍 Thank you

[scottgonzalez]

What's the status of getting scoped packages into the public registry at this point?

[othiym23]

I think the best word for it would be "imminent". No fixed date, but very soon indeed.

[taoqf]

@isaacs i have an exception:
i use webpack for packing my packages , error thorws when i ignore some packages becuase @ could not be a variable name.
cnpm/cnpmjs.org#955