Skip to content

meteor is now on npm #516

Closed
wants to merge 6 commits into from
@substack

This pull request is to inform you that meteor is now on npm. This is beneficial for a number of reasons, but particularly that applications can now depend on a specific version of meteor in their package.json files directly which is not something that the curl ... | sh thing is at all good at.

If you would like to take over this package in order to push updates I will gladly transfer ownership on npm.

@juliangruber

+1

@vizo
vizo commented Dec 1, 2012

+1

@Multiply
Multiply commented Dec 1, 2012

I still prefer using meteorite.

@andreioprisan

+1

@gschmidt
Meteor Development Group member
gschmidt commented Dec 7, 2012

@substack, thanks, this is awesome and could be the start of something beautiful. We discussed this a bit in person last night. We're excited to see Meteor on npm, but similar to the Windows port (win.meteor.com) there's a lot of work to do here before we can call this an officially supported distribution of Meteor. The main items are:

  • the updater: we need to understand how users will find out about new versions, and what if anything 'meteor update' should do in the npm distribution, and make it do it. Also how 'meteor list' and 'meteor add' will work since we are transitioning to a model where Meteor packages can be contributed by anyone and are dynamically downloaded, as described on the roadmap (roadmap.meteor.com). Some of these are similar issues to those addressed by the rpm/deb distribution.

  • the release engineering process: if we are to officially support the npm distribution, we would need to fold it into our internal QA process and guarantee that whenever we push a new version of Meteor to 'curl | sh' users, to rpm/deb users, and to git master checkout users, that we simultaneously push the new version to npm users. This would include testing the npm distribution on each supported platform and also testing the update logic, which is not a small commitment.

Realistically, there's a lot of work here, much of which is QA and ongoing support commitment rather than code. The core team won't be able to take this on in the short term. But we are happy to help as we can with an unofficial distribution, and maybe it the future we will be able to bring this in-house. I'm very interested in continuing to find ways that Meteor and npm can work more closely together, both in this and other ways.

@TomWij
TomWij commented Dec 7, 2012

we need to understand how users will find out about new versions

You shouldn't need to understand how users will find out about new versions, distributions and package managers will take care of that. At best you can notify the user through executing meteor that there is a new version available, which you already do; although it could also mention that they should check their package manager for that.

if anything 'meteor update' should do in the npm distribution

The solution here is simple, meteor update should not exist on any "distributed" version of Meteor; it introduces "yet another way to update" on any platform and it doesn't conform to package managers (Whether that's something like Aptitude, NMP, Homebrew, MSI, or Portage) and hence to do it correctly meteor update would need to update in a different way on each platform, or rather just not exist at all.

On Windows, installing and updating becomes as easy as installing a new MSI file or perhaps even just calling npm install meteor@X.Y.Z (and updating in a similar way) like I did in the start. On Gentoo, given my user overlay is added (till it hits Portage itself) all it takes is to run emerge meteor (which updates along when I update everything) to get the latest version or emerge =www-servers/meteor-X.Y.Z to get a specific version.

Hacks like #474 or bundling along binaries (Node.js / MongDB / ...) through a bundle aren't needed, package managers can do that for you; on Gentoo I already place Meteor in /opt/meteor and symlink /usr/bin/meteor to /opt/meteor/meteor, next on I'm going to patch everything that comes bundled with Meteor away since that only results in installing files the user might already have. Hardcoding the Node.js version is yet another hack, this could as well be a range of supported Node.js versions instead, perfectly expressible in many of the dependency specifications out there.

@TomWij
TomWij commented Dec 7, 2012

Continuing from previous message.

Also how 'meteor list' and 'meteor add' will work since we are transitioning to a model where Meteor packages can be contributed by anyone

Just do NPM, no need for any reimplement the system and no need for using external solutions like Atmosphere that don't work out cross-platform simply because they fail in the same way as meteor update does.

whenever we push a new version of Meteor to 'curl | sh' users

curl | sh users should be an afterthought, I don't think everyone runs a DSL distribution here; what about the majority of users that prefer to use package managers?

to rpm/deb users, that we simultaneously push the new version to npm users

No, you as the upstream package developer should and cannot decide which version of Node.js users have as stable; you somewhat have control over this in NPM and MSI, but that's where it stops. For instance, on the package manager Portage (where Gentoo is based on it) takes a month without bugs for a package to be stabilized and get out of the test keyword.

This would include testing the npm distribution on each supported platform and also testing the update logic, which is not a small commitment.

No, this ain't your task, this is the task of the package maintainers at the individual platforms to check out. They will report to you with any bugs that happen; therefore you shouldn't need to test on Gentoo, people (like me) will do that for you. No commitment involved here.

Package managers for the individual distribution should come first in the installation instructions for this to work, then the MSI installer for Windows user, then the NPM instructions and then the manual instructions.

You should put in front those things where a lot of involvement by your users happens to ensure it is "really stable", not those things where you have direct control over what code runs on your clients. What if you release 0.5.3 tomorrow and break the computers of most of the curl | sh users? Users through the package managers wouldn't have a problem; since it would only break for the package maintainer, perhaps some testers using the "test keyword", but not most of the package manager users.

@TomWij
TomWij commented Dec 7, 2012

+1 Hoping to see Meteor embrace the platforms and package managers it will eventually run on, instead of hacking together a one-shot installer and updater that requires a weird architecture and unnecessary workarounds.

@darylantony

I'm interested in the debate.

It seems that you've made very good points here @TomWij –– yet, I'm not qualified enough in the npm world, or meteor yet to make comment.

However, there must also be a host of good reasons why Meteor would create their own package management?

Can you imagine what they might be @TomWij?

Is there some written literature somewhere to support the design case?

@TomWij
TomWij commented Dec 7, 2012

However, there must also be a host of good reasons why Meteor would create their own package management?

Well, they're doing it for the Meteor packages, although it still makes some sense for them to do so I'd rather also not see them do it. But it wouldn't hurt Meteor as much as not working towards embracing distributions and package managers in general.

Is there some written literature somewhere to support the design case?

Not that I know of, maybe... Make It Work, Make It Right, Make It Fast. ~ Kent Beck

It works now, can we make it right?

@darylantony

Right!

@darylantony

Upon further browsing... this seemed appropriate:

Is that "selling"? It seems more like a manifesto to me, looking back at it. -- KentBeck
Hmm. Are you saying I can't be a techno-crazed toy-slut and still get things done in innovative ways? :-)

from: http://c2.com/cgi/wiki?ExtremeProgrammingExplainedEmbraceChange

@mpj
mpj commented Dec 17, 2012

For the people confused around the reasons why Meteor doesn't use npm, I've written about this here:
http://www.quora.com/Node-js/Why-does-Meteor-use-its-own-package-system-rather-than-NPM/answer/Mattias-Petter-Johansson
(Short version: Meteor is not a node framework, it's a node+browser framework, while npm is only made for node, and doesn't really take the browser much into account).

@TomWij
TomWij commented Dec 17, 2012

For the people confused around the reasons why Meteor doesn't use npm

How can we be confused when there has been no reasoning?

And who are you to tell us reasons that aren't from Meteor DG?

I've written about this here

Why not here in this issue?

Meteor is not a node framework, it's a node+browser framework, while npm is only made for node

You are writing pure non-sense since Meteor is a node framework; assuming that it is illegible just because it serves JS to a client is a weird thing to do, since a lot of Node.js frameworks do such thing.

and doesn't really take the browser much into account

Why would it need to take the browser into account?

You are creating problems where there are none.

@mpj
mpj commented Dec 17, 2012

Whoa! I sense some hostility here. Just trying to add to the discussion - it was not my intention to offend. Just to clarify: I'm not one of the Meteor devs, just a guy observing the framework.

Anyway, here is a clarification of my "pure non-sense" (Real human here, reading your words, by the way :))

One of the very cool things about Meteor is that you don't have to care so much whether you're executing on the server or client. The server and client API is the same. Of course you are aware of it, but it's not as in-your-face as when you're building something with Ruby on Rails, for instance, where almost every single client-server interaction requires extra thinking.

NPM modules are built assuming execution on a server in node, because that is almost always the place where npm modules are executed. They are rarely tested for browser execution. Meteor Smart Packages, on the other hand, is aware of when it is executing in the browser or server, and is also aware of, and makes use of, the Meteor framework. Thus, it can make use of I/O when it has access to it, and make use of browser functionality when it has it. It can also hook into Meteor itself.

An example of such a package is accounts-instagram.
https://atmosphere.meteor.com/package/accounts-instagram

It would just not be possible to build a one-line install for a package like that with npm, it requires a more powerful solution. So, if Meteor were to embrace npm, it would have to be in addition to it's own Smart Packages, unless they significantly scale down what Packages can do.

But even if one thinks that using two package managers for the framework is a good idea, it would still be suboptimal when you use npm modules. While many npm modules would, of course, work fine, it would be an unpredictable experience for the developer - You would not be able to simply require any npm module, because there is a high chance that a module makes use of native node functionality, or other stuff not widely available in browsers (such as Array.forEach, for instance, which breaks in IE 8). I personally think it feels icky.

@ghost
ghost commented Dec 17, 2012

The main problem I have with meteor packages is the "everything is everywhere" mentality. Maybe it's just me, but having packages just dump things into my global scope makes me nervous.

@andreioprisan

The everything is everywhere mentality is part of the core meteor design. The issue with node.js packages is that they're server side only vs everywhere with meteor, then you can override access to certain functions on the server side so that the client cannot execute functions unrestricted, or appends parameters so that all functions have restricted scope (i.e. db update functions that are exposed in the client can add a scope limit server-side to update only entries that are authored by that user).
See http://www.quora.com/Node-js/Why-does-Meteor-use-its-own-package-system-rather-than-NPM/answer/Mattias-Petter-Johansson

@awatson1978
@TomWij
TomWij commented Dec 17, 2012

Whoa! I sense some hostility here.

@mpj: Your first comment had a reason why Meteor sentence part which would impose this is Meteor DG's reasoning, which is not the case; the only reasoning I have seen publicly is the comment before my first one.

But no, I'm just playing devil's advocate since you didn't think this through enough; you don't add to a discussion if you don't back up the statements you make, that's why they don't make much sense to me. No need for offence, but also no need to force subjective opinions as facts (see the reasons why meteor from above), or not backing up the statements you make (see the Can you elaborate why? occurrences below).

I'll elaborate why I think that both your comments don't make much sense to me, so you see my point of view:

NPM modules are built assuming execution on a server in node, because that is almost always the place where npm modules are executed.

Meteor runs on the server, Meteor smart packages run the server, they sent some code and assets to the client; perfect fit for npm since node modules can do the same.

They are rarely tested for browser execution.

Actually, there is quite some testing done in Meteor, ran through several browsers; testing is irrelevant wrt integration with npm.

Meteor Smart Packages, on the other hand, is aware of when it is executing in the browser or server

It is not the smart package that deals with this; a smart package is like a node module, containing code to be ran on the server and some extra data that the server might send along to the client; this is a perfect fit for npm since node modules can the same.

It would just not be possible to build a one-line install for a package like that with npm, it requires a more powerful solution.

Can you elaborate why? npm install meteor-somepackagename would create a folder in node_modules/... (and do the same for dependencies) which Meteor can then use; npm does more out of the box than what we have now.

So, if Meteor were to embrace npm, it would have to be in addition to it's own Smart Packages

Can you elaborate why? As shown npm is a drop in replacement where Meteor has to be slightly adjusted to use the npm directories and style instead.

unless they significantly scale down what Packages can do.
...
it would still be suboptimal when you use npm modules

Can you elaborate why? I don't see how npm would take away features of smart packages.

it would be an unpredictable experience for the developer

Can you elaborate why?

You would not be able to simply require any npm module, because there is a high chance that a module makes use of native node functionality, or other stuff not widely available in browsers (such as Array.forEach, for instance, which breaks in IE 8). I personally think it feels icky.

You don't have to require "any" npm module, you can just lock it down to requiring modules that start with meteor- and have a very predictable behaviour. Of course they use native node functionality, it's the same current packets can do as well. Since they are meteor-, the developer will be aware whether the code has to be run on the client or the server and can develop accordingly so there isn't a problem here.

Developers are actually already doing this at the moment, as they were learned to do that; so it would be a very Meteor-ish approach. I'm heavily defending this because the current and upcoming approach, as well as the Meteorite approach are not the right approaches. I personally feels this is much better than any other solution you could come up with, re-use your environment instead of making alternatives that will never be as good. For elaboration on both these last two sentences, you can see my first two comments to this issue.

Sorry if you have felt offended, I'm just want to discuss towards a better Meteor; have a nice day! :)

@TomWij
TomWij commented Dec 17, 2012

Commenting to others:

Maybe it's just me, but having packages just dump things into my global scope makes me nervous.

@mintplant: They don't have to land in the global scope, installing packages "per Meteor application" can be done.

It's probably increased my productivity 10x for what I'm trying to do, and I just don't see NPM as offering the same kind of focus.

@awatson1978: Why would it not offer that kind of focus? You would still be using the same command and approach; so as a result, there is no difference in productivity.

@alex-okrushko

@TomWij :+1:
Also at the moment Meteorite does not support Windows. Packages have to be downloaded and installed manually. :(

@awatson1978
@awatson1978
@TomWij
TomWij commented Dec 17, 2012

@awatson1978: What do you mean by "subset installation modules"? If it means dependencies, then yes, support for dependencies is present. If you mean installing modules into a Meteor application instead of globally, then yes, support for installing locally is present. If you mean something else, then please elaborate.

I think NPM has all the features that Meteor needs as I can't come up with one that is missing; and indeed, if you haven't gone through what it can do it's worth checking it out.

@belisarius222
@belisarius222
@alex-okrushko

@belisarius222 I commented on the bug for Meteorite, and got the response that it's Windows and there is no time to fix (or make it work). The solution is here pretty simple, actually. :)
my point is that adding yet another packaging system (Meteorite) to npm and Meteor creates more complexity/maintainability problems. (like "oh, it's Windows")

@TomWij
TomWij commented Dec 17, 2012

I think this is all a bit of a moot point at the moment, considering that the Meteor core devs are building their own package management system right now no matter what gets said here.

@belisarius222: They requested feedback here and are still early in that rebuild, so the chance to make a turnaround is either now or after the the prototype they are building has been tested and deemed insufficient and see npm as more superior.

  • execution/threading model: server-side code in Meteor has to run in a Fiber. I would expect this to conflict with many, if not most, out-of-the-box npm modules.

Meteor actually already uses npm modules, see the npm install lines in admin/generate-dev-bundle.sh. But what you pose isn't a problem if we only accept meteor- npm packages an smart packages.

  • deployment: Meteor has its own deployment process that's quite different from anything else in node (and better: node by itself is not known for being easy to deploy).

Meteor itself in npm would change this deployment, and once you have that you can have everything else you need as a dependency. Deploying an application then becomes as easy as npm installing all the dependencies for your application, which is quite trivial. Can you elaborate on why it is not known for being easy to deploy?

  • code organization: Tom, you mentioned that a smart package contains code to be run on the server.

Yes, they do run on the server since it is is the server that ends up processing them and send them to the client. The rest you say regarding "code organization" and "reactivity" assumes you would install any npm package as smart package, but you'd really want meteor- npm packages as smart packages.

May I remind anyone here that the suggestion is to "bring meteor into npm" (including its smart packages) and not "bring npm into meteor" (no, I do not want Meteor to magically support every npm package in the entire world).

If Meteor was expected to be downloaded through npm, it would be misleading

Why would it be misleading? Can you elaborate?

We would end up with hundreds of posts to this group and stack overflow complaining that Meteor broke people's node projects

Why would it break people's projects? Can you elaborate?

which would be true, but it's avoidable by creating reasonable expectations.

You can't assume it to be true, that's no solid ground for changing expectations altogether.

Somewhat conversely, if it was accepted practice to include npm modules into Meteor, we'd have hundreds of posts complaining how including some npm module broke people's Meteor projects.

I have not suggested such thing, see my part above explaining we want to "bring meteor into npm" (including its smart packages).

Either way, it would present enormous difficulties all across the board, and the benefits wouldn't outweigh those difficulties.

Which difficulties? Can you elaborate?

@awatson1978
@TomWij
TomWij commented Dec 17, 2012

@belisarius222 I could contribute to doing that (and actually have already tested part of it in the past to outline the issues with @possibilities) which shouldn't take much work as it was mostly Meteorite checking some wrong places that aren't cross platform (reading the HOME env variable, for instance), but I have decided not to do that so far since Meteorite is a way too intrusive approach. On the other hand, npm is much more cross platform; as I have explained in my first comments. If npm does not end up being accepted, I will look at what ends up being the management system and contribute to that; but in any case I don't want to contribute to multiple systems that are pursuing similar goals, because that's not productive.

@TomWij
TomWij commented Dec 17, 2012

a paternalism, I suppose.

I think I know what you mean, in Portage on Gentoo Linux these are called "blocked packages" to enforce you down certain "patterns" (which fits the word "paternalism"). I think you could implement checks as part of a preinit script to achieve such thing; although, keeping a naming schema and having Meteor only use npm packages within the naming schema is a much better aproach to achieve this.

For the same reason jquery-isotope is called jquery-isotope and not isotope, the former makes it clear it is part of jquery since its a plugin whereas the latter gives you a more meaningless isotope for which you would need more than just the package name to figure out what it does.

@awatson1978
@mpj
mpj commented Dec 18, 2012

Using a meteor- prefix seems a bit hacky. How would a package-as-npm-module define what version of Meteor it requires? That seems like a pretty fundamental requirement for any package management for Meteor.

@substack

I've been writing modules that work in the server and in node for a while now.

But even if one thinks that using two package managers for the framework is a good idea,
it would still be suboptimal when you use npm modules. While many npm modules would,
 of course, work fine, it would be an unpredictable experience for the developer - You
would not be able to simply require any npm module, because there is a high chance that
a module makes use of native node functionality, or other stuff not widely available in
browsers (such as Array.forEach, for instance, which breaks in IE 8).

True enough! I think the answer to this problem is not so much to have a manually curated set of "known working" packages, but an automated system for verifying that packages will work in both server and browser environments. It just so happens that I've built a tool with @pkrumins that does exactly this: http://ci.testling.com

For modules on npm that work in browsers, like falafel:
https://npmjs.org/package/falafel
https://github.com/substack/node-falafel

testling-ci runs the browser tests on every commit and generates badges to show browser compatibility:
http://ci.testling.com/substack/node-falafel
browser status

testling-ci uses browserify to compile tests that use the "files" parameter but there is a "scripts" parameter that works if you've got non-commonjs modules or a custom kind of build output.

Perhaps instead of package curation, packages could just pass a test suite that would encompass both browser and server tests? This approach would be useful whether or not meteor packages live on npm although I am in favor of authors publishing their meteor packages to npm too.

@awatson1978
@TomWij
TomWij commented Dec 18, 2012

Using a meteor- prefix seems a bit hacky.

@mpj: It's just one way to do categorization, a totally different and less hacky approach is if Meteor hosts its own npm repository; which is not that hard to do: http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository

How would a package-as-npm-module define what version of Meteor it requires?

@mpj: It's as easy as making Meteor a dependency of the npm module.

For instance, I just recently added stylus support to my application with a 'meteor add stylus' command. It worked great. No fuss, and I was up and running in like 5 minutes. With the proposed NPM system, one would need to do something like 'npm install meteor-stylus'.

@awatson1978: Well, you could just let the meteor add stylus call npm install meteor-stylus; or when they host their own private repository that could call npm --repository SOME_URL install stylus, see above.

But I guarantee what's going to happen is that people are going to run 'npm install stylus' instead, and get the generic stylus package, instead of the meteor specific one.

@awatson1978: It would be clear to people what smart packages are and that they are meteor specific, so that shouldn't really form a problem unless one really intends to hide this train of thoughts. This problem would not exist when npm hosts its own private repository, see above.

@mpj
mpj commented Dec 18, 2012

@mpj: It's as easy as making Meteor a dependency of the npm module.

Oh, npm handles circular dependencies? I did not know that.

@TomWij
TomWij commented Dec 18, 2012

Oh, npm handles circular dependencies? I did not know that.

What circular dependency are you even talking about?

@mpj
mpj commented Dec 19, 2012

What circular dependency are you even talking about?

Two-way is probably a better way to describe it. Meteor itself is, to a very large extent, built on top of smart packages, which in turn depends on Meteor. I.e. it's a two-way relationship. The parent relies on having children of a certain version, and those only work if the parent is a certain version.

@TomWij
TomWij commented Dec 19, 2012

Two-way is probably a better way to describe it.

Yes, it doesn't involve a third. So the best way would be to just test whether two-way dependencies introduce a problem. I honestly don't think so and if it would we could drop the dependency from Meteor on the smart packages and have Meteor come with them as a default (or refuse to run without certain run-time dependencies present).

@juliangruber

do smart packages need meteor code or rather other smart packages?

node loads modules lazily, so circular dependencies usually aren't a problem

@TomWij
TomWij commented Dec 19, 2012

do smart packages need meteor code or rather other smart packages?

Yes, a lot what Meteor provides (and isn't a meteor ... function) is implemented in the smart packages, the smart packages thus have dependencies between each other and are dependent on some Meteor base code as well.

@smhg
smhg commented Jan 29, 2013

Was something decided related to this discussion?

I strongly support using existing standards (@TomWij 's view).

However, in the end it might make sense to manage Meteor packages on a new level? In the way that it makes sense that npm is one level below, for instance, apt or yum. They are indeed useless for other node projects and have a shared installation logic.
To avoid duplication: couldn't the Meteor package manager (aka Meteorite) be based on/forked from npm? Meaning it would reuse the "higher" level logic (for dependencies, versioning, repository structure,...).

In this light, Meteor itself should definitely be installable through npm.

@awatson1978

So, in the past month, I've come up with probably a dozen Smart Packages that I'm planning on building and releasing into the Meteor/Meteorite package manager. They include:

fonts-pictographic
fonts-handwritten
ui-scaffolding
ui-gestures
ui-header
ui-footer
ui-keybindings
ui-drag-and-drop
ui-jquery-tools
files-upload
files-gridfs
d3-calendaring
d3-statistics
d3-networks
foundation-stylus
icons-file-management
icons-social-media

I'm thinking of creating these Meteor Smart Packages, because I do a lot of prototyping of applications. And I'm constantly doing things like building out a basic application scaffolds, but sometimes I want headers and footers and interactive graphing, and other times I just want to simply prototype how gestures might work with a carousel interactive. I want to be able to do a simple

meteor add ui-scaffold
meteor add ui-header
meteor add d3
meteor add d3-statistics

and wind up with a workable linear regression application, using data from a Mongo datastore. That would rock. Might not be nirvana, but it would be close to it. Moreover, once boiler plate application code is verified and written, it can be encapsulated and packaged into a Smart Package, which also removes it from the active development space, keeping the code that a developer has to manage clean and concise. Meteor Smart Packages offer a way to do that, at the application level (rather than the operating system level).

But would it make sense to add those packages to NPM? At the very least, they'd all need to be renamed to something like the following:

meteor-fonts-pictographic
meteor-fonts-handwritten
meteor-ui-scaffolding
meteor-ui-gestures
meteor-ui-header
meteor-ui-footer
meteor-ui-keybindings
meteor-ui-drag-and-drop
meteor-ui-jquery-tools
meteor-files-upload
meteor-files-gridfs
meteor-d3-calendaring
meteor-d3-statistics
meteor-d3-networks
meteor-foundation-stylus
meteor-icons-file-management
meteor-icons-social-media

And I'm not sure how much it adds to Node.js to have packages specifying the styling of headers and footers and icon packs. And lastly, if using standards is so great, why use NPM at all? By this logic, shouldn't the Node.js folks be porting their code to Apt and Yum?

@smhg
smhg commented Jan 29, 2013

If this is in response to my questions: did you read what I wrote? :)

As an answer to your question at the end:
Should apt and yum be JavaScript based, it might be very smart to somehow build npm on top of them, reusing things like dependency management, versioning, testing,... whatever they both need to do. Of course they are not JavaScript.
Since the Meteor package manager is written in JavaScript and, as far as my limited knowledge of these goes, basically needs to do everything npm does in addition to meteor related stuff, it sounds logical to avoid building dependency management, versioning, testing,... again in the Meteor package manager.

The end result would be:

  • npm install meteor
  • meteor create MyApp
  • meteor install bootstrap (with "install" being "inherited" from the underlying npm)

I rest my case if npm is not reusable.

@TomWij
TomWij commented Jan 29, 2013

@awatson1978:

At the very least, they'd all need to be renamed to something like the following:

Not necessarily, because a separate npm repository can be made and packages can be installed to a separate non-global node_modules folder.

And I'm not sure how much it adds to Node.js to have packages specifying the styling of headers and footers and icon packs.

They don't necessarily need to specify that as far as I can see, not sure how this is relevant.

And lastly, if using standards is so great, why use npm at all?

Because npm puts the maintenance of a package in the hands of the people.

By this logic, shouldn't the Node.js folks be porting their code to Apt and Yum?

No, this would require people to become maintainers / developers for Aptitude, Yum, Portage and so on...

npm is this regard centralizes it to one repository instead of many distribution repositories, smart packages can then be installed on any distribution that supports npm instead of those that support the smart packages. Smart packages can also easily be added to npm without having to deal with becoming a maintainer / developer, or the use of vendor-specific patching.

Wrt smart packages; this is not about getting smart packages into the main npm repository, this is about not reinventing the wheel by re-using npm. If they do want to reinvent the wheel and write their own non-npm based repository and package manager, they can go ahead and do that; although it can be considered a waste of time better spent on other features. Unless they have a very good reason not to use npm and thus roll their own...

@gschmidt
Meteor Development Group member

@avital has been working hard on a new package system, and will be posting more about it soon! It will have npm support :)

Some considerations in package system design:

  • Client vs server. We need something that works on both the client and the server. npm's CommonJS-based approach (require()/exports) has clearly won on the server side. (Though presumably once the ECMAScript Modules proposal is available, node will want to switch away from require()/exports to something based on modules.) But -- on the browser side, which actually has a much larger install base and a larger amount of experience behind it, there has been a ton of experimentation and innovation. There's @substack's Browserify but there's also Twitter's Bower, component from TJ Holowaychuk, Ender, and others (I hear Volo and Jam are popular.) Meteor should have one thing that works everywhere, seamlessly. And we should take into account the experience of teams like Twitter's in maintaining complicated sites with large teams, rather than assuming that since CommonJS works well on the server, it'll work well on the client.

  • Asset building and bundling. Meteor apps (and packages) contain more than just JavaScript. They contain assets like HTML, CSS, and images, and they contain resources that need to have a build step applied to them, like CoffeeScript and LESS/SASS stylesheets. The core 'meteor' command-line tool is a hybrid of a package system like rpm or npm and a build tool like make or SCons. Meteor packages need to be able to define build rules (like LESS preprocessing, or compiling other languages to JavaScript), and they need to be able to inject resources into the client bundle. In other words, one way or another, the package system needs to integrate nicely with Meteor's asset building/bundling system. (Of course this doesn't rule out defining a special format for "Meteor npm packages" and distributing them through npm.)

  • Curation and the Paradox of Choice. We want a two-level namespace, where packages that are "safe bets" (widely used, mature, supported, tested) get a short name like "forms", and experimental, up-and-coming, or personal packages get a name prefixed with the author's name, like "gschmidt:forms". (And are then promoted when/if they mature and get mindshare.) This is to address the "paradox of choice" you see in the npm ecosystem, where there could be dozens of packages that do the same thing, but only two are really supported, and you have to do an hour or two of research to figure out which one to pick. Also, fragmentation has a cost. When possible, it'd be great to encourage the community to rally around 1 or 2 alternatives for any given task, rather than splitting our efforts.

  • Official, tested release snapshots. Right now Meteor works in terms of releases. When you update to Meteor 0.5.0, you're pulling a new set of core packages, which are guaranteed to work together (because we've tested them, a lot!) We want to keep this experience, where you don't have to micromanage the versions of all of your packages, and where "meteor update" always gives you a consistent known-good snapshot. This is not a hypothetical concern.. when we do the testing for each release, we often find problems in upstream npm packages that we have to patch or work around, and if paid, fulltime MDG staff doesn't do this work, once, then we're forcing everyone else to do it, O(n) times :) Of course, it should still be possible to override the default package versions in each release and use whatever you want, accepting the release engineering burden yourself. But we think that most people (especially people that have high-value sites in production) will want to use tested/QA'd Meteor snapshots most of the time. Essentially we want to try out an Ubuntu-style model where more of the release engineering work is taken on by a central party (at least for some set of "most critical" packages.)

  • Repeatability. We feel strongly that every package in your app should be always pinned to a specific version, and those version pins should be checked into source control. Whenever you run your app, on any machine, you should get the exact same code -- until you type 'meteor update' and specifically ask to float all (or some) of your packages up to the latest available version (which will then be visible in source control, so you can see who did it and when, and back it out.) When new versions of your dependencies are released, you should be notified (just like with new versions of Meteor today), but your project shouldn't be moved to those new versions until you ask for it. In production, depending on a floating version of a package from a third party is just asking for trouble. Imagine if a buggy version of underscore was pushed, and it took down every website that depended directly or indirectly on underscore, with no way to see what happened or revert. npm now has support for this with 'npm shrinkwrap', but it has some limitations, for example you can't update the version of just one package without taking new versions of all packages.

  • Deploy hooks and configuration. When you meteor add email and deploy your app to the public galaxy (meteor deploy), it autoprovisions a free Mailgun account for your app, so you can call Mail.send and it just works! I have configured SMTP far too many times in my life, and I find this a really nice convenience. Right now this hook is hardcoded into the Galaxy server. But in the future, there should be an open interface that lets any package define deploy-time hooks (and there should be a cleaner interface for managing deploy configuration, say when you want to use Sendgrid, an existing Mailgun account, or your existing SMTP server.)

  • Docs integration. Rather than have the "official Meteor documentation" for the core packages, and making you separately hunt down the docs for non-core packages that you're using, we want to create a level playing field for everyone. Eventually, the docs for a package (core or otherwise) should live inside that package, and you should be able to view the docs for all the packages you're using in one unified interface.

If you think about it, if we tried to build all of the above on top of npm, it would be npm in name only. Even if Meteor packages were in npm, you wouldn't use the npm tool to find or install packages (you'd use a wrapper that implemented the 'release snapshots', 'curation', 'repeatability' considerations), nor could you drop them directly into existing npm applications (per 'client vs server' and 'asset bundling').

What's really missing is the ability to use npm packages from Meteor packages, because since npm's functionality is a subset of Meteor's packages, including an npm package into the server side of your Meteor app can be as easy as calling Npm.require() (with Meteor doing npm shrinkwrapping behind the scenes..) This is coming very soon!

It would also be highly desirable to figure out how to use certain Meteor packages outside of Meteor. The two obvious cases right now are using livedata (DDP client and server) from npm-based apps (such as the meteor command line tool!), and using spark from the browser jQuery-style. AFAICT, the best route here is to make it easy to dual-publish these packages into other package systems without their Meteor-specific features (either in npm or in your client-side browser framework of choice, or as a jQuery-style self-contained download.) Today this is done in an ad-hoc way with shell scripts.

@glasser
Meteor Development Group member
glasser commented Apr 24, 2013

@substack: Before we can merge your pull request, you'll need to sign the Meteor Contributor Agreement: https://contribute.meteor.com/

@grigio
grigio commented May 2, 2013

+1 I think Meteor should integrate better some emerging "JS standard" technlogies like NPM, Grunt, Bower ( RequireJS ), because many frontend js libs like jquery, bootstrap,.. are already providing this kind of support mainstream.

@meteor-bot

@substack: Please do not create pull requests based on the master branch of the meteor repository. Typically, pull requests should be based on devel. See https://github.com/meteor/meteor/wiki/Contributing-to-Meteor for more suggestions on contributing to Meteor.

@meteor-bot meteor-bot closed this Dec 7, 2013
@dandv
dandv commented Nov 30, 2014

Hey @substack, any chance to revive this project? I've been leading efforts to get the maintainers of popular libraries to add support for meteor directly, and the #1 complaint from d3 and Moment was that Meteor required them to curl https://install.meteor.com | sh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.