global npm packages (Fixes #39)

[seanewest]

Fixes #39

I implemented a minimalist solution for global npm packages:

ndm <cmd> -g <package>

ndm generate -g <package>
ndm interview -g <package>
ndm start -g <package>
ndm remove -g <package>
#etc ...

If you want to see this interface in action, I'm using it for my project npmfs

I see a couple of benefits in doing it this way:

• looks just like npm <cmd> -g <package> (but it can't do ndm -g <cmd> <package>)
• avoids name collision with npm start <local-service>
• relatively simple change

There are three main things currently missing (at least from how this feature was discussed):

• the syntax ndm <cmd> <package> is not supported
• ndm start <package> currently runs just like it does for local ndm wrappers and does not shortcut the interview -> generate -> start process.
• It uses another flag --prefix which defaults to /usr/local to determine the location of npm modules, rather than programatically asking npm. The npm module requires an async load method which I was having a hard time working in with the currently synchronous initialization phase.

I'd like to hear your feedback and suggestions.

[bcoe]

@seanewest I LOVE the simplicity of this approach, a couple thoughts about how we could make things even more powerful:

• I think I'd like the flag -e and --explore better than -g. This better matches the npm client, which I think would help people understand the feature better:
  • https://www.npmjs.org/doc/cli/npm-explore.html
• A few thoughts about the prefix variable.
  • I'd be tempted to rename it to something slightly more specific, like module-prefix.
  • I think it would be cool if rather than checking a single prefix, we had logic for explore that was somewhat fault-tolerant, perhaps we could walk an array like this:

// begin by looking in the local ./node_modules folder, then look in common system paths.
// we should still respect the module-prefix variable.
["./", "/usr", "/usr/local"]

• I think if a non-local ./node_modules is used, e.g., /usr/local, we should think about using a different default directory for logging. Perhaps /var/logs?

Awesome feature 💯 did you try it out with any packages?

[seanewest]

Ok cool. How does this sound:

• rename --global to --explore
• rename --prefix to --module-prefix
• discover packages looking in the following directories in order:
  • <module-prefix>/node_modules or <module-prefix>/lib/node_modules (if --module-prefix set by user)
  • require.resolve(package_name) (./node_modules or node_modules of any parent)
  • /usr/local/lib/node_modules
• If non-local ./node_modules is used then redirect the logs

I noticed that /var/log has root permissions (at least on my macbook air). Could we redirect global logging to /usr/local/var/log or <module-prefix>/var/log?

[seanewest]

Are there any other directories that one might put packages in?

[bcoe]

on Ubuntu, it's /usr/lib/node_modules I think. I think we should just make it an array of paths, with some special logic for handling, have defaults been overridden with --module-prefix?

I'm not quite sure what the best practice would be for logs, on Linux machines /var/log is the right place. We could make it an OS-specific variable, where are other .plist files logging?

[seanewest]

On my mac ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist logs to /usr/local/var/postgres/server.log.

  <key>WorkingDirectory</key>
  <string>/usr/local</string>
  <key>StandardErrorPath</key>
  <string>/usr/local/var/postgres/server.log</string>

I tried logging to /var/log/service.log from a .plist and it would only work after changing the permissions to that log file. I also tried running ndm generate -d to get it to log to that directory but to no avail.

[seanewest]

I also noticed a ~/Library/Logs/ that has a lot of logs for Mac apps (but not services) such as Facetime, Homebrew, Spotify, GoogleSoftwareUpdateAgent

[bcoe]

Let's punt on the logs for a bit, and get everything else working; I can dig into the permissions issues with logs on OSX/Centos/Ubuntu when I get a moment.

[bcoe]

Haven't forgotten about this @seanewest, been swamped with ops work this week. Let's get this feature shipped this weekend.

[seanewest]

Sounds good to me. I'm free Saturday and Sunday to work on this.

On Thursday, August 7, 2014, Benjamin E. Coe notifications@github.com
wrote:

Haven't forgotten about this @seanewest https://github.com/seanewest,
been swamped with ops work this week. Let's get this feature shipped this
weekend.

—
Reply to this email directly or view it on GitHub
#48 (comment).

[rmg]

Background: A couple weeks ago I wrote https://github.com/strongloop/strong-service-install so that I could add an install command to another module. For testing purposes, I threw a minimist wrapper around it that let it install upstart wrappers for arbitrary executables.

For your prefix query, if the global module in question has a "binary" associated with it, could you inspect the symlink to determine the module's location and from there find the metadata?

[bcoe]

@rmg I like the idea of using the symlink to determine the module directory, that's what we should default to :)

As for deploying services with ndm, from a globally installed module, I was picturing this workflow:

1. install the module foo with npm.
2. optionally run ndm interview --explore foo, which will look at a service.json checked in with the module and ask a user questions about ENV variables that need to be set.
3. run ndm generate --explore foo to create the upstart scripts.
4. run ndm start|stop|restart --explore foo.

Any thoughts about simplifying this flow?

[rmg]

From the bottom up:

• once the Upstart job is installed, why does ndm need to be used to start/stop?
• generate and interview seem like they would most often be run together, why not make them one command? (install seems an obvious choice to me)
• if the npm install -g foo results in a foo command on the path requiring --explore seems like unnecessary typing

Actually, overall I don't really get the --explore requirement.. is it to save specifying the full path to the service.json? Looks really annoying to have to keep on specifying it. I guess it's not such a pain if ndm is called from scripts, but in that case you're already in write-your-own-ndm land anyway I think.

[bcoe]

ndm grows out of my need to run many services at the same time in a service oriented architecture. It's nice that start and stop can launch an entire set of dependent services. It's also nice that start and stop standardizes the verbs used to run services:

• on Centos it's initctl start foo.
• on OSX it's `launchctl [path-to-foo.plist].
• on Ubuntu it's service foo start.

This gets confusing, hence the abstraction.

• I'm also not a huge fan of the --explore flag, maybe we could figure out a way to handle the global package use-case without explore. Maybe it would be enough to have ndm install foo-package and if no service.json is present in the current directory, use the explore behavior.
• I like the idea of hiding some of the ndm specifics by embedding it into "installable packages".
• I like the idea of the A LOT of adding an install command that runs both generate and interview 💯.

[rmg]

For Ubuntu and CentOS using Upstart like ndm does, start foo should work on both. The OS X one is a pain, I agree, and I would actually use ndm start for it.

For service orchestration, if the services are on the same machine Upstart lets you use other jobs as start/stop triggers. That's how foreman (and node-foreman) generate top level jobs that kick off the other services.

You'd create a top level job, say /etc/init/my-service.conf that looks like:

description "All the services"
start on runlevel [235]
stop on runlevel [016]

And then your sub-services, /etc/init/my-dep-service.conf:

description "Very important micro-service"
start on my-service
exec /my/bin/here

And another, /etc/init/my-other-dep-service.conf:

description "Another very important micro-service"
start on my-service
exec /my/bin/here

Then you'd just do start my-service and you're off to the races.

It's a pretty natural progression to generate that from a Procfile, but it's not so clear to me how ndm could pull in service inter-dependencies to model the same thing.

[bcoe]

@rmg I've made the conscious decision not to go down the path of implementing an orchestration tool for a set of interdependent services.

ndm helps you share configuration between, and boot multiple isolated web-services.

[rmg]

@bcoe I missed the multiple-serivces part of services.json on first look. I understand now.

The generation of a top-level Upstart job would still work. The theoretical benefit is that $init becomes aware of the dependencies and "do the right thing", but I realize that doesn't actually translate to anything real (except in Windows Services land, I suppose).

[bcoe]

@rmg I think this is worth talking about more. Here's what I'm thinking:

• Let's create a ticket for self-installable services which I think is a killer feature, and we can get built pretty fast.
• Let's continue the talk about using upstart to compose together services, I know @groundwater has some thoughts on this too.

[rmg]

Coincidentally, I took over node-foreman from @groundwater a few months back, and we happened to meet in person just last night.

[bcoe]

@seanewest I'm going to be hacking on ndm this afternoon and tonight, I'll hop in the npm IRC channel 👍

[seanewest]

@bcoe roger

[bcoe]

Here's what I'm thinking, let's make it so you can pass a service name to every command:

ndm generate foo
ndm start foo
ndm remove foo

If no service.json is found in the current working directory, look for the package in ./node_modules., /usr/lib/node_modules, etc.

[seanewest]

I think that sounds good. My pull request has a couple of commits that try to resolve the ubuntu/mac os separate prefix paths and also changes the prefix flag to module-prefix.

So then I was thinking the next changes would be to remove the explore flag and instead use explore internally when we can't find a local service.json.

[bcoe]

@seanewest I've created a global branch on npm/ndm, which you also have access to. I'm in the process of refactoring allServices so that we can make it smarter about finding service.json. My thinking is that:

1. we have all commands go through allServices(serviceNameFilter).
2. if no service.json if found, we check a chain of logical places for it ./node_modules/, ../node_modules, etc.
3. if allServices returns a service from a different path, we modify baseWorkingDirectory accordingly . what do you think?