Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Npm exec implementation for #3313 #4058

Closed
wants to merge 5 commits into from

6 participants

@can3p

Hi

I've extracted the exec code from the lifecycle.js and added an npm exec command to run scripts in the package context. In part it fixes #3313. In reality I couldn't eliminate lifecycle at all, because it contains some specific logic like hooks.

I know that @grncdr works on the same task, but it happened that I've made this work too and I'll be happy if it will be useful.

I'm ready to fix things if needed.

@grncdr

I'm not too attached to my implementation, and haven't had a lot of spare time to work on it. I may come back to it later, but I got distracted by my desire to have a more faithful sh interpreter. For me, that's the most important part of what I was doing, as it would allow scripts (and exec commands) to be portable to environments that don't include a posix shell (windows).

@can3p

I'm not sure that I got your point.

When I was working on the implementation my intention was to implement exec command in such a way that it will run the commands in the same context as run-script did but allow me to specify the arguments for the command.

If you're talking about the same then what will be different on windows in this case?

@grncdr

It turns out I was quite ignorant of what windows cmd.exe supports. I didn't know it already supported command chaining with boolean operators (||, &&) and output redirection, if that's the case there's really less of a need for the bashful integration I was working on and I don't see any compelling reason to prefer my patch.

@grncdr

If you're talking about the same then what will be different on windows in this case?

I was mostly talking about more complicated scripting support such as command line chaining, output redirection etc. (See http://substack.net/task_automation_with_npm_run for examples of the kinds of things that I'd like "scripts" to support cross-platform). In the interest of getting those sorts of scripts to work in windows (without having to require an out-of-band dependency on bash) I integrated bashful into my npm-exec repo and was using that in npm to replace most of the lifecycle code (much like what you've done here).

However, bashful is not complete, and I was working on making it more complete when you opened this PR. Seeing that windows cmd.exe supported some useful shell operators, I started to think that maybe the bashful integration was not as useful/important after all.

But today I found out that cmd.exe doesn't support backgrounding jobs with &, and you can't have multiple commands separated with a semicolon. (E.g. try echo hello; echo hello2 in both cmd.exe and any posix shell). IMO these are common enough operations that they justify using bashful (or something like it) to implement npm exec (and run npm scripts etc) so I'm going to start back up on bashful.

@can3p

@isaacs, can you define a functionality that is sufficient in this case?

@ricardobeat ricardobeat referenced this pull request
Closed

Add command npm-exec #4554

@domenic
Collaborator

I really want this. Hopefully we can put this at the top of @isaacs's priority queue to review.

@jeffmo

I also really want this. What can I do to help move it forward?

@mzgol

This would be a godsend to Grunt users. It would make it extremely easy to migrate from grunt task to npm run task; no more excuses to install anything globally!

@othiym23
Owner

/cc @bcoe There's a bunch of useful stuff that could be incorporated from this into npm-exec as well. @can3p, we sort of routed around this PR in favor of #5518, but there's a lot of good stuff in here! Thanks for the work!

@othiym23 othiym23 added the npm-exec label
@othiym23
Owner

This is another patch that got overtaken by events elsewhere in the npm project. I'm sorry we let it sit for so long. A few things have happened:

  • npm@2 added the ability to pass arguments to commands defined in "scripts"
  • a few of us agreed that it would be handy for npm to include a separate command, npm-exec, that allows you to run commands in the environment provided to child processes by run-script, but without all of the command-line argument parsing that regular npm does. This is in part because decoupling that argument parsing from npm for a single command was deemed sufficieintly complicated that it made more sense to put in a separate command.
  • many tweaks have been made to npm run-script since the release of npm@2, so this has drifted further from where the code is now.

There is clearly still demand for an npm-exec, so if you wanted to take this code and use it as the basis for that and open it as a new PR, that would be awesome. :fireworks: This patch, however, is dead – npm exec isn't something that makes sense right now, and the code here and the code in master have gone their separate ways. I apologize for letting this patch sit so long without feedback from the core team, and I thank you for taking the time to put together such a complete patch.

@othiym23 othiym23 closed this
@othiym23 othiym23 removed the review label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 16, 2013
  1. @can3p

    initial exec stub

    can3p authored
Commits on Oct 18, 2013
  1. @can3p
Commits on Oct 25, 2013
  1. @can3p
Commits on Oct 26, 2013
  1. @can3p
  2. @can3p

    add remove verbose message

    can3p authored
This page is out of date. Refresh to see the latest.
Showing with 283 additions and 221 deletions.
  1. +24 −0 lib/exec.js
  2. +1 −0  lib/npm.js
  3. +29 −221 lib/utils/lifecycle.js
  4. +229 −0 lib/utils/pkgexec.js
View
24 lib/exec.js
@@ -0,0 +1,24 @@
+module.exports = exec
+
+var path = require("path")
+ , pkgexec = require('./utils/pkgexec')
+ , chain = require("slide").chain
+ , log = require("npmlog")
+ , readJson = require("read-package-json")
+
+exec.usage = "npm exec <command> [<options>]"
+
+function exec(args, cb) {
+ if (args.length < 1) return cb(exec.usage);
+
+ var command = args[0]
+ , cmd = args.slice(1)
+ , wd = process.cwd()
+
+ log.verbose("exec ", command)
+
+ chain([
+ [readJson, path.resolve(wd, "package.json")]
+ , [pkgexec, command, cmd, chain.last, wd]
+ ], cb)
+}
View
1  lib/npm.js
@@ -161,6 +161,7 @@ var commandCache = {}
, "restart"
, "run-script"
, "completion"
+ , "exec"
]
, plumbing = [ "build"
, "unbuild"
View
250 lib/utils/lifecycle.js
@@ -1,26 +1,12 @@
-
exports = module.exports = lifecycle
exports.cmd = cmd
var log = require("npmlog")
- , spawn = require("child_process").spawn
, npm = require("../npm.js")
+ , pkgexec = require("./pkgexec.js")
, path = require("path")
, fs = require("graceful-fs")
, chain = require("slide").chain
- , constants = require("constants")
- , Stream = require("stream").Stream
- , PATH = "PATH"
-
-// windows calls it's path "Path" usually, but this is not guaranteed.
-if (process.platform === "win32") {
- PATH = "Path"
- Object.keys(process.env).forEach(function (e) {
- if (e.match(/^PATH$/i)) {
- PATH = e
- }
- })
-}
function lifecycle (pkg, stage, wd, unsafe, failOk, cb) {
if (typeof cb !== "function") cb = failOk, failOk = false
@@ -45,56 +31,25 @@ function lifecycle (pkg, stage, wd, unsafe, failOk, cb) {
return cb()
}
- // set the env variables, then run scripts as a child process.
- var env = makeEnv(pkg)
- env.npm_lifecycle_event = stage
- env.npm_node_execpath = env.NODE = env.NODE || process.execPath
- env.npm_execpath = require.main.filename
-
- // "nobody" typically doesn't have permission to write to /tmp
- // even if it's never used, sh freaks out.
- if (!npm.config.get("unsafe-perm")) env.TMPDIR = wd
-
- lifecycle_(pkg, stage, wd, env, unsafe, failOk, cb)
- })
-}
+ log.verbose("unsafe-perm in lifecycle", unsafe)
-function checkForLink (pkg, cb) {
- var f = path.join(npm.dir, pkg.name)
- fs.lstat(f, function (er, s) {
- cb(null, !(er || !s.isSymbolicLink()))
+ lifecycle_(pkg, stage, wd, failOk, cb)
})
}
-function lifecycle_ (pkg, stage, wd, env, unsafe, failOk, cb) {
- var pathArr = []
- , p = wd.split("node_modules")
- , acc = path.resolve(p.shift())
-
- // first add the directory containing the `node` executable currently
- // running, so that any lifecycle script that invoke "node" will execute
- // this same one.
- pathArr.unshift(path.dirname(process.execPath))
-
- p.forEach(function (pp) {
- pathArr.unshift(path.join(acc, "node_modules", ".bin"))
- acc = path.join(acc, "node_modules", pp)
- })
- pathArr.unshift(path.join(acc, "node_modules", ".bin"))
+function lifecycle_ (pkg, stage, wd, failOk, cb) {
+ var packageLifecycle = pkg.scripts && pkg.scripts.hasOwnProperty(stage)
+ , env = {}
- // we also unshift the bundled node-gyp-bin folder so that
- // the bundled one will be used for installing things.
- pathArr.unshift(path.join(__dirname, "..", "..", "bin", "node-gyp-bin"))
+ if (!packageLifecycle) return cb()
- if (env[PATH]) pathArr.push(env[PATH])
- env[PATH] = pathArr.join(process.platform === "win32" ? ";" : ":")
+ // define this here so it's available to all scripts.
+ env.npm_lifecycle_event = stage
+ env.npm_lifecycle_script = pkg.scripts[stage]
- var packageLifecycle = pkg.scripts && pkg.scripts.hasOwnProperty(stage)
-
- if (packageLifecycle) {
- // define this here so it's available to all scripts.
- env.npm_lifecycle_script = pkg.scripts[stage]
- }
+ // "nobody" typically doesn't have permission to write to /tmp
+ // even if it's never used, sh freaks out.
+ if (!npm.config.get("unsafe-perm")) env.TMPDIR = wd
if (failOk) {
cb = (function (cb_) { return function (er) {
@@ -110,184 +65,37 @@ function lifecycle_ (pkg, stage, wd, env, unsafe, failOk, cb) {
}})(cb)
}
- chain
- ( [ packageLifecycle && [runPackageLifecycle, pkg, env, wd, unsafe]
- , [runHookLifecycle, pkg, env, wd, unsafe] ]
- , cb )
+ runPackageLifecycle(pkg, env, wd, cb)
}
-function validWd (d, cb) {
- fs.stat(d, function (er, st) {
- if (er || !st.isDirectory()) {
- var p = path.dirname(d)
- if (p === d) {
- return cb(new Error("Could not find suitable wd"))
- }
- return validWd(p, cb)
- }
- return cb(null, d)
- })
-}
-
-function runPackageLifecycle (pkg, env, wd, unsafe, cb) {
+function runPackageLifecycle (pkg, env, wd, cb) {
// run package lifecycle scripts in the package root, or the nearest parent.
var stage = env.npm_lifecycle_event
- , user = unsafe ? null : npm.config.get("user")
- , group = unsafe ? null : npm.config.get("group")
- , cmd = env.npm_lifecycle_script
- , sh = "sh"
- , shFlag = "-c"
-
- if (process.platform === "win32") {
- sh = "cmd"
- shFlag = "/c"
- }
+ , cmd = env.npm_lifecycle_script.split(/\s+/)
+ , sh = cmd.shift()
+ , args = cmd
- log.verbose("unsafe-perm in lifecycle", unsafe)
-
- var note = "\n> " + pkg._id + " " + stage + " " + wd
- + "\n> " + cmd + "\n"
-
- console.log(note)
-
- var conf = { cwd: wd, env: env, customFds: [ 0, 1, 2] }
- var proc = spawn(sh, [shFlag, cmd], conf)
- proc.on("close", function (er, stdout, stderr) {
+ pkgexec(sh, args, pkg, wd, env, function(er) {
if (er && !npm.ROLLBACK) {
- log.info(pkg._id, "Failed to exec "+stage+" script")
- er.message = pkg._id + " "
- + stage + ": `" + env.npm_lifecycle_script+"`\n"
- + er.message
- if (er.code !== "EPERM") {
- er.code = "ELIFECYCLE"
- }
- er.pkgid = pkg._id
er.stage = stage
- er.script = env.npm_lifecycle_script
- er.pkgname = pkg.name
return cb(er)
- } else if (er) {
- log.error(pkg._id+"."+stage, er)
- log.error(pkg._id+"."+stage, "continuing anyway")
- return cb()
}
- cb(er)
- })
-}
-
-function runHookLifecycle (pkg, env, wd, unsafe, cb) {
- // check for a hook script, run if present.
- var stage = env.npm_lifecycle_event
- , hook = path.join(npm.dir, ".hooks", stage)
- , user = unsafe ? null : npm.config.get("user")
- , group = unsafe ? null : npm.config.get("group")
- , cmd = hook
- fs.stat(hook, function (er) {
- if (er) return cb()
-
- var conf = { cwd: wd, env: env, customFds: [ 0, 1, 2] }
- var proc = spawn("sh", ["-c", cmd], conf)
- proc.on("close", function (er) {
- if (er) {
- er.message += "\nFailed to exec "+stage+" hook script"
- log.info(pkg._id, er)
- }
- if (npm.ROLLBACK) return cb()
- cb(er)
- })
- })
+ cb()
+ });
}
-function makeEnv (data, prefix, env) {
- prefix = prefix || "npm_package_"
- if (!env) {
- env = {}
- for (var i in process.env) if (!i.match(/^npm_/)) {
- env[i] = process.env[i]
- }
-
- // npat asks for tap output
- if (npm.config.get("npat")) env.TAP = 1
-
- // express and others respect the NODE_ENV value.
- if (npm.config.get("production")) env.NODE_ENV = "production"
-
- } else if (!data.hasOwnProperty("_lifecycleEnv")) {
- Object.defineProperty(data, "_lifecycleEnv",
- { value : env
- , enumerable : false
- })
- }
-
- for (var i in data) if (i.charAt(0) !== "_") {
- var envKey = (prefix+i).replace(/[^a-zA-Z0-9_]/g, '_')
- if (i === "readme") {
- continue
- }
- if (data[i] && typeof(data[i]) === "object") {
- try {
- // quick and dirty detection for cyclical structures
- JSON.stringify(data[i])
- makeEnv(data[i], envKey+"_", env)
- } catch (ex) {
- // usually these are package objects.
- // just get the path and basic details.
- var d = data[i]
- makeEnv( { name: d.name, version: d.version, path:d.path }
- , envKey+"_", env)
+function validWd (d, cb) {
+ fs.stat(d, function (er, st) {
+ if (er || !st.isDirectory()) {
+ var p = path.dirname(d)
+ if (p === d) {
+ return cb(new Error("Could not find suitable wd"))
}
- } else {
- env[envKey] = String(data[i])
- env[envKey] = -1 !== env[envKey].indexOf("\n")
- ? JSON.stringify(env[envKey])
- : env[envKey]
- }
-
- }
-
- if (prefix !== "npm_package_") return env
-
- prefix = "npm_config_"
- var pkgConfig = {}
- , keys = npm.config.keys
- , pkgVerConfig = {}
- , namePref = data.name + ":"
- , verPref = data.name + "@" + data.version + ":"
-
- keys.forEach(function (i) {
- if (i.charAt(0) === "_" && i.indexOf("_"+namePref) !== 0) {
- return
- }
- var value = npm.config.get(i)
- if (value instanceof Stream || Array.isArray(value)) return
- if (!value) value = ""
- else if (typeof value !== "string") value = JSON.stringify(value)
-
- value = -1 !== value.indexOf("\n")
- ? JSON.stringify(value)
- : value
- i = i.replace(/^_+/, "")
- if (i.indexOf(namePref) === 0) {
- var k = i.substr(namePref.length).replace(/[^a-zA-Z0-9_]/g, "_")
- pkgConfig[ k ] = value
- } else if (i.indexOf(verPref) === 0) {
- var k = i.substr(verPref.length).replace(/[^a-zA-Z0-9_]/g, "_")
- pkgVerConfig[ k ] = value
- }
- var envKey = (prefix+i).replace(/[^a-zA-Z0-9_]/g, "_")
- env[envKey] = value
- })
-
- prefix = "npm_package_config_"
- ;[pkgConfig, pkgVerConfig].forEach(function (conf) {
- for (var i in conf) {
- var envKey = (prefix+i)
- env[envKey] = conf[i]
+ return validWd(p, cb)
}
+ return cb(null, d)
})
-
- return env
}
function cmd (stage) {
View
229 lib/utils/pkgexec.js
@@ -0,0 +1,229 @@
+module.exports = pkgexec;
+
+var npm = require("../npm.js")
+ , log = require("npmlog")
+ , fs = require('fs')
+ , path = require('path')
+ , chain = require("slide").chain
+ , spawn = require("child_process").spawn
+ , Stream = require("stream").Stream
+ , PATH = "PATH"
+
+// windows calls it's path "Path" usually, but this is not guaranteed.
+if (process.platform === "win32") {
+ PATH = "Path"
+ Object.keys(process.env).forEach(function (e) {
+ if (e.match(/^PATH$/i)) {
+ PATH = e
+ }
+ })
+}
+
+function pkgexec(sh, args, pkg, wd, env_vars, cb) {
+ if (typeof cb !== "function") cb = env_vars, env_vars = {}
+
+ pkgexec_(wd, sh, args, pkg, env_vars, cb)
+}
+
+function pkgexec_(wd, shell_command, args, pkg, env_vars, cb) {
+ var pathArr = []
+ , p = wd.split("node_modules")
+ , acc = path.resolve(p.shift())
+
+ // set the env variables, then run scripts as a child process.
+ var env = makeEnv(pkg)
+
+ // allow to pass env variables from outer code
+ for (var i in env_vars) {
+ env[i] = env_vars[i]
+ }
+
+ env.npm_node_execpath = env.NODE = env.NODE || process.execPath
+ env.npm_execpath = require.main.filename
+
+ // "nobody" typically doesn't have permission to write to /tmp
+ // even if it's never used, sh freaks out.
+ if (!npm.config.get("unsafe-perm")) env.TMPDIR = wd
+
+ // first add the directory containing the `node` executable currently
+ // running, so that any lifecycle script that invoke "node" will execute
+ // this same one.
+ pathArr.unshift(path.dirname(process.execPath))
+
+ p.forEach(function (pp) {
+ pathArr.unshift(path.join(acc, "node_modules", ".bin"))
+ acc = path.join(acc, "node_modules", pp)
+ })
+ pathArr.unshift(path.join(acc, "node_modules", ".bin"))
+
+ // we also unshift the bundled node-gyp-bin folder so that
+ // the bundled one will be used for installing things.
+ pathArr.unshift(path.join(__dirname, "..", "..", "bin", "node-gyp-bin"))
+
+ if (env[PATH]) pathArr.push(env[PATH])
+ env[PATH] = pathArr.join(process.platform === "win32" ? ";" : ":")
+
+ chain
+ ( [
+ [runCommand, wd, shell_command, args, env, pkg]
+ , env.npm_lifecycle_event &&
+ [runHookLifecycle, env.npm_lifecycle_event, pkg, env, wd]
+ ]
+ , cb )
+}
+
+function runCommand(wd, command, args, env, pkg, cb) {
+ var sh = "sh"
+ , shFlag = "-c"
+ , cmd
+ , cmdArgs
+
+ if (process.platform === "win32") {
+ sh = "cmd"
+ shFlag = "/c"
+ }
+
+ var note = "\n> " + pkg._id + " " + command + " " + wd
+ + "\n> " + command + ' ' + args.join(' ') + "\n"
+
+ console.log(note)
+
+ cmd = [command].concat(args).join(' ')
+
+ cmdArgs = [shFlag, cmd]
+
+ var conf = { cwd: wd, env: process.env, customFds: [ 0, 1, 2] }
+ var proc = spawn(sh, cmdArgs, conf)
+ proc.on("close", function (er, stdout, stderr) {
+ if (er && !npm.ROLLBACK) {
+ log.info(pkg._id, "Failed to exec "+command+" script")
+ er.message = pkg._id + " "
+ + command + ": `" + cmd
+ + er.message
+ if (er.code !== "EPERM") {
+ er.code = "ELIFECYCLE"
+ }
+ er.pkgid = pkg._id
+ er.script = env.command
+ er.pkgname = pkg.name
+ return cb(er)
+ } else if (er) {
+ log.error(pkg._id+"."+command, er)
+ log.error(pkg._id+"."+command, "continuing anyway")
+ return cb()
+ }
+ cb(er)
+ })
+}
+
+function makeEnv (data, prefix, env) {
+ prefix = prefix || "npm_package_"
+ if (!env) {
+ env = {}
+ for (var i in process.env) if (!i.match(/^npm_/)) {
+ env[i] = process.env[i]
+ }
+
+ // npat asks for tap output
+ if (npm.config.get("npat")) env.TAP = 1
+
+ // express and others respect the NODE_ENV value.
+ if (npm.config.get("production")) env.NODE_ENV = "production"
+
+ } else if (!data.hasOwnProperty("_lifecycleEnv")) {
+ Object.defineProperty(data, "_lifecycleEnv",
+ { value : env
+ , enumerable : false
+ })
+ }
+
+ for (var i in data) if (i.charAt(0) !== "_") {
+ var envKey = (prefix+i).replace(/[^a-zA-Z0-9_]/g, '_')
+ if (i === "readme") {
+ continue
+ }
+ if (data[i] && typeof(data[i]) === "object") {
+ try {
+ // quick and dirty detection for cyclical structures
+ JSON.stringify(data[i])
+ makeEnv(data[i], envKey+"_", env)
+ } catch (ex) {
+ // usually these are package objects.
+ // just get the path and basic details.
+ var d = data[i]
+ makeEnv( { name: d.name, version: d.version, path:d.path }
+ , envKey+"_", env)
+ }
+ } else {
+ env[envKey] = String(data[i])
+ env[envKey] = -1 !== env[envKey].indexOf("\n")
+ ? JSON.stringify(env[envKey])
+ : env[envKey]
+ }
+
+ }
+
+ if (prefix !== "npm_package_") return env
+
+ prefix = "npm_config_"
+ var pkgConfig = {}
+ , keys = npm.config.keys
+ , pkgVerConfig = {}
+ , namePref = data.name + ":"
+ , verPref = data.name + "@" + data.version + ":"
+
+ keys.forEach(function (i) {
+ if (i.charAt(0) === "_" && i.indexOf("_"+namePref) !== 0) {
+ return
+ }
+ var value = npm.config.get(i)
+ if (value instanceof Stream || Array.isArray(value)) return
+ if (!value) value = ""
+ else if (typeof value !== "string") value = JSON.stringify(value)
+
+ value = -1 !== value.indexOf("\n")
+ ? JSON.stringify(value)
+ : value
+ i = i.replace(/^_+/, "")
+ if (i.indexOf(namePref) === 0) {
+ var k = i.substr(namePref.length).replace(/[^a-zA-Z0-9_]/g, "_")
+ pkgConfig[ k ] = value
+ } else if (i.indexOf(verPref) === 0) {
+ var k = i.substr(verPref.length).replace(/[^a-zA-Z0-9_]/g, "_")
+ pkgVerConfig[ k ] = value
+ }
+ var envKey = (prefix+i).replace(/[^a-zA-Z0-9_]/g, "_")
+ env[envKey] = value
+ })
+
+ prefix = "npm_package_config_"
+ ;[pkgConfig, pkgVerConfig].forEach(function (conf) {
+ for (var i in conf) {
+ var envKey = (prefix+i)
+ env[envKey] = conf[i]
+ }
+ })
+
+ return env
+}
+
+function runHookLifecycle (stage, pkg, env, wd, cb) {
+ // check for a hook script, run if present.
+ var hook = path.join(npm.dir, ".hooks", stage)
+ , cmd = hook
+
+ fs.stat(hook, function (er) {
+ if (er) return cb()
+
+ var conf = { cwd: wd, env: env, customFds: [ 0, 1, 2] }
+ var proc = spawn("sh", ["-c", cmd], conf)
+ proc.on("close", function (er) {
+ if (er) {
+ er.message += "\nFailed to exec "+stage+" hook script"
+ log.info(pkg._id, er)
+ }
+ if (npm.ROLLBACK) return cb()
+ cb(er)
+ })
+ })
+}
Something went wrong with that request. Please try again.