Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Passing flags and arguments into run-script #3494

Closed
ELLIOTTCABLE opened this Issue · 80 comments

37 participants

ELLIOTTCABLE Martin Cooper Domenic Denicola Luke Arduini Evan Tahler Hisanaga MAKINO Pēteris Ņikiforovs Ian Langworth ☠ Jono Spiro Mariusz Nowak Tyler Akins Iļja Ketris Dane Springmeyer Tom Vincent Stephen Sugden TruongSinh Tran-Nguyen Morgan Herlocker Florian Bezagu Nik Mike Smullin Alex Kocharin Travis Webb Will Hoover Malte Legenhausen and others
ELLIOTTCABLE

At the moment, there's really no way to pass arguments to scripts specified in the "scripts" field with npm run-script.

I have three options, currently:

  1. Hard-code the flags into the "scripts" field. This obviously doesn't cover all situations; there's plenty of situations where I want to vary options I pass to some of the scripts, or make the scripts I write more widely applicable / configurable.

  2. Pack the flags into an environment variable. This isn't very composable, it's quite un-UNIX-y, and it requires me to define all of the flags in one place. (Not to mention that WITH="--debug --optimize" npm run-scripts compile just ain't very pretty. ;)

  3. Wrap the script in question (if it's not my own), and query the user interactively. Again, not remotely composable or UNIX-y; and to boot, means I have to interact with the script every single time I run it.


Here's my suggestion:

  • -fLGs and --flags that come after the second extant argument to run-script are appended to the string in the "scripts" field before it's run
  • anything after -- is appended to the string in the "scripts" field, before it's run

Examples:

  • npm run-scripts test -- something/something_else.js
  • npm run-scripts a_module compile --debug --optimize
  • npm run-scripts clean -- -dXf

This is completely backwards-compatible with what exists now; it does mean that we'd have to explicitly name the current module if we didn't use --, but that's minor, when it guarantees backwards compatibility.

Martin Cooper
Collaborator

There's a good deal of flexibility already. Take a look at:

  • The config section in npm help json
  • The ENVIRONMENT section in npm help scripts
  • The Per-Package Config Settings section in npm help config
Domenic Denicola
Collaborator

I agree this would be nice; it's a pain to switch from npm test to npm install mocha -g; mocha --grep "stuff" or node node_modules/.bin/mocha --grep "stuff" just to use a feature of my test framework. I'd love @isaacs to chime in on this idea.

ELLIOTTCABLE

@mfncooper I'm using all of those, yes; but the current design is extremely un-UNIXy. Neither do any of those help with the particular use-case @domenic and I are talking about. (=

Luke Arduini

-1

ELLIOTTCABLE

@st-luke reasoning? Commentary?

Luke Arduini
ELLIOTTCABLE

@st-luke … and given the use-case put forward by myself, and echoed by @domenic, could you please explain how it's still “totally unnecessary?”

Luke Arduini

@elliottcable if it's special npm functionality/complexity to facilitate using the feature a module that is meant to be installed globally, then I think it seems incredibly silly to add this in. We should wait for the final word from @isaacs, but I truly believe there are other and better ways to do what you are trying to do than to shove this kind of code into the package manager.

Domenic Denicola
Collaborator

to facilitate using the feature a module that is meant to be installed globally

That's the thing though: installing testing runners globally is a nightmare. Much better to use the version bundled with the package you're developing/testing, which is precisely what npm test does.

Luke Arduini

That's the thing though: installing testing runners globally is a nightmare.

Agreed, there are multiple good reasons to not do this.

Much better to use the version bundled with the package you're developing/testing, which is precisely what npm test does.

Agreed, why not require('mocha') in your test script and use the features you need from that?

Maybe I'm missing something, but what's the advantage of adding this feature other than saving you a small amount of code in your test files or scripts themselves?

Domenic Denicola
Collaborator

@st-luke Here's the scenario. I have npm test set up to run mocha, which because of "scripts" works great. It runs the local copy in node_modules/.bin/mocha. This is great!

Now I am happily developing, and I run my tests with npm test. But one day I want to narrow it down to only the tests I am working on, e.g. the tests with "foo" in their name. If I were using the global mocha command, I would do

$ mocha --grep "foo"

But I am being a good boy and am using the locally installed mocha, via npm test. So I try

$ npm test --grep "foo"

but as discussed in this bug, this doesn't work! So what are my workarounds? Either (a) I have to either revert to using the global mocha install, with all its perils, or (b) I have to look at the contents of my "scripts" hash, find mocha there, then mentally translate that to node node_modules/.bin/mocha, then finally type

$ node node_modules/.bin/mocha --grep "foo"

which is pretty horrid and abstraction-boundary-breaking. In both cases I lose the benefits of the "scripts" hash.

The proposal in the OP is to just allow

$ npm test -- --grep "foo"

which seems reasonable to me.

Luke Arduini

but as discussed in this bug

(this isn't a bug)

So this is mainly for mocha? How many other situations is this applicable to?

Because these examples from the original post:

npm run-scripts a_module compile --debug --optimize
npm run-scripts clean -- -dXf

are inappropriate use of npm's scripts.

ELLIOTTCABLE

@st-luke that's my bad, then, for not taking long enough to come up with better examples for the original report I posted. The first example, I can see how you'd consider it inappropriate. I was only using them as examples of my proposed syntax; I made the mistake of assuming that nobody sane would dispute the usefulness of the concept itself.

As for examples of the use, then, let's focus on @domenic's example. It's quintessential and applicable.

Domenic Denicola
Collaborator

So this is mainly for mocha? How many other situations is this applicable to?

It's applicable to pretty much every test runner I've used.

More generally, given the niceness of npm's scripts as a cross-platform alternative to make etc., it can serve well for many situations. Two that I've run into personally:

  • node node_modules/.bin/grunt build:production vs. the proposed npm run build -- --production
  • node node_modules/.bin/jshint --verbose vs. the proposed npm run lint -- --verbose
ELLIOTTCABLE

Basically, it boils down to this: there's currently no UNIX-y way to argument scripts in npm. You have to modify the script (whether modifying the invocation in package.json, or duplicating the script itself so you've got two versions that operate slightly differently) to modify the behavior slightly. There's more than one way to npm test your codebase (with coverage instrumentation, or not? compile the client-side tests and open them in the system's browser, too? start a file-watcher so the tests re-run each time you change one of the source-files in realtime? generate pipe-able output, or user-friendly coloured output?), so there should be more than one way to invoke npm test.

The only alternatives are either multiple "scripts" fields for the same actual purpose (and you can't really be suggesting that one have npm test-with-coverage, npm test-and-compile-client-tests, npm test-with-coverage-and-compile-client-tests, npm test-with-coverage-with-colour, npm test-with-coverage-and-colour-and-watch …), or aping flags via environment variables or temporarily changing configuration variables, both of which are absurdly un-UNIX-y / messy.

ELLIOTTCABLE

Not that it's especially relevant in a pure, abstract way (as npm has a different purpose than any of these) … but since it does demonstrate that something like this is desirable, all of the following have a method for this.

Make

target:
        $(MAKE) --something=$(FOO) --something-else=$(BAR)

$ make target FOO=123 BAR=456

Rake

task :target, :foo, :bar do |args|
   do_something args.foo + args.bar
end

$ rake target[123,456]

Cake

option '-f', '--foo [VAL]'
option '-b', '--bar [VAL]'

task 'target', 'Blah blah!', (options) ->
   console.log options.foo + options.bar

$ cake target --foo 123 --bar 456

Grunt

grunt.registerTask('target', "Blah blah blah.", function(foo, bar) {
  console.log(foo + bar);
});

$ grunt target:123:456

Apache Ant

<target name="target">
   <java executable="something.exe">
      <arg value="${foo}"/>
      <arg value="${bar}"/>
   </java>
</target>

$ ant -Dfoo=123 -Dbar=456 target
Luke Arduini

Basically, it boils down to this: there's currently no UNIX-y way to argument scripts in npm. You have to modify the script (whether modifying the invocation in package.json, or duplicating the script itself so you've got two versions that operate slightly differently) to modify the behavior slightly. There's more than one way to npm test your codebase (with coverage instrumentation, or not? compile the client-side tests and open them in the system's browser, too? start a file-watcher so the tests re-run each time you change one of the source-files in realtime? generate pipe-able output, or user-friendly coloured output?), so there should be more than one way to invoke npm test.

It sounds like you just want a makefile.

ELLIOTTCABLE

@st-luke that doesn't solve the problem that the go-to invocation for all module users, is going to be npm test or similar; in fact, the npm documentation says that this should be the case. (Not to mention that I don't want to make autotools/make a development dependency of my JavaScript project.)

npm's approach is great: just make shell scripts, and use the package.json as a central place to list/name them, allowing npm as a central target for invoking them. Unfortunately, in the particular respect documented by this bug-report, this is inflexible / untenable.

Evan Tahler

I would offer an alternative reason to want this feature:

Say you have a framework like Sails or ActionHero, and you want provide tools for folks to generate files and run a server. If folks install the package globally (npm install -g), the contents of the project's ./bin become available in the $PATH and are generally available. However, I'm under the impression that installing packages globally should be avoided, especially when different projects might require different versions.

I know I can run "local" binaries with ./node_modules/.bin/{myBinary}, but that is a rather tedious collection of commands for newcomers.

Global

  • npm install -g myPackage
  • myPackage generate --name=thing
  • myPackage start server

Local

  • npm install myPackage
  • ./node_modules/.bin/myPackage generate --name=thing
  • ./node_modules/.bin/myPackage start server

What is being discussed here would be:

  • npm install myPackage
  • npm run-script myPackage generate --name=thing
  • npm run-script myPackage start server

The run-script (or run) option is a nice solution which
1) doesn't require a global install which is likely to confuse folks down the line when they need to change versions
2) doesn't require the creation of a "binary"
3) doesn't require knowledge of "hidden" NPM directories


Ideas copied in from the discussion here https://groups.google.com/forum/?fromgroups#!msg/npm-/XR5sRLt_9gk/hulhvU44SKQJ

ELLIOTTCABLE

@evantahler not sure I'm understanding how what you're talking about, is an alternative to what I'm talking about. Are you proposing adopting what I'm suggesting (from your --name=thing example, I feel that this is so) and further suggesting that we dispose of the "scripts" field in favour of ./bin files? Or what?

Evan Tahler

My example was probably more convoluted than I intended, sorry. I'm agreeing with @ELLIOTTCABLE in asking for the ability to pass argument to run-scrpt, and demonstrating an alternative use case to @domenic, one that isn't about testing (which might be simper to discus as there are less dependancies).

Basically, I would love to be able to pass arguments into run-script to use generators. As a module developer, I don't really have a preference whether or not I create "scripts" or "binaries" (as they are just node *.js files in my case).

As a user of the module, it seems that there are only 2 ways to execute code from a module which can take arguments:
1) global install with globally available binaries (myPackage generate --name=thing)
2) local install and access at ./node_modules/.bin/* (./node_modules/.bin/myPackage generate --name=thing)

I think both of these options would be less user-friendly being able to pass arguments to run-script for the reasons above. I posit that npm run-script myPackage generate --name=thing would be a great alternative to # 1 and # 2

Domenic Denicola
Collaborator

FYI @isaacs says this should probably be implemented after #3313 since that framework will definitely be used.

ELLIOTTCABLE

@domenic @isaacs I ran into exactly #3313 when trying to implement a proof-of-concept of this, ripping out the relevant bits of bin/npm, run-script.js, and lifecycle.js. Can't wait for that to be completed.

I'm all for waiting on #3313, then this issue's suggested changes would be an absolute breeze. (=

Hisanaga MAKINO

+1

Austin Birch austinbirch referenced this issue in Akkroo/JSMongoQuery
Merged

Debug tests #6

Pēteris Ņikiforovs

+1

I'm also using mocha and would like to pass --grep test. I'm using npm test instead of mocha because with coffee script arguments the mocha command becomes very long. When I'm on Windows I don't know how to use aliases in cmd.exe so now I manually edit package.json every time I need to add --grep.

Ian Langworth ☠

+1

Supporting this would make our, and presumably a lot of other peoples', lives easier. Arguments against this seem to be unnecessary mandates.

Jono Spiro

+1

Mariusz Nowak

Definitely +1.

I don't use Makefile as I want to use something that's environment agnostic (not just for *nix systems). I don't use grunt as I already use npm which provides very close functionality with scripts. The only thing I miss from npm scripts is ability to pass the optional arguments to underlying scripts.

The best way to achieve it, would probably be to append anything after -- to the command mentioned in scripts field (assuring it will also work on Windows).

Tyler Akins

+1

Currently I use npm test which runs a shell script that will execute each part of the testing suite we have. To run them separately, there's scripts: npm run-script vows, npm run-script karma, npm run-script e2e, etc. Yes, we have multiple testing frameworks and so invoking each one with the right flags gets tedious. I'd love to be able to combine the tests into a single shell script so I can use npm test or npm test --vows or npm test --karma, and so forth. Adding this, even if this requires special syntax, would be tremendously helpful. For instance, you might want to say "arguments go here" in the package.json file like this, borrowing a bit from bash: scripts: { "test": "util/bin/run_tests $*" }

Iļja Ketris

+1

Domenic Denicola
Collaborator

Further +1s on this bug are not helpful and just serve to spam the maintainers. To show your support for this, please submit a pull request fixing #3313.

Jono Spiro
Domenic Denicola
Collaborator

Click the "Watch thread" button literally pixels below the "comment" button instead.

Jono Spiro
Luke Arduini
Stephen Sugden

For the people who are interested in this issue, please give npm-exec a try. I'm aiming to solve #3313 with it eventually, but more testers rooting out the bugs now would be awesome :+1:

TruongSinh Tran-Nguyen

+1, i would like to run it with grunt, e.g. npm test unit:alpha

Morgan Herlocker

Just wanted to throw in one more use case around mocha. It would be really nice to specify reporters, which everyone has a different preference for. Personally, I switch often between a couple reporters depending on what I want to visualize:

npm test -R spec
npm test -R min
Florian Bezagu

@morganherlocker +1 I'd like to do this too !

Nik

+1

Mike Smullin

unsubscribing from this thread because they're obviously not going to do anything about it. i guess npm is mostly a package manager and not a build tool. since i originally posted, i've noticed most other people using either install.sh bash scripts or a Makefile and the popular unix make tool, as an alternative.

i've embraced this solution because i've also found other problems when running things via npm; it wraps the process but it doesn't forward the os kill SIGNALs to the child. so a ctrl+c could kill npm but not your node process. and sending other signals like SIGUSR2 for debugging is also a challenge. so i've just begun avoiding npm for running scripts altogether.

Evan Tahler

@sbimikesmullin here's a little test case I have about the signals issue you mention: #4603

Alex Kocharin

For instance, you might want to say "arguments go here" in the package.json file like this, borrowing a bit from bash: scripts: { "test": "util/bin/run_tests $*" }

Since bash seem to support that syntax inline:

$ bash -c "echo \$*" q w e r t y 
w e r t y

It might be done relatively easily, if you don't mind annoying people who is still using windows. :)

Alex Kocharin

@ELLIOTTCABLE ,

anything after -- is appended to the string in the "scripts" field, before it's run

-- is already used by nopt parser, it means "stop expanding attributes".

So you can define:

scripts: {
  '-g': 'echo hello world'
}

And run it with npm run -- -g, it'll work.

So I'm afraid your suggestion is not backward compatible, and might be confusing.

ELLIOTTCABLE

@rlidwka not sure I see how that's incompatible. That's exactly how I suggest it work, here. (That is, they are completely un-parsed after --, just as nopt already parses; and that un-parsed opaque block of text is appended as-is.)

Alex Kocharin

@ELLIOTTCABLE , npm run takes two arguments, script name (required) and folder name (optional).

So npm run-scripts clean -- -dXf is currently a valid syntax. It means "run script named clean in the folder named -dXf.

ELLIOTTCABLE

@rlidwka that's fine; that doesn't seem to present much of a problem, as the -dXf is still “the second argument.” It seems clear to me that we could dictate that the second argument is only optional if there are to be no further arguments … if there are, then the second argument must be provided before them.

(Meaning, npm run-scripts clean -- -dXf -f still operates as you described, as -dXf is still the second argument, and -f is the first pass-through argument … and to treat -dXf as additional flags, you'd have to do npm run-scripts clean . -- -dXf. Intuitive enough, and easy to document, yes?)

Edit: I should also point out, although it's probably obvious, that in my suggestions and commentary, I consider intuitiveness and coherence with other usual UNIX / command-line interface styles, to be far more important than backwards-compatibility. Especially in such an edge-case. The rest of the community, or the maintainers, might have different priorities, of course, and that's totally fine; so take my suggestions with a grain of salt.

If we can get both, that's best of all. I'm just very against abandoning the usual and widely-known semantics of --, for the sake of backwards compatibility in an edge case.

Travis Webb

+1

Being able to pass arguments into npm scripts would not just align behavior to that of all other Linux command-line and build tools I use every day. It would also be very useful and make a lot of routine tasks more convenient, regardless of which flavor of CLI religion you subscribe to.

Travis Webb

To strengthen the argument made by @ELLIOTTCABLE, -- semantics aren't merely usual and widely-known, they are bona fide POSIX standard. See man -s 3 getopt

Strider

@ELLIOTTCABLE, I think, this is great idea. If do so, I can use less make and more npm run-script. The positive side is run-script (and derivative commands) appends node_modules/.bin to $PATH which means more clear scripting.

On the negative side: npm run-script has two positional parameters, script and package, where second one is optional. Passing additional arguments will lead to ambiguity (is 2nd parameter a package name or positional argument?).

Anyway, I would like to see this feature.

HE Shi-Jun
hax commented

+1

ELLIOTTCABLE

Just a thought: I can imagine that the maintainers are aware that this is a desired feature at this point (28 participants! I'm so happy!). Maybe let's stop triggering an e-mail to them with every :+1:, and either pull-request, or wait for someone else to? (=

(A bit meta: Sorry for the extra e-mail, to say this.)

Dustin Farris dustinfarris referenced this issue in ember-cli/ember-cli
Closed

`npm build` doesn't work #748

Jeff Morrison

I just wanted to write a quick follow-up on how we're hacking around a specific manifestation of this problem for Jest (in case anyone else wants to use the same approach):

To restate the specific issue: The Jest CLI test runner accepts arguments that indicate which tests you'd like to run from the command line. Ideally one would just set {"scripts": {"test": "jest"}} and then used 'npm test MyTestFile'.

An alternative to this would be to 'npm install -g jest-cli', and then just use that global script to run tests for various repos and libraries. The problem here is that Jest is versioned -- and it's possible that the version you installed globally isn't compatible with the tests or test configuration the particular project you're running against was built for.

To hack around this, we would up making our 'jest' bin aware of the CWD in order to:

  • Determine if the CWD sits within some npm project repo (recursively walks the dir ancestry until it finds a package.json, or hits root)
  • If it finds a package.json, look in it for dependencies on a specific version of Jest (and defer off to the project-local version of the jest if a dependency is found)
  • If we don't find a package.json while walking the dir ancestry or the if it just doesn't indicate any dependencies on Jest, go ahead and run the globally-installed CLI

https://github.com/facebook/jest/blob/master/bin/jest.js

Anyway, I'm sure we're not the first ones to take this approach but I figured I'd put it here in case it solves this particular scenario for anyone else who came across this issue for the same reason.

All that said, it would make me really happy to be able to delete all that code in favor of just recommending people set up 'npm test' and them just being able to pass args through.

ELLIOTTCABLE

@jeffmo quick Q (and to anybody else, getting e-mail responses for this repository): Would a secondary module (basically, a command-line shim) that temporarily provides this functionality be of use to your case? I don't know how likely people are to suggest their users/consumers npm install npm-run before npm-run test blahblahblah. It'd be fairly difficult to fully simulate npm's environment, and it seems like such a small set of use-cases would benefit from such a third-party solution … but it's taking so long to see a solution to such an important issue, that I've been considering it.

It'd help my decision a lot to know that I'd be helping others out, as well, so …

Jesús Leganés Combarro
Mariusz Nowak

@ELLIOTTCABLE it should be part of npm, really. One of the most missed features

ELLIOTTCABLE ELLIOTTCABLE referenced this issue in gulpjs/gulp
Closed

Allow .coffee by default #123

Cătălin Mariș alrra referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Martin Jagusch

+1

isaacs
Owner

Literally everyone in the entire universe has +1'ed this feature forever. @bcoe has talked about doing it recently, the better to make ndm a winful thing.

Patch/test welcome.

isaacs
Owner

See also: #1543

Jesús Leganés Combarro
Albert Engelbrecht

Awesome, glad to see a patch would be welcomed!

@piranna See this comment.

Jesús Leganés Combarro
Albert Engelbrecht

@piranna If I understand the issue correctly, npm exec <blah> would require the ability to do npm run-script <script> <args>

And I'm with you as far as testing. I recently hacked together a Makefile for this exact reason.

Jesús Leganés Combarro
Benjamin E. Coe
Owner

Just some context, I'd love this feature because it makes it easier to run npm packages as services. Currently when someone publishes a package, like statsd, which is meant to be run as a daemon, a consumer's workflow looks something like this:

  • install the statsd npm package.
  • create a wrapper script that imports the package, and sets up the appropriate flags.
  • deploy the service to a server, perhaps wrapping it in upstart, or whatever.

If there was a better convention for simply passing the appropriate execution arguments into a run-script this workflow would be simplified.

Travis Webb

As a command-line tool itself which provides the ability to invoke other arbitrary commands, npm has naturally already become a speed-dial to other CLI tools that are relevant to managing, building, and deploying packages. I can see some warranted apprehension if there is fear of npm becoming an all-purpose build tool, but @isaacs doesn't seem too shy about declaring clearly in TFM which practices may lead to future harm, and I think the absence of this feature will either cause undesirable hackery, or de-facto global dependency requirements for other tools, both of which practices I think are more evil than this feature.

Domenic Denicola
Collaborator

That "lock issue" button is starting to look really tempting, to stop the flood of silly +1s...

(Remember, you can subscribe to notifications by clicking "Subscribe," without sending an email to everyone subscribed as a +1 comment does.)

Alex Kocharin

Github notifications aren't great (there's no "mark unread" button for example), and it's easy to lose a notification there. +1s can ensure that a feature that's wanted by people won't get lost.

(And you also can unsubscribe from notifications by clicking "Unsubscribe", if such comments are too annoying).

ELLIOTTCABLE

@rlidwka relevant commentary about the issue is something a reader may (we are.) still be interested in; and having to unsubscribe because of all these people not paying attention to the existence of a "Subscribe" button is less-than-ideal.

tl;dr: don't put this on us, for needing or wanting to continue to consume the thread. This is on the people paying no attention to the thread and throwing in wanton +1's.


Commented on the new pull-request, and thought it was relevant here:

Just want to throw in my -1 against forcing --.

I definitely understand the restrictions and reasoning; but we're already going to have to flag this as a breaking change. My two-cents: if we're going to break backwards with some of the flag-parsing, let's Do It Right and break backwards entirely, in such a way that the (really common.) npm test and npm run clean type stuff works as expected by a newcomer, instead of having to teach everybody for the rest of eternity that npm specifically requires some strange -- stuff.

... continued: #5518 (comment)

Samuel Giles samgiles referenced this issue in Financial-Times/polyfill-service
Merged

Add forever to package.json #26

Forrest L Norvell othiym23 added npm-exec and removed nice to have labels
Forrest L Norvell othiym23 added this to the 2.0.0 milestone
Forrest L Norvell
Owner

So we do have a means for passing parameters into run scripts now, and some people aren't happy with the way it works right now. I'm going to mark this one as complete, as we have the functionality we want for npm 2 – if you want something that doesn't require --, chime in on #6053 with your requirements, if they differ from what we have there.

Forrest L Norvell othiym23 closed this
Mariusz Nowak

@othiym23 when will npm 2 be production ready? When we can expect it to land with Node?

Forrest L Norvell
Owner

when will npm 2 be production ready?

Release candidate in a week, promoted to latest the week after that.

When we can expect it to land with Node

That's not under our control, but I would very much like 2.0.0 to ship with Node 0.12.

Markus Dolic yelworc referenced this issue from a commit in ElevenGiants/eleven-server
Markus Dolic yelworc add possibility to supply arguments to "npm run" calls
Since npm run-script currently does not have a way to pass additional
arguments to scripts (see <npm/npm#3494),
add a "wildcard" variable that can be used as a workaround, e.g. to
run specific tests or benchmarks:

ARG="--grep objrefProxy" npm -s run test
ARG=utils.js npm -s run bench
e3cce46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.