Skip to content
This repository

child_process.spawn ignores PATHEXT on Windows #2318

Open
OrangeDog opened this Issue December 12, 2011 · 29 comments
James Howe

For example require('child.process').spawn('mycmd') won't find C:\util\mycmd.bat when PATH contains C:\util and PATHEXT contains .BAT.

Ye olde code (https://github.com/joyent/node/blob/v0.4/src/node_child_process_win32.cc) looks like it would have worked, but I have no idea where the v0.6 equivalent is.

Igor Zinkovsky
Collaborator
Bert Belder
Collaborator

At the moment child_process.spawn() can only run exe files. This is a limitation of the CreateProcess API. child_process.exec() can run batch files though. We could make libuv prefix cmd /c if the file is not an exe - but iirc Peter was strongly against that.

@drpizza @igorzi thoughts?

Bert Belder
Collaborator

@OrangeDog as a workaround, you could use

require('child_process').spawn('cmd', ['/s', '/c', '"C:\\util\\mycmd.bat"'], { 
  windowsVerbatimArguments: true
});

(this option is internal and not guarateed to stick btw)

James Howe

Not very helpful for writing portable code though.

Bert Belder
Collaborator

@OrangeDog Well you can't really write a portable batch file anyway.

@DrPizza suggested today that we could add a { shell: true } option to spawn. I kind of like the idea. It allows using spawn for the same purpose as exec without buffering all the output. We also currently have the weird distinction between exec and execFile; we could just make those the same function but with a different default for the shell option. @ry, @bnoordhuis, what do you guys think?

Ben Noordhuis

It'd be superfluous on Unices - the two things a shell script needs is a valid shebang and the executable bit set.

Bert Belder
Collaborator

@bnoordhuis not completely because people may want to do ls -r | grep bla

David Ellis

If I can interject into this conversation; the shebang indicates which executable will actually run the file and the executable bit flags that the user (or a user) has granted permission to run the file (probably why auto-executing of shell scripts in the Windows %PATH% was objected to before, since there's no executable bit there).

Perhaps a { shell: "shellname" } option would be better? This would indicate which program you want to pass the file to for execution (being explicit that you do want to execute a script rather than an application), and it could still be useful for Unices.

Basically, the shebang line is invalid syntax in Javascript, but Node.js tolerates it because Unix shells will automatically interpret that line and pass the file to the specified program -- but then that means you can't use that JS file in a browser without modification. A developer using browser-require might instead prefer to spawn the script with { shell: "node" } instead and not include the shebang at all.

EDIT: Of course, in Unix, a shebanged shell script could still be run the normal way.

James Howe

@piscisaureus but you can write batch/perl/etc. scripts to allow spawn('scp') or spawn('readlink') to behave in the same way on Windows as on *nix.

I thought the whole point of adding libuv was to give portable cross-platform support.

Phi Tien

@piscisaureus: hi, I am just new in node js, and I get the same error when try to resize image using gm, this seems to resolve my problem, but I dont know how to fix this, could you give me some sample code?

Thank a lot.

Ben Noordhuis

@piscisaureus Is this still an issue?

Bert Belder
Collaborator

@bnoordhuis It is.

Isaac Z. Schlueter
Collaborator

I think @piscisaureus's suggestion is on to something.

It's kind of annoying right now that there's a "run in a shell" function that buffers the output (exec), and a "run as-is" option that doesn't (spawn), and a "run as-is" that buffers the output (execFile), but no "run in a shell" that doesn't buffer the output.

However, it can't be spawn(cmd, args, {shell: true}), because that doesn't really work for the ls -laF | grep foo case, right? The cmd is either "sh" or "cmd" depending on platform, and the arg is always either /c $cmd or -c $cmd.

What about this?

child = child_process.spawnShell('util.bat glerp gorp', {options...})

Which would be sugar for:

child = child_process.spawn(isWin ? 'cmd' : 'sh', [isWin?'/c':'-c', arg], options)
Domenic Denicola

So, I don't understand anything about shells on Unix, so take this with a grain of salt. In particular I don't understand how sh -c would affect things. I guess maybe that would let it run chmodded shebanged files?

But for me the problem is that there is lots of published code out there that is not cross-platform because of this. (npm with git and npm-www with couchdb recently, but much more I've seen) A fix that retroactively makes all that code work would be ideal, instead of throwing a new method in the already-confusing medley of child_process and having to evangelize "use this if you want your code to work on Windows."

So I'd say

We could make libuv prefix cmd /c if the file is not an exe - but iirc Peter was strongly against that.

is the most useful thing I've seen so far. Alternately, if there's an alternative to the CreateProcess API that doesn't have this suckiness, that would be nice.

Isaac Z. Schlueter
Collaborator

@domenic sh -c would allow you to do shell syntax stuff, pipes and whatnot. sh -c "ls -laF | grep foo > output.log" would list all the files, search the output lines for "foo" and then write the results to output.log. "ls -laF | grep foo > output.log" isn't an executable name, it's a shell program. (An executable name is also a valid shell program, of course.)

If we start wrapping every command in a shell, then that's not so great.

Domenic Denicola

@isaacs I see. Then that seems like a fairly orthogonal concern to the fact that Windows executables come in many flavors (exe, bat, cmd, etc.). People using spawn will not expect shell syntax, but they will expect that spawn("couchdb") works cross-platform without any extra { shell: true } or spawnShell or the like.

It sounds like spawnShell would be independently useful for the

"run in a shell" that doesn't buffer the output

case, but regardless I think spawn should work with .bats, .cmds, etc.

Bert Belder
Collaborator

libuv used to use PATHEXT, but this was reverted in joyent/libuv@8ed2ffb because it didn't work for non-exe files. I would be ok with putting this back in and running all non-exe files with "cmd /c". I take patches. (Note that the escaping rules are quite complicated when running stuff with cmd /c)

Domenic Denicola

@piscisaureus Would using ShellExecuteEx instead of CreateProcess be a good route toward this?

Otherwise, if I were to work on a patch for this, would you suggest it as a libuv patch or Node patch? Probably libuv, right?

Bert Belder
Collaborator

@domenic Afaik ShellExecuteEx doesn't allow redirection of stdio handles.

Chris Jaynes

The problem seems to be that child_process got a bit muddied up trying to handle some of these cases for shell commands, but we've reached a point where it's getting uncomfortable to continue to add shims in there to support more shell-friendly options. I don't think this is a cross-platform problem. It's a problem of executables vs. shell scripts. This isn't a problem for most Unix devs, because they already know the difference between the two.

I'm primarily a Windows guy myself, but it doesn't seem fair that Windows should get special treatment at the child_process level just because cmd and bat files are treated as shell commands by the OS. Windows devs will just need to learn the difference between an executable and a shell command for their platform. (This isn't just a problem with Node, BTW. I've seen this play out in other platforms, too.) We need to respect Windows's idea of what is executable, and what is not, and be careful not to break that.

I initially liked the idea of adding a {shell:true} sort of option to spawn(), but after looking at the code, I'd expect it to be supported in exec() and execFile(), too, and that would break things. (We could have different defaults for spawn() and exec(), but that's still pretty confusing.)

Rather than try to make the mess of spawn(), exec(), and execFile() do even more black magic, and risk making things even more unclear, I'd prefer that we keep things honest and add @isaacs's suggestion of adding a spawnShell() method. If we clean up the docs a bit, and make it more clear what each of these methods does, we've got a shot at having a fairly complete set of use cases.

In the long term, it might be worth considering the deprecation of exec() and execFile(), in favor of a set of functions with a more deliberate separation between executables and shell commands. The code would get a lot cleaner, and with just slightly better documentation and error messages, it would also be an easier API for new developers to understand.

Jan Kolkmeier jouz referenced this issue in balupton/bal-util October 05, 2012
Closed

Failure to spawn certain processes on Windows #2

Nathan Rajlich TooTallNate referenced this issue in component/component November 06, 2012
Closed

Issue with windows #138

Nathan Rajlich
Collaborator

bump

Peter Halliday

I'd like to add my 2 cents with regard to using

require('child_process').spawn('cmd', ['/s', '/c', '"C:\\util\\mycmd.bat"']);

on windows. This is giving me a real headache as doing this with long running processes results in those processes being orphaned when I call child.kill() ie. the cmd process is killed but the process it spawns is not.

See this gist for an example:

https://gist.github.com/4117163

I'm still trying to figure out if this is a bug that should be raised separately (child.kill seems to deliberately not kill grandchildren (which is annoying also) and this might be considered a grandchild)

Dennis Kehrig DennisKehrig referenced this issue in jdiehl/brackets-extension-manager December 03, 2012
Closed

Extension installs not working - Windows 7 x64 #3

Kevin Mårtensson kevva referenced this issue in yeoman/yeoman April 09, 2013
Closed

Windows Support #216

Josh Lubawy jlubawy referenced this issue in rendrjs/rendr May 15, 2013
Closed

rendr does not install on Windows #34

mauricioaraldi mauricioaraldi referenced this issue in taboca/mural-web-fisl-14 July 03, 2013
Closed

Forever monitor in Windows #5

Ahmed Fasih fasiha referenced this issue in rvagg/node-pygmentize-bundled July 10, 2013
Closed

Example fails in Windows? #2

Alex T. shoomyth referenced this issue in bower/bower July 20, 2013
Closed

git.bat in PATH doesn't work #626

Thomas Parisot oncletom referenced this issue in oncletom/grunt-crx July 22, 2013
Closed

Fatal error: spawn ENOENT #24

Pierre Martin real34 referenced this issue from a commit July 24, 2013
Commit has since been removed from the repository and is no longer available.
Pierre Martin real34 referenced this issue in indieisaconcept/grunt-styleguide July 24, 2013
Merged

Modified spawn command to call node directly #18

Kevin Mårtensson kevva referenced this issue in yeoman/generator July 28, 2013
Closed

spawnCommand is missing a callback argument #238

Kevin Mårtensson kevva referenced this issue in yeoman/generator July 28, 2013
Closed

Improve conversion of args to string on win32 #245

Reza Akhavan jedireza referenced this issue in jedireza/drywall August 17, 2013
Closed

npm install on Windows fails #24

Daniel Temme dmt referenced this issue in linemanjs/lineman September 14, 2013
Closed

spec-ci not running on windows #55

Alan altano referenced this issue in jpillora/grunt-source September 15, 2013
Closed

No longer works on Windows #3

Simon Paitrault Freyskeyd referenced this issue in Freyskeyd/generator-laravel October 17, 2013
Open

Unhandled 'error' event #1

Belleve Invis be5invis referenced this issue in martine/ninja October 20, 2013
Open

Windows: support PATHEXT #675

martin-naumann martin-naumann referenced this issue in centralway/grunt-init-lungo-angular October 22, 2013
Open

grunt serve not working #7

Lon Ingram lawnsea referenced this issue in WaterfallEngineering/SpookyJS October 29, 2013
Open

Example hello.js can't run [Window] #71

Tammo van Lessen vanto referenced this issue in brunch/sass-brunch December 04, 2013
Merged

Fix for #33: Closed socket. #36

Tammo van Lessen vanto referenced this issue in gruntjs/grunt-contrib-sass December 04, 2013
Closed

Socket is closed error #33

Sindre Sorhus sindresorhus referenced this issue in gruntjs/grunt-contrib-sass December 13, 2013
Closed

Incorrect/faulty Windows' path handling. #35

ot-brett-bim ot-brett-bim referenced this issue in wycats/handlebars.js December 19, 2013
Closed

Can't build, no 'parser' task? #673

Egor Suvorov yeputons referenced this issue in oortcloud/meteorite January 07, 2014
Open

Windows support: first attempt #224

Egor Suvorov

@piscisaureus Do you have any news with this issue? I've noticed that you said 'Yes, I am actively working on that.', in this comment a year ago.

asfgit asfgit referenced this issue from a commit in apache/cordova-android January 16, 2014
agrieve CB-5801 Add spawn work-around on windows for it not being able to exe…
…cute .cmd files

More info: joyent/node#2318
22e4039
Egor Suvorov

I've digged into this issue a bit and here are all the information I gathered:

  1. require('child_process').exec does the following:

      if (process.platform === 'win32') {
        file = 'cmd.exe';
        args = ['/s', '/c', '"' + command + '"'];
        // Make a shallow copy before patching so we don't clobber the user's
        // options object.
        options = util._extend({}, options);
        options.windowsVerbatimArguments = true;
      } else {
        file = '/bin/sh';
        args = ['-c', command];
      }
    
  2. There is no argv on Windows - OS passes the full command line to application and it's up to it (or its runtime library) to process this string in any way it wants. That means, that different processes can have different escaping rules theoretically. But, there are formal rules from Microsoft that Visual C++ runtime uses. As almost everything is compiled using this runtime (MinGW GCC uses it too), I think we can assume that each executable follow these rules. That's how arguments of spawn are escaped by libuv, if windowsVerbatimArguments == false. Otherwise, it just joins all arguments with spaces.
  3. In cmd.exe /s /c the second argument (/c) is mandatory. It means 'execute the following command' and exit.
  4. /s is not mandatory, but it's required in a very specific case. Tl;dr: you have both c:\a.exe and c:\a b\c.exe, and if you run cmd /c "c:\a b\c.exe", it will not cut out quotes, but if use add /s it will run c:\a.exe with b:\c.exe as an argument.
  5. cmd.exe has its own rules for escaping - ^ is used as escape symbol. For example, if I run cmd.exe /c "echo 1^^2", it will print 1^2. &<>()@^| have special meaning too and should be escaped if we don't want them to be interpreted by cmd.exe.
  6. So, if you run executable using cmd.exe, you should escape your arguments twice, in order: first for your runtime library and then for cmd.exe. But, as the former escapes space characters, quotemarks, and backslashes; the former escapes another set of characters; we have the right to do escaping in reverse order (escape for cmd.exe first, and escape for VCRT then).
  7. spawn('a.bat') works perfectly - it spawns cmd.exe despite it is not explicitly specified it. It's behavior of CreateProcess - if you specify a.bat in both lpApplicationName and lpCommandLine, it will run command interpreter. Unfortunatelly, I haven't found a place where such behavior is documented. This doesn't work with another scripts, though - no .vbs, no .js. But still works with .cmd files.
  8. cmd.exe do something weird with ^ in batch files. It looks like parameters are unescaped several times:

    1. When you run cmd.exe /s /c
    2. When some command is run from inside the batch file

    As a consequence, if you have, say, two nested batches (x.bat calls y.bat, which calls z.exe), then you need to run cmd.exe /s /c "x.bat ^^^^^^^^" (yes, eight) to get z.exe ^ started:
    cmd.exe /s /c cuts half of them, then cuts another half while expanding arguments for call y.bat, and the new cmd.exe cuts another half. My gist.
    I was unable to pass odd number of ^ to z.exe through two batch.

  9. But cmd.exe behaves differently if ^ is inside quotation marks. I was unable to understand the logic behind this.
  10. Summarizing: if you have ^ in your arguments, you're gonna have a bad time, because exec always run cmd.exe and do not escape ^. But spawn, in contrast, may or may not
    use cmd.exe and thus, may or may not require escaping of ^. But if you don't use ^, you can always use one of two options;

    1. spawn('a.bat', [/*...*/]) and everything will be escaped properly to work as executable's parameter. You shouldn't use parameters inside batch itself (for example, try to calculate parameter's length using embedded cmd.exe commands, because it know nothing about backslashes). I don't know cases where you use parameters not to run external commands, because both type and echo are executables. Only some kind of for, may be. I think it's one of the best workarouns for now, despite I don't know why it works (I've tested on Windows XP, Windows Server 2003 and Windows 7)
    2. spawn('c.cmd', ['/s', '/c/', '"a.bat ' + args + '"'], {windowsVerbatimArguments: true}), if you carefully escape args for use with VCRT by yourself.
  11. For example, in bower/bower#626 spawn(which.sync(command). [/*...*/]) was considered as a workaround.

  12. Unfortunatelly, there is no universal escaping solution.
Egor Suvorov

I know two modules that partially solve this issue:

  1. child-proc. It uses windowsVerbatimArguments (which is undocumented) and do not perform escaping. This can
    lead to differences in behavior with child_process. Say, the following code:

    var spawn = require('child-proc').spawn;
    spawn('bats\\a.bat b\\b.bat').stdout.pipe(process.stdout);
    

    runs a.bat, but if you replace child-proc by child_process, b.bat will be started.

  2. spawn-cmd. It do not concatenate arguments by itself and do not use /S. However,
    this is not a problem, because cmd.exe do not remove quotes if the first character is not a quote, and file names rarely start with quote (UPD: except when they have spaces in path, in that case file name is quoted by libuv and we have a bad time).
    The only problem with it is that presence of process.env.compspec is used instead of os.platform() to detect Windows (UPD: fixed in 0.0.2)

Forbes Lindesay

win-spawn was my attempt at this, although reading your last post (would probably be worth making into a blog post somewhere) I realise there would be a lot more work to do, it was just a quick hack to solve the problem I was having at the time.

Patrick Lee

Thanks for this thread it was a real time saver. @ForbesLindesay win-spawn worked like a charm, thanks.

asfgit asfgit referenced this issue from a commit in apache/cordova-amazon-fireos January 16, 2014
agrieve CB-5801 Add spawn work-around on windows for it not being able to exe…
…cute .cmd files

More info: joyent/node#2318
5a0a8ab
asfgit asfgit referenced this issue from a commit in apache/cordova-amazon-fireos January 16, 2014
agrieve CB-5801 Add spawn work-around on windows for it not being able to exe…
…cute .cmd files

More info: joyent/node#2318
c118ada
Egor Suvorov

Wow, I've found a real-life case when /S and quoting the whole command may be considered interesting: cmd /C "C:\Program Files (x86)\Git\cmd\git.EXE" clone "hello world" - in this example, cmd cuts first and last quotes, leaving us with 'C:\Program' is not recognized as ...

I've described this in featurist/spawn-cmd#3

Estelle DeBlois brzpegasus referenced this issue in dbashford/mimosa-testem-require February 15, 2014
Closed

Testing not possible #8

Anachron

Still experiencing problems with this. What will we do about it?

Andrew Paprocki apaprocki referenced this issue in bloomberg/node-blpapi March 13, 2014
Open

Error: The specified module could not be found. #12

Flow11 Flow11 referenced this issue in diegonetto/generator-ionic March 19, 2014
Closed

build fails on windows (spawn issue) #15

Miroslav Bajtoš bajtos referenced this issue in strongloop/registry-validator March 21, 2014
Open

Windows #9

Brandon Winchester bwinchester referenced this issue in gruntjs/grunt-contrib-imagemin March 24, 2014
Closed

Empty images after running imagemin repeatedly #140

Christian Bromann christian-bromann referenced this issue in webdriverjs/grunt-webdriver March 25, 2014
Closed

Required Options Usage Example - event.js:72 error #12

Sergey Kamardin gobwas referenced this issue in Modernizr/Modernizr March 31, 2014
Open

Update build.js #1290

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.