Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Lost stdout output when executed through child_process.spawn #120

Closed
simondean opened this Issue · 17 comments

6 participants

@simondean

Hi. I'm writing a node module for executing cucumber-js features across multiple processes and multiple machines. I've hit an issue when executing cucumber-js via child_proces.spawn. 5-15% of the time, cucumber-js exits without outputting anything to stdout (normally I'm using the JSON formatter).

You can see the bug by running the following code (spawn_cucumber.js):

var ChildProcess = require('child_process');

var count = 1000;
var finishCount = 0;
var failCount = 0;

function test() {
  var child = ChildProcess.spawn('node', ['node_modules/cucumber/bin/cucumber.js', 'features'], {
    cwd: '.',
    stdio: ['ignore', 'pipe', process.stderr],
    env: process.env
  });

  var stdoutData = [];

  child.stdout.on('data', function(data) {
    stdoutData.push(data);
  });

  child.on('exit', function(code, signal) {
  });

  child.on('close', function(code, signal) {
    finishCount++;

    if (stdoutData.length == 0) {
      failCount++;
    }

    if (finishCount < count) {
      test();
    }
    else {
      console.log('Flush succeeded ' + (count - failCount) + ' times');
      console.log('Flush failed ' + failCount + ' times');
      console.log('Flush failed ' + (failCount / count * 100) + '% of the time');
    }
  });

  child.on('disconnect', function() {
    console.log('disconnect');
  });
}

test();

There's also a copy of the above code here: git://github.com/simondean/node-spawn-lost-output-bug.git

Here's some example ouput:

>node spawn_cucumber.js
Flush succeeded 929 times
Flush failed 71 times
Flush failed 7.1% of the time

The issue is in bin/cucumber.js:

  var code = succeeded ? 0 : 1;
  var exitFunction = function() {
    process.exit(code);
  };

  // --- exit after waiting for all pending output ---
  var waitingIO = false;
  process.stdout.on('drain', function() {
    if (waitingIO) {
      // the kernel buffer is now empty
      exitFunction();
    }
  });
  if (process.stdout.write("")) {
    // no buffer left, exit now:
    exitFunction();
  } else {
    // write() returned false, kernel buffer is not empty yet...
    waitingIO = true;
  }

If the above is replaced with the following it fixes the issue:

  var code = succeeded ? 0 : 1;

  process.on('exit', function() {
    process.exit(code);
  });

Thanks
Simon

@jbpros
Owner

Could you make this a pull request?

@simondean

Will do. Thanks

@simondean simondean referenced this issue from a commit in simondean/cucumber-js
@simondean simondean Ensure no stdout output is lost
Fixes issue #120
b585d45
@simondean simondean referenced this issue from a commit in simondean/cucumber-js
@simondean simondean Ensure no stdout output is lost
Fix for issue #120
1da47fb
@simondean

I've found that resource leaks (thinks that keep the event loop active) can cause cucumber-js to never exit. The fix is to replace

  var code = succeeded ? 0 : 1;

  process.on('exit', function() {
    process.exit(code);
  });

with

  var code = succeeded ? 0 : 1;

  process.on('exit', function() {
    process.exit(code);
  });

  var timeoutId = setTimeout(function () {
    console.error('Cucumber process timed out after waiting 60 seconds for the node.js event loop to empty.  There may be a resource leak.  Have all resources like database connections and network connections been closed properly?');
    process.exit(code);    
  }, 60 * 1000);

  if (timeoutId.unref) {
    timeoutId.unref();
  }
  else {
    clearTimeout(timeoutId);
  }

This is compatible with old and new versions of node but you only get the benefit of the timeout with node 0.10 onwards (see the if (timeoutId.unref) conditional above). See http://nodejs.org/api/timers.html#timers_unref for the documentation on the new unref method.

@simondean simondean closed this
@simondean simondean reopened this
@jbpros
Owner

Hi Simon,

Is this still relevant?

Also, I'm thinking the warning message and the timeout could become pretty annoying when testing applications with servers running. An express application, for example, will not let the event loop empty and forcing a process.exit() call right after the tests were run is a pretty efficient way of short-circuiting this.

WDYT?

@jbpros
Owner

@simondean Please reopen if you still experience this issue.

@jbpros jbpros closed this
@simonlampen

Hi Simon and Julien,
This still seems relevant, I was experiencing a test failure running the cucumber-js tests themselves issue , (cli.features) because the stdout was not flushed, and Simon your patch does fix it.
I am on the latest node (v0.10.26) on Win7 64bit.

Cheers
Simon

@jbpros jbpros referenced this issue from a commit
@simondean simondean Ensure no stdout output is lost
Fix for issue #120
da5c977
@jbpros jbpros referenced this issue from a commit
@eddieloeffen eddieloeffen Document AfterFeatures event (close #171)
I spent ages doing the wrong setup for AfterFeatures to try and close my browser at the end of the tests, so I thought that including a test and better documentation might help other people.

This also fixes the unimplemented cli.feature step, and failing assert due to console coloring.

I had to apply the patch from comments in #120 to get these tests to run correctly.

Conflicts:
	bin/cucumber.js
	features/cli.feature
	features/step_definitions/cli_steps.js
ca9bea3
@joshtombs

I'm still experiencing this problem. When i run

require('cucumber').Cli(specs).run(function (succeeded) {
  var code = succeeded ? 0 : 1;

  process.on('exit', function () {
    console.log('exiting')
    process.exit(code);
  });

  var timeoutId = setTimeout(function () {
    console.error('Cucumber process timed out after waiting 60 seconds for the node.js event loop to empty.  There may be a resource leak.  Have all resources like database connections and network connections been closed properly?');
    process.exit(code);
  }, 60 * 1000);

  if (timeoutId.unref) {
    timeoutId.unref();
  }
  else {
    clearTimeout(timeoutId);
  }
});

And a test fails, nothing is logged. I am trying to log the duration of the tests right before process.exit but even the log "exiting" isnt being executed.

Not sure if this issue needs reopening, or if maybe theres a bug in my code but for whatever reason I can't catch process.on('exit')

@simondean

@joshtombs Which version of node.js are you using?

@simondean

process.on('exit') will only fire when the node.js event loop is empty. Having something like an open database or HTTP connection will prevent the process from exiting. Does the timeout fire? Do you see the output from the console.error?

@joshtombs

screen shot 2014-08-21 at 3 07 50 pm

Above is what's being outputted from terminal. As you can see my test failed and the error was reported. The feature file finished, and this is at the end of the "(::) failed steps (::)" report. It displays the error and then exits. The log isnt being hit in

 process.on('exit', function(){
    console.log('exiting');
    process.exit(code)
});

Then as shown when I check the status of the previous exit it is 0. Thanks!

@simondean

Is spec your command line args for Cucumber?

Does it work properly if you use the offical bin for cucumber-js? E.g. ./node_modules/.bin/cucumber-js

@joshtombs

Yeah, so I forked cucumber-js and ran the default cucumber-js features/ directory and everything works as expected. When I manipulated the ./bin/cucumber.js file and adjusted the "on exit" function, it worked as I would expect it to, logging "exiting" right before exiting the process. Also, when running a failing feature it exits with the correct exit code.

That said, I am using cucumber-js inside a project that I'm working on, which can just be thought of as a layer that sits on top of cucumber right now. specs is just an array that is passed instead of the true process.argv. We manipulate command line arguments and have created some of our own, so we groom them and then pass them in specs to cucumber in a way that cucumber understands.

I would like to be able to pass my own callback function to the runtime, so that is why I am just calling require('cucumber').Cli(specs).run(func...).

@simondean

Does it sound like the bug is maybe in your code then? It's worth checking you don't have more than 1 listener for the process exit event. It's worth checking the docs for the exit event too: http://nodejs.org/api/process.html#process_event_exit It gives some guidance for things not to do in an exit listener.

Calling process.exit from the exit event is really trick. The process would have exited anyway, the trick just provides a way to change the exit code from the default of zero.

@joshtombs

@simondean you were right! So after digging further into our code, I noticed that we were using our own

@registerHandler "AfterFeatures", (event, callback) =>
    # code

and not actually calling that callback function! so the tree walker wasn't actually finishing.

Thanks for the help

@kAworu

Cucumber hanging 60sec is quite annoying, there is no way to know what ressource is keeping the event loop busy and a lot of modules (ex. connect-mongostore, zombiejs) don't provide a way to clean their ressource(s).

@bitplanets

For mongoose you can do:

var hooks = function(){
    this.After(function(cb){
        this.mongoose.connection.close()
        cb();
    });
};
module.exports = hooks;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.