Undefined processes on Max crash limit #343

Closed
sambol opened this Issue Sep 27, 2012 · 7 comments

Projects

None yet

7 participants

@sambol

Steps to reproduce

Make a process that crashes when instructed. (Example at https://gist.github.com/3793584 )
Start running under forever with command: forever start -l break.txt -m 1 break.js
Make sure it is running correctly under forever: forever list
Kill Process (in my example visit "http://localhost/10002").
run forever list again

Expected

Nothing about the process is returned

Actual

forever list returns:
data: [0] undefined node undefined undefined 0:0:0:0.0

In the logfile for that process, at the botton is the following:

Forever detected script exited with code: 1

node.js:201
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
Error: Cannot stop process that is not running.
    at Array.0 (/usr/local/lib/node_modules/forever/lib/forever/monitor.js:327:26)
    at EventEmitter._tickCallback (node.js:192:40)

My System

Tested on Debian and Ubuntu server editions.
Debian
Node 0.6.9 (Latest through Aptitude)
Forever 0.9.2 (Latest through npm)

Ubuntu
Node 0.8.4
Forever 0.9.2

@lwille

I'm also getting this error sometimes when a monitored script exits, although I'm using forever.Monitor instead of the CLI.
The error seems related to the watch option:

new (forever.Monitor)(scriptFile, {
  watch: true,
  watchDirectory: watchDir
})

I added a stacktrace to the place where it occurs:

    at Monitor.kill (./node_modules/forever/node_modules/forever-monitor/lib/forever-monitor/monitor.js:312:17)
    at Monitor.restart (./node_modules/forever/node_modules/forever-monitor/lib/forever-monitor/monitor.js:291:15)
    at ./node_modules/forever/node_modules/forever-monitor/lib/forever-monitor/plugins/watch.js:66:17
    at ./node_modules/forever/node_modules/watch/main.js:77:19
    at Object.oncomplete (fs.js:297:15)

./node_modules/forever/node_modules/forever-monitor/node_modules/broadway/node_modules/eventemitter2/lib/eventemitter2.js:270
          throw arguments[1]; // Unhandled 'error' event
                         ^
Error: Cannot stop process that is not running.
    at ./node_modules/forever/node_modules/forever-monitor/lib/forever-monitor/monitor.js:315:26
    at process._tickCallback (node.js:244:9)

2.6.36-gentoo-r5
Forever 0.10.0
node 0.8.12

@a-shark

I have what appears to be a working quick & dirty fix for this.

Modify the letChildDie() definition in monitor.js to look like:

    function letChildDie() {
      self.running = false;
      self.forceStop = false;
      self.emit('exit', self, spinning);
      self.kill(true);
    }

with the key line being the self.kill(true);

I have no idea if this is the right way to fix it.

I haven't run any formal test against it.

I have fired up scripts with simple combinations of sleep and exit, with and without the -m option, and they appeared to do exactly what they're supposed to.

So, anyone else banging your head against this rather large bug, try this out, but test it to your satisfaction first. I say again: quick & dirty fix; 90+% of forever's functionality has not been tested with this fix in place.

I'd fix it properly and submit pull request, but honestly I can only stand digging around in Javascript for so long :-)

@smoodiver

I believe I've traced this back to a section of monitor.js within the forever-monitor package:

if (!this.running) {
  //
  // TODO: Return settings from this forever instance
  // with a state indicator that it is currently stopped.
  //
  return {};
}

childData = {
  ctime: this.ctime,
  command: this.command,
  file: this.args[0],
  foreverPid: process.pid,
  logFile: this.logFile,
  options: this.args.slice(1),
  pid: this.child.pid,
  silent: this.silent,
  uid: this.uid,
  spawnWith: this.spawnWith
};

The 'running' flag has already been set to false when the max limit is reached. Subsequent calls from 'forever list' make socket calls to the process for their child data but because of the condition above all they get back is an empty hash.

Resolving this to return childData always only passes the problem further downstream. With some small logic changes stop/stopall works but restart/restartall is proving problematic. If I crack it I'll submit a pull request :)

@smoodiver

Found it... the whole restarting stuff assumed that the process was still running and hadn't reached its max restart limit. I've added some extra logic but it feels a bit hacky. I'll tidy up the logging of a stopped process so it doesn't lie about its uptime then submit a pull request

@smoodiver smoodiver referenced this issue in foreverjs/forever-monitor Mar 12, 2013
Closed

Crashed process returns undefined #22

@indexzero
foreverjs member

Should be fixed in master because we merged in the change to forever-monitor

@indexzero indexzero closed this Apr 21, 2013
@carolineBda

We have the same issue with:

Node: 0.10.15
Forever: 0.10.8
CentOS 6.3

this is the stacktrace:

/var/sky/popcorn-megadashboard/.nvm/v0.10.15/lib/node_modules/forever/node_modules/forever-monitor/node_modules/broadway/node_modules/eventemitter2/lib/eventemitter2.js:283
throw arguments[1]; // Unhandled 'error' event
^
Error: Cannot stop process that is not running.
at /var/sky/popcorn-megadashboard/.nvm/v0.10.15/lib/node_modules/forever/node_modules/forever-monitor/lib/forever-monitor/monitor.js:332:26
at process._tickCallback (node.js:415:13)

@christianpbrink

Same issue with forever 0.11.1 and forever-monitor 1.2.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment