Skip to content

Loading…

missing running scripts in forever list #175

Closed
demian85 opened this Issue · 84 comments
@demian85

Just updated node 0.6.3 and forever list is broken.
I started 4 scripts using forever start and forever list only shows 2 of them.
ps aux shows all my scripts running, but somehow forever did not register them.
Any clues?

@mmalecki

Duplicate of #144.
Fixed in #163.

@mmalecki mmalecki closed this
@demian85

OK, but the problem is... it's not fixed for me despite having the latest version!

@mmalecki

No, because it isn't merged in yet. Please help by testing #163 and current version.

@AvianFlu AvianFlu reopened this
@AvianFlu

I'm leaving this open until #163 is merged and published.

@demian85

logs don't work either, -e and -o parameters are completely ignored.

@edwardhotchkiss

This is still broken in master (just noticing the open/close/open/close)

@mmalecki

Yes it is. #163 didn't get merged yet, forever won't work on 0.6 until then

@indexzero
An elegant process monitor for anything written in Node. member

I think I fixed this here: d6c7590

I threw some debugging in the socket handlers there and data was actually getting returned, but the problem was that next() was being called eagerly before the data was actually parsed. I'm going to close this and say fixed in forever@0.7.5

@edwardhotchkiss @mmalecki @avianflu @demian85 Please confirm.

@indexzero indexzero closed this
@bradleyg

I just tried with forever@0.7.5 and it still looks broken.

http://hastebin.com/etxW72NCFn.m

@indexzero
An elegant process monitor for anything written in Node. member

@bradleyg what version of node are you running? That message "the sys module is now util" makes me think you are very out of date. In which case, please update to at least node@0.4.12 and try again.

@demian85

Isn't forever the cause of that warning? I'm sick of that message and i don't know how to avoid it. Using node 0.6

@edwardhotchkiss
@bradleyg

Hmm, just tried v0.6.3 and v0.6.4. Same problem and the warning.

@demian85

It's not working for me with node@0.6.3 & forever@0.7.5
I started 4 scripts and only one of them shows using the list cmd.

@indexzero
An elegant process monitor for anything written in Node. member

@demian85 I'll check it out on 0.6.3, but can anyone confirm it not working on 0.4.12?

@indexzero
An elegant process monitor for anything written in Node. member

Confirmed in 0.6.3.

@indexzero indexzero reopened this
@indexzero
An elegant process monitor for anything written in Node. member

Confirmed bug that is. I'm going to guess it has nothing to do with the forever code and just some upgrade path in node core since I can't reproduce this on 0.4.12

@indexzero
An elegant process monitor for anything written in Node. member

I've been receiving some reports from other users about daemon.node generally just failing with node@0.6.x. I installed 0.6.4 and attempted to run any of the sample code or tests I maintain, but none of it is working

@mmalecki You did the 0.6 upgrade work for daemon.node, did it ever work? I think .fork() is a much better approach, but I want to update the README on daemon.node to avoid confusion.

@edwardhotchkiss @demian85 @bradleyg We've got a fix for 0.6.x that we'll probably be rolling out in the next 1-2 days. Need to make sure we can have a clean break to >= node 0.6.4 across all of our projects. Bear with us; these kinds of breaking changes don't happen often.

@indexzero
An elegant process monitor for anything written in Node. member

@edwardhotchkiss @demian85 @bradleyg I've confirmed that pull-request #163 is not reproducing this behavior on node@0.6.4. Although it is going to be a couple of days before we can push it out you can link it locally now:

  $ git clone https://github.com/nodejitsu/forever.git
  $ cd forever
  $ git checkout 0.6
  $ [sudo] npm link
  $ forever start now-youre-on-that-branch.js

Let me know. Thanks.

@mmalecki

@indexzero I done some testing on daemon.node and it works. However, I suspect that there's some weird issue with file descriptors being lost in process or forking. It may cause unexpected behaviors. I'm not even sure how to debug it and these are just my thoughts about what could be going on here.

@mikosh75

same here
just installed node 0.6.5 and forever from git (0.7.5), i launch 1 forever script with

forever start myserver.js

i see node running in processes (with ps axwf), but

forever list

shows

The "sys" module is now called "util". It should have a similar interface.
info: No forever processes running
@edwardhotchkiss

Any ETA on this? I'm not trying to be a pain, but I'm using it in production and have migrated a lot of items forward to 0.6.5 and (stupidly) didn't think that forever wouldn't be a top priority (right now medium priority).

@indexzero
An elegant process monitor for anything written in Node. member

@edwardhotchkiss @mikosh75 This is fixed; it is simply not merged into master because of the breaking nature of the change between node 0.6.x and 0.4.x.

Have you tried this on the 0.6 branch? Can you reproduce this error when you use that branch? I was unable to.

You can install this directly from git:

  $ git clone https://github.com/nodejitsu/forever.git
  $ cd forever
  $ git checkout 0.6
  $ npm install
  $ [sudo] npm link
  # working version of forever with node@0.6.x is now installed globally
@mmalecki

Label changed. Should be fixed very soon.

Edit: @indexzero is right. I'm just working on a better solution.

@edwardhotchkiss

Thanks guys, fixed now!

@mikosh75

Thank you, with @indexzero install, "forever list" works, it outputs

The "sys" module is now called "util". It should have a similar interface.
info:   Forever processes running
data:       uid  command script                       forever pid   logfile                          uptime      
data:   [0] egL4 node    /home/ec2-user/www/server.js 20999   21002 /home/ec2-user/.forever/egL4.log 0:0:0:4.455 

it's ok, but this is my .forever folder:

.forever/
.forever/config.json
.forever/pids
.forever/sock
.forever/sock/forever.sock

as you can see i have no pid file and no log file at all.
I started app with

forever start /home/ec2-user/www/server.js

thanks for your hard work!

@indexzero
An elegant process monitor for anything written in Node. member

@mikosh75 The refactor of forever for node@0.6.x no longer uses *.pid files because it no longer depends on the daemon module.

@icodeforlove

thanks @indexzero its working for me now too

@mikosh75

Ah ok, i understand @indexzero ... thank you! And what about missing log file?

@coolsp

Could this be related to the fork not passing the command-line arguments?

(forever.js:346)
if (!process.send) {
fork(process.argv[1], ['start', script], {env: process.env});
process.exit(0);
}

@demian85
  $ git clone https://github.com/nodejitsu/forever.git
  $ cd forever
  $ git checkout 0.6
  $ npm install
  $ sudo npm link

i did that and the last command breaks with "not ok" and forever cmd cannot be found!

@xdenser

npm install -g

works for me
but where are log files ?

@xdenser

in fact it does not work - i.e. running scripts are restarted continously, while with 0.7.5 they were working but w/o forever list.

@demian85

using branch 0.6 but logs do not work either. OMFG we are all going to DIE!!

@ebaizel

fwiw i am getting the same problem of 'forever list' not showing any of my processes i've started with forever. items to note:

  • i don't see any *.fvr files in /tmp or anywhere on the system
  • i'm on node 0.6.3
  • not sure how to check which version of Forever i'm using. i installed it via 'npm install forever -g'
  • output of forever list is: forever list The "sys" module is now called "util". It should have a similar interface. info: No forever processes running
@cjpartridgeb

The sys error is coming from the watch package, fyi

@ebaizel

thanks for the heads up. what is the watch package? i'm guessing it's bundled with node.js since i didn't specifically install it.

@cjpartridgeb

@ebaizel The watch package is a dependency of forever, and is generally installed with forever.

Also appears logging isn't working due to the fact that it is now using child_process.fork (in the 0.6 branch) and not attaching listeners to stdin/stdout. However after a quick play with the source it appears the forked process has no stdin/stdout, I may be wrong here.

@tingham

Sorry if this seems dense, but there are a lot of redirects in this comment thread. Should npm install -g be installing a working version of forever for node 0.6.5 at present? I'm loathe to apply the 0.6 link if what is up now is intended to be the working model for my version of node.

Thanks

@demian85

Come on, let's solve this fucking bug once and FOREVER!!!

This should be ULTRA MAXIMUM PRIORITY

@stolsma

@cjpartridgeb You are right regarding fork and stdout/stderr!!

@indexzero @indutny child_process.fork only connects stdout and stderr to the parents stdout and stderr. I had the same problem with my 'node-fork' package. It wasn't possible to use customFds as option so I needed to rewrite the whole node 0.6 fork function... see my code how I did it as example (https://github.com/stolsma/node-fork/blob/master/lib/fork06.js#L127)

@stolsma

@indutny Sorry, included you instead of someone else!!! Stupid Github @ list thinghy...... :-(

@mmalecki

Yes, I'm aware of this stdout and stderr. I'll try working it out today (I basically wasted shitload of time trying to come up with some non-retarded fork solution)

@stolsma

@mmalecki Yep, I also had the idea that there had to be an easier solution, took me too much time so I just copied the fork code and changed it a little... :-(

@indexzero
An elegant process monitor for anything written in Node. member

@mikosh75 @stolsma @coolsp @demian85 @tingham @cjpartridgeb @ebaizel @xdenser @edwardhotchkiss

The crux of the problem here is that there was a major breaking change in node@0.6.x in that it will not fork(2) properly anymore. @mmalecki has been working on a major refactor to work around this issue but the current fix is simply not good enough since all processes still run in the foreground. If you want to see traction here +1 this issue: nodejs/node-v0.x-archive#2334

@cjpartridgeb

@tingham the package delivered by npm is NOT compatible with node@0.6.x

@indexzero @mmalecki
Thanks for your hard work on this guys

@indexzero
An elegant process monitor for anything written in Node. member

Thanks for the +1 on the issue above, but it looks like we're going to have to make due without it. The 'pseduo-fork' from @mmalecki in foreverjs/forever@c18993c

@cjpartridgeb

@indexzero @mmalecki What are your thoughts on capturing stdout/stderr with the pseudo-fork code? I was toying with the idea of passing the options via process.send ?

@AvianFlu

Please test the current master branch - forever should, at this point, work under 0.6.

@mlegenhausen

master? In the package.json is only node in version 0.4 supported.

@indexzero
An elegant process monitor for anything written in Node. member

@mlegenhausen Install from HEAD:

  $ git clone https://github.com/nodejitsu/forever.git
  $ cd forever
  $ [sudo] npm link
@mlegenhausen

forever list is working for me, but --watch is broken?

@icodeforlove

I wasn't able to get the logs to work in 0.6, but start stop and list work great

@mikosh75
@manobi

Node -> 0.6.6;
Forever install via branch 0.6;

Works:
forever start server.js

Break:
forever start server.js params params

@0xfede

forever list and stop doen't work for me

root@dev vvc]# forever start vvc.js
The "sys" module is now called "util". It should have a similar interface.
info: Forever processing file: vvc.js
[root@dev vvc]# forever list
The "sys" module is now called "util". It should have a similar interface.
info: No forever processes running
[root@dev vvc]# forever stop vvc.js
The "sys" module is now called "util". It should have a similar interface.
error: Forever cannot find process with index: vvc.js
[root@dev vvc]# ps -ef | grep node
root 30899 1 0 13:22 ? 00:00:00 node /usr/local/bin/forever start vvc.js
node 30900 30899 4 13:22 ? 00:00:00 node /var/www/node.js/vvc/vvc.js
root 30908 30861 0 13:22 pts/6 00:00:00 grep node
[root@dev vvc]# node --version
v0.6.6
[root@dev vvc]# npm --version
1.0.106

@manobi

@0xfede are you using 0.6 branch ?
0.6 branch fix some issues, but still blocking me to start my process when passing arguments to script.
Like:
forever start myscript manobi

Is my case a knowing issue ?
Let me know if I can do something to help you merge this.

@indexzero
An elegant process monitor for anything written in Node. member

This should be fixed in forever@0.8.0

@indexzero indexzero closed this
@wilsonpage

Wicked! Thanks guys, this is now working for me.

@demian85

Still having the same issue... this is gonna last forever....

w3:~/nsmetrics/application/scripts# ./start.sh 
info:   Forever processing file: proxyd.js
info:   Forever processing file: maind.js
info:   Forever processing file: saitod.js

w3:~/nsmetrics/application/scripts# forever list
info:   Forever processes running
data:       uid  command script    forever pid   logfile                 uptime      
data:   [0] jyvy node    proxyd.js 17136   17139 /root/.forever/jyvy.log 0:0:0:3.400

w3:~/nsmetrics/application/scripts# ps aux | grep saitod
root     17190  0.0  0.0  58872  7340 ?        Ss   12:59   0:00 node /usr/local/bin/forever start -e logs/saito.err -o logs/saito.log saitod.js
root     17191  0.1  0.0  55868  8508 ?        Sl   12:59   0:00 node /var/www/nsmetrics.com/application/scripts/saitod.js
root     17397  0.0  0.0   2164   768 pts/0    S+   13:02   0:00 grep saitod
@bradleyg

I'm having a similar issue: OSX, V0.6.6, 0.8.2

$ forever start server.js 
info:   Forever processing file: server.js
$ forever list
info:   No forever processes running
$ forever start server.js 
info:   Forever processing file: server.js
$ forever list
info:   Forever processes running
data:       uid  command script    forever pid   logfile                          uptime      
data:   [0] Kn9a node    server.js 34649   34660 /Users/Bradley/.forever/Kn9a.log 0:0:0:0.272 

$ ps aux | grep node
Bradley        35587  31.5  0.4  3046164  14988   ??  R     1:52pm   0:00.16 node /Users/Bradley/Dropbox/Dev/monstermash/server.js
Bradley        34773   0.8  0.3  3051000  14512   ??  Ss    1:48pm   0:01.97 node /usr/local/bin/forever start server.js
Bradley        34644   0.0  0.2  3040524   7808   ??  Ss    1:47pm   0:00.16 node /usr/local/bin/forever start server.js
Bradley        34645   0.0  0.4  3048648  14900   ??  S     1:47pm   0:00.41 node /Users/Bradley/Dropbox/Dev/monstermash/server.js
@indexzero
An elegant process monitor for anything written in Node. member

@bradleyg Unable to reproduce Mac OSX v0.6.6, forever 0.8.2. Try compiling node from source? Probably an installation issue.

[forever]$ forever start examples/server.js -p 8080
info:   Forever processing file: examples/server.js
[forever]$ forever start examples/server.js -p 8081
info:   Forever processing file: examples/server.js
[forever]$ forever list
info:   Forever processes running
data:       uid  command script                     forever pid logfile                          uptime      
data:   [0] kD6p node    examples/server.js -p 8080 402     403 /Users/Charlie/.forever/kD6p.log 0:0:0:5.334 
data:   [1] KgNT node    examples/server.js -p 8081 405     406 /Users/Charlie/.forever/KgNT.log 0:0:0:1.873 
@indexzero
An elegant process monitor for anything written in Node. member

@demian85

  1. Operating system?
  2. npm version?
  3. How node was installed?

A couple of other attempts to fix / debug:

  1. Can you reproduce after rm -rf ~/.forever?
  2. Are you running anything as sudo?
  3. Contents of start.sh?
  4. After running start.sh can you paste the result of ls -la ~/.forever/sock?
@indexzero
An elegant process monitor for anything written in Node. member

@demian85 I am unable to reproduce this on both MacOS and Ubuntu using the following start.sh script:

forever start examples/server.js -p 8080
forever start examples/server.js -p 8081
forever start examples/server.js -p 8082
forever list

This is the result I get

[forever]$ ./start.sh 
info:   Forever processing file: examples/server.js
info:   Forever processing file: examples/server.js
info:   Forever processing file: examples/server.js
info:   Forever processes running
data:       uid  command script                     forever pid logfile                          uptime      
data:   [0] VTwL node    examples/server.js -p 8080 469     471 /Users/Charlie/.forever/VTwL.log 0:0:0:0.826 
data:   [1] 2ilX node    examples/server.js -p 8081 472     474 /Users/Charlie/.forever/2ilX.log 0:0:0:0.556 
data:   [2] _pUA node    examples/server.js -p 8082 475     477 /Users/Charlie/.forever/_pUA.log 0:0:0:0.285 
@manobi

@demian85 Are you using @visonmedia/n to install node ?

@demian85

Yes, I think node was installed using "n latest".
start.sh simply does "forever start script.js -e err.log -o log.log"

I'm at home now, I'll try to debug it on Monday.

@indexzero
An elegant process monitor for anything written in Node. member

@demian85 Try installing node from source. This may have been an n problem all along

@bradleyg

I've moved my app to production, set up a new server (Ubuntu 11.10) with v0.6.6 and v0.8.2 and still have the same problem.

root@li305-57:/home/ubuntu/www/# forever start server.js 
info:   Forever processing file: server.js
root@li305-57:/home/ubuntu/www/# forever list
info:   No forever processes running
root@li305-57:/home/ubuntu/www/# node -v
v0.6.6
root@li305-57:/home/ubuntu/www/# npm -v
1.1.0-beta-4
root@li305-57:/home/ubuntu/www/# ps aux | grep node
root     30366  0.2  1.4  58252  7368 ?        Ss   18:55   0:00 node /usr/local/bin/forever start server.js
root     30367  1.1  2.1  58160 10808 ?        Sl   18:55   0:00 node /home/ubuntu/www/server.js
@indexzero
An elegant process monitor for anything written in Node. member

@bradleyg I was unable to reproduce this using the installation instructions below:

apt-get -y update 
apt-get -y install git-core build-essential libssl-dev pkg-config
git clone http://github.com/joyent/node.git
cd node
git checkout v0.6.7
./configure
make
make install
cd ../
npm install forever -g
git clone git://github.com/nodejitsu/forever.git
cd forever

Here is the output I receieve:

root@ocelot-forever:~/forever# forever start examples/server.js -p 8080
info:   Forever processing file: examples/server.js
root@ocelot-forever:~/forever# forever list
info:   Forever processes running
data:       uid  command script                     forever pid   logfile                 uptime     
data:   [0] dRaq node    examples/server.js -p 8080 11296   11340 /root/.forever/dRaq.log 0:0:0:0.43 

Both your and @demian85 remaining issues seem to be related to installation procedure of node which may not be building native modules correctly. Build node from source and make sure that openssl and pkgconfig are properly installed.

@indexzero
An elegant process monitor for anything written in Node. member

@bradleyg For reference, I spun up a fresh VM of Ubuntu 11.10 (Oneiric Ocelot), with node@0.6.7 and forever@0.8.2 on Rackspace cloud.

@indexzero
An elegant process monitor for anything written in Node. member

I've created a wiki page based on this issue: https://github.com/nodejitsu/forever/wiki/Detailed-installation

@bradleyg

Reinstalled node the same way as you but to no avail... this is on a fresh install at Linode. Anything else you want me to try?

root@li305-57:/home/ubuntu/www/# forever start server.js 
info:   Forever processing file: server.js
root@li305-57:/home/ubuntu/www/# forever list
info:   No forever processes running
root@li305-57:/home/ubuntu/www/# node -v
v0.6.7
root@li305-57:/home/ubuntu/www/# ps aux | grep node
root     14594  0.1  1.4  58320  7380 ?        Ss   19:57   0:00 node /usr/local/bin/forever start server.js
root     14595  0.8  2.1  57960 10708 ?        Sl   19:57   0:00 node /home/ubuntu/www/server.js
@indexzero
An elegant process monitor for anything written in Node. member

Sorry. No clue. It has to do with your node.js installation though. Not sure what differs between Linode and Rackspace. Maybe it's time to switch.

@indexzero
An elegant process monitor for anything written in Node. member

@bmeck Suggested it was probably a permissions issue. Clearly the files representing the sockets which the forever list and forever start ... processes use to communicate are not properly being created.

Are you using sudo or anything like that?

@bradleyg

No, I'm not using sudo. Oddly it works the 2nd time round, but I still have 2 processes running.

ubuntu@li305-57:~/www/$ rm -rf ~/.forever
ubuntu@li305-57:~/www/$ forever start server.js 
info:   Forever processing file: server.js
ubuntu@li305-57:~/www/$ forever list
info:   No forever processes running
ubuntu@li305-57:~/www/$ forever start server.js 
info:   Forever processing file: server.js
ubuntu@li305-57:~/www/$ forever list
info:   Forever processes running
data:       uid  command script    forever pid  logfile                        uptime    
data:   [0] Kpam node    server.js 3890    3930 /home/ubuntu/.forever/Kpam.log 0:0:0:0.6 
@mikosh75
@indexzero
An elegant process monitor for anything written in Node. member

@mikosh75 Looks like github had some downtime. Also, make sure you have git installed on your system.

@mikosh75

@indexzero 404 persists...
both https://github.com/bmeck/node-fork.git and git://github.com/bmeck/node-fork.git gives 404
git is installed

Please, can you investigate with github?
or maybe can i workaround downloading zipball and install it manually before forever@0.8.2 ?

thanks for your work!

@mikosh75

sorry, i think i have a network issue...my provider block tcp port 9418 (git port)
If i try from the command line git clone http://github.com/bmeck/node-fork.git (with http protocol instead of git) it works!
But my issue now is that i cannot make my provider open that port for me...

Maybe is it possible in https://registry.npmjs.org/forever to let git clone node-fork over http and not over git protocol?
Or maybe have node-fork as a npm package?
please, i need forever@0.8.2 on this machine (on this provider...)
thanks again!

@stolsma

I don't know the reason why @bmeck is referencing his git repo of node-fork (please let me know if there is a blocking issue) but I have already pushed his commits to my (original node-fork) master version and published it to npm. Maybe you can use that one ??

Change the git reference in package.json to 'node-fork' and you should get the same version but then from NPM...

@mikosh75

@stolsma only forever package maintainer can "Change the git reference in package.json to 'node-fork' ", right?
Every time i do

npm install forever -g

it downloads package.json file, right? Or am I missing something?
Thanks
Mikosh

@indexzero
An elegant process monitor for anything written in Node. member

@stolsma We used the git repository because when we published it you had not yet accepted them. I always knew you would :), but just wanted to publish it ASAP.

I'll update to the npm package and bump forever@0.8.3.

@indexzero
An elegant process monitor for anything written in Node. member

Actually @stolsma we need this commit pulled in: bmeck/node-fork@e174563. I've opened a pull request here stolsma/node-fork#4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.