Process can not be killed

[jonywtf]

Hi!
In my Gruntfile.coffee i have tasks:

shell:
    stopall:
        options:
            failOnError: false
        command: 'pm2 kill'
    backend:
        command: 'coffee pm2.coffee backend ' + backend_app_path + ' ' + logs_path
    restfull:
        command: 'coffee pm2.coffee restfull ' + restfull_app_path + ' ' + logs_path

and i have script for restart pm2 process

shelljs = require 'shelljs'
return if process.argv < 2

name = process.argv[2]
script_path = process.argv[3]
log_path = process.argv[4]
log_path += name

shelljs.exec 'pm2 stop ' + name
shelljs.exec 'pm2 start ' + script_path + ' -f --max-memory-restart 50M --name ' + name + ' --output ' + log_path + '.out --error ' + log_path + '.err '

Sometimes in pm2 logs i get errors

PM2: 2015-03-05 05:34:13: 16172 pid can not be killed { [Error: spawn ENOMEM] code: 'ENOMEM', errno: 'ENOMEM', syscall: 'spawn' }
PM2: 2015-03-05 05:34:14: Process with pid 16172 already exists

Help me please! How can i fix this trouble?

[jshkurti]

https://swordfish1987.wordpress.com/2009/04/09/enomem-error/

[jshkurti]

Although, I'm looking if we can do something PM2-side :)

[jonywtf]

I have 70-150Mb free memory on my DigitalOcean droplet. And i can kill process with kill -9 pid.
Will try to add swap or to use droplets with more memory. Thanks!

[manuelmhtr]

I can't delete my processes either. Using PM2 v0.12.7 and node v0.12.0 :(
PM2 stop showing the processes on list, but they still there.

[jshkurti]

Does your app by any chance catch SIGTERM signal ?

[neonstalwart]

i have this same problem that started to appear when i upgraded to 0.12.7 from 0.12.3. for me, the problem occurs about once a day when i restart (often it's the first time i restart each day so it's been ~12 hours since the last restart) and i have about 5 - 10 restarts per day with continuous deployment.

i don't have anything catching SIGTERM.

for the sake of anyone else who might read this thread and not know how to get back up running when this happens, here's what i do:

pm2 kill
pm2 startOrRestart app.json

[jshkurti]

What logs do you got @neonstalwart when this occurs ?

[neonstalwart]

extracted from ~/.pm2/pm2.log

2015-03-08 23:06:20: Process with pid 14291 still not killed, retrying...
2015-03-08 23:06:20: Process with pid 14291 still not killed, retrying...
2015-03-08 23:06:20: Process with pid 14291 still not killed, retrying...
2015-03-08 23:06:20: Process with pid 14291 still not killed, retrying...
2015-03-08 23:06:20: Process with pid 14291 still not killed, retrying...
2015-03-08 23:06:20: App closed with code: 143
2015-03-08 23:06:20: App name:API id:0 exited with code 143
2015-03-08 23:06:20: Process with pid 14291 killed
2015-03-08 23:06:20: Starting execution sequence in -fork mode- for app name:API id:0
2015-03-08 23:06:20: App name:API id:0 online
2015-03-09 06:51:31: Reloading logs...
2015-03-09 06:51:31: Reloading logs for process id 0
2015-03-09 06:51:34: Reloading logs...
2015-03-09 06:51:34: Reloading logs for process id 0
2015-03-09 09:57:09: Process with pid 13043 still not killed, retrying...
2015-03-09 09:57:09: Process with pid 13043 still not killed, retrying...
2015-03-09 09:57:09: Process with pid 13043 still not killed, retrying...
2015-03-09 09:57:09: Process with pid 13043 still not killed, retrying...
2015-03-09 09:57:09: Process with pid 13043 still not killed, retrying...
2015-03-09 09:57:09: Process with pid 13043 killed
2015-03-09 09:57:09: Process with pid 13043 already exists
2015-03-09 09:57:13: PM2 is being killed via kill method
2015-03-09 09:57:13: RPC socket closed
2015-03-09 09:57:13: PUB socket closed
2015-03-09 09:57:34: [PM2][WORKER] Started with refreshing interval: 30000
2015-03-09 09:57:34: [[[[ PM2/God daemon launched ]]]]
2015-03-09 09:57:34: BUS system [READY] on port /home/deployer/.pm2/pub.sock
2015-03-09 09:57:34: RPC interface [READY] on port /home/deployer/.pm2/rpc.sock
2015-03-09 09:57:34: Starting execution sequence in -fork mode- for app name:API id:0
2015-03-09 09:57:34: App name:API id:0 online

23:06 - successful restart
06:51 - log rotation
09:57:09 - unsuccessful restart
09:57:13 - kill pm2 daemon
09:57:34 - app successfully restarted

[jshkurti]

It looks like the process exited in an unusual way and PM2 didn't receive the 'exit' signal.
I just released PM2 0.12.8 in which I have refactored a bit the stop/restart procedure.
Can you guys give it a try ?

[neonstalwart]

It looks like the process exited in an unusual way and PM2 didn't receive the 'exit' signal

that's strange since i simply use pm2 restart 0.

I just released PM2 0.12.8 in which I have refactored a bit the stop/restart procedure.
Can you guys give it a try ?

yeah, i'll try it. i'll definitely know by tomorrow if this has improved or not because i see this error the first time i restart my processes everyday.

[jshkurti]

Hey, it looks like it's logs reloading which triggers the bug.
Can you pkill --signal SIGUSR2 PM2 (it forces log reloading) and try a restart after that ?

[jshkurti]

Try it with npm i -g pm2@0.12.7 then npm i -g pm2@0.12.8 and finally with npm i -g git://github.com/Unitech/PM2#development please :)

[neonstalwart]

ok, trying... 0.12.7 fails. i'll try the others

[neonstalwart]

i see 0.12.7 fail. 0.12.8 and #development both work for me.

[jshkurti]

Great, so it's fixed :)

[neonstalwart]

fyi - 0.12.3 is the earliest version that worked (and was the previous version i was using before 0.12.7) so somewhere between 0.12.3 and 0.12.4 was when the bug was introduced.

[jshkurti]

Thanks again for the feedback 👍