fork mode does not close stdout/stderr files on reload

[aronatkins]

This is with 0.9.6, but is likely still a problem in 0.10.8.

Using lsof, I am able to see the pm2 application retain file handles long after signaling a log-reload. This is because the stdout and stderr streams are not closed prior to re-open in ForkMode.js.

We start our application with a pm2 command-line like:

pm2 start app.js \
    --error logs/app.err.log \
    --output logs/app.out.log \
    --instances 1 \
    --execute-command

You can see pm2 hanging on to log files with the following steps:

#1. start application as above.
#2. run lsof to see open filehandles prior to log rotate.
lsof -p $(cat .pm2/pm2.pid)
#3. rename existing logs (just like logrotate)
mv logs/app.err.log logs/app.err.log.1
mv logs/app.out.log logs/app.out.log.1
#4. trigger the reload.
kill -USR2 $(cat .pm2/pm2.pid)
#5. run lsof again to see that pm2 now has open filehandles to both the old and new log files.
lsof -p $(cat .pm2/pm2.pid)

I have this patched locally with the following redefinition of _reloadLogs in ForkMode.js but am not certain that this is the correct solution which also avoids losing log messages; it is similar to the code already in place in ProcessContainer.js:

          cspr._reloadLogs = function(cb) {
            stdout.end();
            stderr.end();
            startLogging(cb);
          };

I expect that these filehandles will be reclaimed when the garbage collector reaps the objects, but this is not always guaranteed to happen.

This is causing a problem for us because these log files are subsequently removed. The filesystem is not reclaiming the space because pm2 still has a live filehandle; the file is not truly deleted until the filehandle is closed.

[Unitech]

This has been fixed prior 0.11.1 and next 0.12 release, thanks for the feedback

[jrnt30]

We are using 0.12.1 and still seeing this error occur. Did this resolve the issues for others? If so, I will take a closer look to ensure it's not on our end.

[aronatkins]

I just tried 0.12.1 and do not see lingering file handles, but this is with a toy app.

How high is your log volume? I'm wondering if writes to your logs fall behind and those need to flush before the file handle is finally closed. Does the pm2 continue to have an open file handle to the rotated file-location for a long time? Is that file still changing?

[jrnt30]

I believe the issue we were seeing had to do more with permissions. If you are running PM2 as a privileged user and run pm2 logs as root, there are some file permission issues that crop up. I have not been able to reproduce the file handle issue without that thrown into the mix.

Thanks for your response.