Close File Handles on HUP Signal #627

Closed
avaitla opened this Issue Oct 10, 2013 · 3 comments

Projects

None yet

2 participants

@avaitla
Contributor
avaitla commented Oct 10, 2013

This is related to #624 (comment). Please let me know if any additional details are required, or if I can assist in some way.

I first proposed closing explicitly, but probably not the best way to do it. Another option is to hold on to the same logger object, and just update the cfg, and call reopen on it? This is probably more complicated though...

The issue was first noticed on CentOS release 6.4 (Final) x86_64 machine, it can still be reproduced on that os, it has sporadically failed on OS X 10.8:

$ gunicorn -c conf.py myapp:app &
[1] 18042
$ lsof -p `cat gunicorn.pid` | grep accesslog_test.txt
gunicorn 18042 avaitla    5w   REG    202,2        0     3305 /home/avaitla/gunicorn/accesslog_test.txt
$ lsof -p `cat gunicorn.pid` | grep errorlog_test.txt
gunicorn 18042 avaitla    1u   REG    202,2     2522     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    2u   REG    202,2     2522     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    3u   REG    202,2     2522     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    4w   REG    202,2     2522     3304 /home/avaitla/gunicorn/errorlog_test.txt

$ kill -HUP `cat gunicorn.pid`
$ lsof -p `cat gunicorn.pid` | grep accesslog_test.txt
gunicorn 18042 avaitla    5w   REG    202,2        0     3305 /home/avaitla/gunicorn/accesslog_test.txt
gunicorn 18042 avaitla   15w   REG    202,2        0     3305 /home/avaitla/gunicorn/accesslog_test.txt
$ lsof -p `cat gunicorn.pid` | grep errorlog_test.txt
gunicorn 18042 avaitla    1u   REG    202,2     3145     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    2u   REG    202,2     3145     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    3u   REG    202,2     3145     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    4w   REG    202,2     3145     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla   13u   REG    202,2     3145     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla   14w   REG    202,2     3145     3304 /home/avaitla/gunicorn/errorlog_test.txt

$ kill -HUP `cat gunicorn.pid`
$ lsof -p `cat gunicorn.pid` | grep accesslog_test.txt
gunicorn 18042 avaitla    5w   REG    202,2        0     3305 /home/avaitla/gunicorn/accesslog_test.txt
gunicorn 18042 avaitla   11u   REG    202,2        0     3305 /home/avaitla/gunicorn/accesslog_test.txt
gunicorn 18042 avaitla   15w   REG    202,2        0     3305 /home/avaitla/gunicorn/accesslog_test.txt
$ lsof -p `cat gunicorn.pid` | grep errorlog_test.txt
gunicorn 18042 avaitla    1u   REG    202,2     3768     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    2u   REG    202,2     3768     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    3u   REG    202,2     3768     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    4w   REG    202,2     3768     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla    9u   REG    202,2     3768     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla   10u   REG    202,2     3768     3304 /home/avaitla/gunicorn/errorlog_test.txt
gunicorn 18042 avaitla   14w   REG    202,2     3768     3304 /home/avaitla/gunicorn/errorlog_test.txt
@avaitla
Contributor
avaitla commented Oct 13, 2013

The reason I think this doesn't have to do with close_on_exec, is because the Master process doesn't get recreated when HUP signal is received, only the workers. So I am thinking that close_on_exec doesn't apply in this case, since we are dealing with the masters handles and not the workers.

@benoitc benoitc added a commit that referenced this issue Nov 19, 2013
@benoitc makes sure that file handles are correctly reopened on HUP
This change makes sure we don't recreate the Logger class here so we
don't reinitialise the log handlers.

fix #627
48498e9
@benoitc
Owner
benoitc commented Nov 19, 2013

@avaitla the branch fix/627 should fix it. On my machine I have now:

$ lsof -p 63703 |grep error.log 
python2.7 63703 benoitc    1u   REG                1,2       389 12552334 /Users/benoitc/work/gunicorn_env/src/gunicorn/examples/error.log
python2.7 63703 benoitc    2u   REG                1,2       389 12552334 /Users/benoitc/work/gunicorn_env/src/gunicorn/examples/error.log
python2.7 63703 benoitc    3u   REG                1,2       389 12552334 /Users/benoitc/work/gunicorn_env/src/gunicorn/examples/error.log
python2.7 63703 benoitc    4w   REG                1,2       389 12552334 /Users/benoitc/work/gunicorn_env/src/gunicorn/examples/error.log
$ kill -HUP 63703
$ lsof -p 63703 |grep error.log 
python2.7 63703 benoitc    1u   REG                1,2       883 12552334 /Users/benoitc/work/gunicorn_env/src/gunicorn/examples/error.log
python2.7 63703 benoitc    2u   REG                1,2       883 12552334 /Users/benoitc/work/gunicorn_env/src/gunicorn/examples/error.log
python2.7 63703 benoitc    3u   REG                1,2       883 12552334 /Users/benoitc/work/gunicorn_env/src/gunicorn/examples/error.log
python2.7 63703 benoitc    4w   REG                1,2       883 12552334 /Users/benoitc/work/gunicorn_env/src/gunicorn/examples/error.log
@avaitla
Contributor
avaitla commented Dec 2, 2013

Thanks for taking a look. It works great on my end as well. Thanks!

@avaitla avaitla closed this Dec 2, 2013
@benoitc benoitc added a commit that referenced this issue Sep 23, 2015
@benoitc @tilgovi + tilgovi makes sure that file handles are correctly reopened on HUP
This change makes sure we don't recreate the Logger class here so we
don't reinitialise the log handlers.

fix #627
365e80d
@benoitc benoitc added a commit that referenced this issue Sep 23, 2015
@benoitc @tilgovi + tilgovi makes sure that file handles are correctly reopened on HUP
This change makes sure we don't recreate the Logger class here so we
don't reinitialise the log handlers.

fix #627
2221225
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment