New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cpu spike on terminal resize #1269

Closed
w-p opened this Issue May 17, 2016 · 7 comments

Comments

Projects
None yet
4 participants
@w-p

w-p commented May 17, 2016

I'm not sure if what follows is an issue with gunicorn, Terminator, both, or neither. I run Debian (8), I use Terminator to split up my shells, I run web apps with gunicorn. Pretty straightforward.

While resizing any of the terminals adjacent to that which is running gunicorn, for every movement event tracked, the two lines below are printed and multiple cores of my cpu (i7 4712) are pegged at 100% (or close), until I kill off gunicorn and start it up again.

[2016-05-17 16:09:55 -0400] [21121] [INFO] Handling signal: winch
[2016-05-17 16:09:55 -0400] [21121] [DEBUG] SIGWINCH ignored. Not daemonized

Clearly this is running in the foreground while debugging things. This behavior is not present if the terminal size remains unchanged. Not a big issue, just an annoyance.

@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc May 18, 2016

Owner

@w-p only these 2 lines once or it appears more often?

Owner

benoitc commented May 18, 2016

@w-p only these 2 lines once or it appears more often?

@w-p

This comment has been minimized.

Show comment
Hide comment
@w-p

w-p May 18, 2016

The lines are repeated for every row / col change (SIGWINCH) to the terminal. In other words, while dragging the resize handle of the terminal window, lines are output. When not resizing, the lines do not continue - but elevated CPU usage does.

w-p commented May 18, 2016

The lines are repeated for every row / col change (SIGWINCH) to the terminal. In other words, while dragging the resize handle of the terminal window, lines are output. When not resizing, the lines do not continue - but elevated CPU usage does.

@tilgovi

This comment has been minimized.

Show comment
Hide comment
@tilgovi

tilgovi May 20, 2016

Collaborator

It would be really interesting to know if there is a commit that introduced that we can find via bisect. It's possible we're not clearing some state due to some kind of race or something.

Collaborator

tilgovi commented May 20, 2016

It would be really interesting to know if there is a commit that introduced that we can find via bisect. It's possible we're not clearing some state due to some kind of race or something.

@vlt

This comment has been minimized.

Show comment
Hide comment
@vlt

vlt May 20, 2016

I experience the same reproducable behaviour.

I’m running Django 1.9.6 with Python 3.4.3 and gunicorn 19.5.0 in a virtualenv behind nginx on Ubuntu 14.04 (DigitalOcean).

As soon as I resize my “screen” terminal (by splitting, the :resize command, or just by re-attaching to the session from another terminal) the workers’s load goes up to 25 %CPU each. Permanently. The system load avg approaches n.00 (n being the number of workers).
After sending SIGHUP (booting new workers) evrything calms down.

vlt commented May 20, 2016

I experience the same reproducable behaviour.

I’m running Django 1.9.6 with Python 3.4.3 and gunicorn 19.5.0 in a virtualenv behind nginx on Ubuntu 14.04 (DigitalOcean).

As soon as I resize my “screen” terminal (by splitting, the :resize command, or just by re-attaching to the session from another terminal) the workers’s load goes up to 25 %CPU each. Permanently. The system load avg approaches n.00 (n being the number of workers).
After sending SIGHUP (booting new workers) evrything calms down.

@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc May 20, 2016

Owner

I can reproduce it as well. Working on it :)

Owner

benoitc commented May 20, 2016

I can reproduce it as well. Working on it :)

@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc May 20, 2016

Owner

@tilgovi so b0c0333 introduced the issue.

When you receive a SIGCHLD, it signal.set_wakeup_fd continuously write on the pipe and always wake up the worker. Which triggers the CPU spike. Maybe an issue in Python 3.5 ? cc @berkerpeksag

Anyway trying to find a proper fix.

Owner

benoitc commented May 20, 2016

@tilgovi so b0c0333 introduced the issue.

When you receive a SIGCHLD, it signal.set_wakeup_fd continuously write on the pipe and always wake up the worker. Which triggers the CPU spike. Maybe an issue in Python 3.5 ? cc @berkerpeksag

Anyway trying to find a proper fix.

@benoitc benoitc closed this in 39f62ac May 20, 2016

@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc May 20, 2016

Owner

@berkerpeksag actually nevermind. My bad I was not reading the signal bit written in the pipe. fixed now.

Owner

benoitc commented May 20, 2016

@berkerpeksag actually nevermind. My bad I was not reading the signal bit written in the pipe. fixed now.

@benoitc benoitc added this to Other in Bugs Feb 23, 2017

@benoitc benoitc removed this from Other in Bugs Feb 23, 2017

fofanov pushed a commit to fofanov/gunicorn that referenced this issue Mar 16, 2018

make sure to remove the signal from the worker pipe
The signal was never removed from the pie which was always waking up the worker
triggering a CPU spike.

fix #1269
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment