Fix delayed socket close caused by circular ref from assigning traceback to local var #1228

Merged
merged 1 commit into from Mar 20, 2016

Projects

None yet

4 participants

@paulnivin
Contributor

As part of pulling in 2e231d2 and a3f6df5, EPIPEs result in the EnvironmentError exception handler in handle() being run. In many environments, EPIPEs can be extremely common. The exception handler stores the traceback into a local variable, creating a circular reference, and prevents the system socket (holding a fd open along with other socket resources) from being closed until the next Python full garbage collection cycle. The behavior of sys.exec_info() is documented at https://docs.python.org/2/library/sys.html#sys.exc_info with a large red warning statement. This new exception path for EPIPEs can create significant additional resource usage for gunicorn installations.

Removing the assignment into exc_info causes the socket fd to be freed quickly (handled by ref counting), as was the case before the two prior patches were merged.

This commit also fixes the ssl.SSLError path.

@tilgovi
Collaborator
tilgovi commented Mar 18, 2016

💯

@paulnivin there are a few other places where this happens. Want to 🔥 them all?

@benoitc
Owner
benoitc commented Mar 18, 2016

+1

@paulnivin
Contributor

I'll fix up the other places as well and update the branch.

@tilgovi
Collaborator
tilgovi commented Mar 18, 2016

@paulnivin did you push?

@paulnivin
Contributor

Not yet. Will shortly. I'm going through the codebase and fixing up all the uses of sys.exec_info(). The async EnvironmentError path was the only one that caused a dramatic change in gunicorn performance/resource utilization for us.

@paulnivin
Contributor

Updated the branch to clean up all the calls to sys.exc_info() to conform to recommended best practices. We're not running 9392e7a in production, but it passes the gunicorn tests. 😑

@berkerpeksag berkerpeksag commented on the diff Mar 19, 2016
gunicorn/workers/base.py
@@ -134,11 +134,14 @@ def load_wsgi(self):
self.log.exception(e)
- exc_type, exc_val, exc_tb = sys.exc_info()
- self.reloader.add_extra_file(exc_val.filename)
-
- tb_string = traceback.format_exc(exc_tb)
- self.wsgi = util.make_fail_app(tb_string)
+ try:
+ exc_type, exc_val, exc_tb = sys.exc_info()
+ self.reloader.add_extra_file(exc_val.filename)
+
+ tb_string = traceback.format_exc(exc_tb)
+ self.wsgi = util.make_fail_app(tb_string)
+ finally:
+ del exc_tb
@berkerpeksag
berkerpeksag Mar 19, 2016 Collaborator

We could use a comment to explain what del exc_tb does here (and perhaps a reference to this PR).

Could you also squash the commits into one?

Thanks!

@paulnivin
paulnivin Mar 20, 2016 Contributor

Added a comment and squashed the commits.

@tilgovi
Collaborator
tilgovi commented Mar 19, 2016

Also, sorry @paulnivin, I misread your comment and thought it said you already had fixed up the other places when I asked if you had pushed. Wasn't trying to be pushy. 😜

@tilgovi
Collaborator
tilgovi commented Mar 20, 2016

Excellent. Thanks, again.

@tilgovi tilgovi merged commit 5b32dde into benoitc:master Mar 20, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment