Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added graceful shutdown on SIGHUP #48

Closed
wants to merge 6 commits into from

Conversation

pomarec
Copy link

@pomarec pomarec commented Nov 18, 2013

No description provided.

@mcdonc
Copy link
Member

mcdonc commented Nov 18, 2013

Thanks for the pull request! I think the functions you added need some unit tests, however. Would it be possible for you to extend the test suite to include tests for them?

@pomarec
Copy link
Author

pomarec commented Nov 18, 2013

Yes I'll try to provide it.

I have a question concerning logging:
I would like to add self.logger.info('Graceful shutdown') but it does not show up when running with https://github.com/mozilla-services/chaussette. I understand the fact that by default, waitress logger has its log level set to warning. I'd like to have waitress logger somehow inherit from the chaussette's one ? (thus inherit its level)
Do you have any lead ? (I not that familiar with logging so I'm probably completely wrong)

@pomarec
Copy link
Author

pomarec commented Nov 18, 2013

I think my log issue will be solved on the "chaussette side", I will send them a pull request so the chaussette logger configuration propagates to waitress.

@pomarec
Copy link
Author

pomarec commented Dec 16, 2013

Do the test I provided are good enough ?

@pomarec pomarec mentioned this pull request Jan 2, 2014
@@ -229,10 +229,12 @@ def close_on_finish():
response_headers.append(('Connection', 'Keep-Alive'))
else:
close_on_finish()
self.channel.is_keepalive = True
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand the placement of this. It seems that the code will consider any HTTP/1.0 request a keepalive request as the result of this line? Should it not be moved to after line 229?

@mcdonc
Copy link
Member

mcdonc commented Jul 14, 2014

When merging this patch into the master, I get a number of test failures when running tests under any Python version. The output of Python 2.6 is provided below:

[chrism@thinko waitress]$ tox
GLOB sdist-make: /home/chrism/projects/waitress/setup.py
py26 inst-nodeps: /home/chrism/projects/waitress/.tox/dist/waitress-0.8.10dev.zip
py26 runtests: PYTHONHASHSEED='1004523404'
py26 runtests: commands[0] | python setup.py dev
running develop
running egg_info
writing dependency_links to waitress.egg-info/dependency_links.txt
writing waitress.egg-info/PKG-INFO
writing top-level names to waitress.egg-info/top_level.txt
writing requirements to waitress.egg-info/requires.txt
writing entry points to waitress.egg-info/entry_points.txt
reading manifest file 'waitress.egg-info/SOURCES.txt'
writing manifest file 'waitress.egg-info/SOURCES.txt'
running build_ext
Creating /home/chrism/projects/waitress/.tox/py26/lib/python2.6/site-packages/waitress.egg-link (link to .)
Adding waitress 0.8.10dev to easy-install.pth file
Installing waitress-serve script to /home/chrism/projects/waitress/.tox/py26/bin

Installed /home/chrism/projects/waitress
Processing dependencies for waitress==0.8.10dev
Searching for setuptools==3.6
Best match: setuptools 3.6
Adding setuptools 3.6 to easy-install.pth file
Installing easy_install-3.4 script to /home/chrism/projects/waitress/.tox/py26/bin
Installing easy_install script to /home/chrism/projects/waitress/.tox/py26/bin

Using /home/chrism/projects/waitress/.tox/py26/lib/python2.6/site-packages
Finished processing dependencies for waitress==0.8.10dev
running easy_install
Searching for waitress[testing]
Best match: waitress 0.8.10dev
Removing waitress 0.8.10dev from easy-install.pth file
Adding waitress 0.8.10dev to easy-install.pth file
Installing waitress-serve script to /home/chrism/projects/waitress/.tox/py26/bin

Using /home/chrism/projects/waitress/.tox/py26/lib/python2.6/site-packages
Processing dependencies for waitress[testing]
Finished processing dependencies for waitress[testing]
py26 runtests: commands[1] | python setup.py nosetests --processes=4
running nosetests
running egg_info
writing dependency_links to waitress.egg-info/dependency_links.txt
writing waitress.egg-info/PKG-INFO
writing top-level names to waitress.egg-info/top_level.txt
writing requirements to waitress.egg-info/requires.txt
writing entry points to waitress.egg-info/entry_points.txt
reading manifest file 'waitress.egg-info/SOURCES.txt'
writing manifest file 'waitress.egg-info/SOURCES.txt'
/home/chrism/projects/waitress/.tox/py26/lib/python2.6/site-packages/nose-1.3.3-py2.6.egg/nose/config.py:435: DeprecationWarning: Use of multiple -w arguments is deprecated and support may be removed in a future release. You can get the same behavior by passing directories without the -w argument on the command line, or by using the --tests argument in a configuration file.
  DeprecationWarning)
..................................................................................EE.EEEEE.EEEE.......................................................................................................................................................................................................................................E.....................................................................................................................E
======================================================================
ERROR: test_handle_close (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 293, in test_handle_close
    inst.handle_close()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 293, in handle_close
    self.server.on_channel_close(self)
AttributeError: 'DummyServer' object has no attribute 'on_channel_close'

======================================================================
ERROR: test_handle_read_error (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 196, in test_handle_read_error
    result = inst.handle_read()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 175, in handle_read
    self.handle_close()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 293, in handle_close
    self.server.on_channel_close(self)
AttributeError: 'DummyServer' object has no attribute 'on_channel_close'

======================================================================
ERROR: test_handle_write_close_when_flushed (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 151, in test_handle_write_close_when_flushed
    result = inst.handle_write()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 156, in handle_write
    self.check_shutdown_gracefully()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 280, in check_shutdown_gracefully
    if self.server.shutdown_gracefully and self.is_keepalive and \
AttributeError: 'DummyServer' object has no attribute 'shutdown_gracefully'

======================================================================
ERROR: test_handle_write_no_requests_force_flush (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 124, in test_handle_write_no_requests_force_flush
    result = inst.handle_write()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 156, in handle_write
    self.check_shutdown_gracefully()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 280, in check_shutdown_gracefully
    if self.server.shutdown_gracefully and self.is_keepalive and \
AttributeError: 'DummyServer' object has no attribute 'shutdown_gracefully'

======================================================================
ERROR: test_handle_write_no_request_with_outbuf (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 69, in test_handle_write_no_request_with_outbuf
    result = inst.handle_write()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 156, in handle_write
    self.check_shutdown_gracefully()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 280, in check_shutdown_gracefully
    if self.server.shutdown_gracefully and self.is_keepalive and \
AttributeError: 'DummyServer' object has no attribute 'shutdown_gracefully'

======================================================================
ERROR: test_handle_write_no_requests_no_outbuf_will_close (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 110, in test_handle_write_no_requests_no_outbuf_will_close
    result = inst.handle_write()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 156, in handle_write
    self.check_shutdown_gracefully()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 280, in check_shutdown_gracefully
    if self.server.shutdown_gracefully and self.is_keepalive and \
AttributeError: 'DummyServer' object has no attribute 'shutdown_gracefully'

======================================================================
ERROR: test_handle_close_outbuf_raises_on_close (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 303, in test_handle_close_outbuf_raises_on_close
    inst.handle_close()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 293, in handle_close
    self.server.on_channel_close(self)
AttributeError: 'DummyServer' object has no attribute 'on_channel_close'

======================================================================
ERROR: test_handle_write_outbuf_raises_othererror (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 96, in test_handle_write_outbuf_raises_othererror
    result = inst.handle_write()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 156, in handle_write
    self.check_shutdown_gracefully()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 280, in check_shutdown_gracefully
    if self.server.shutdown_gracefully and self.is_keepalive and \
AttributeError: 'DummyServer' object has no attribute 'shutdown_gracefully'

======================================================================
ERROR: test_handle_write_outbuf_raises_socketerror (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 82, in test_handle_write_outbuf_raises_socketerror
    result = inst.handle_write()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 156, in handle_write
    self.check_shutdown_gracefully()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 280, in check_shutdown_gracefully
    if self.server.shutdown_gracefully and self.is_keepalive and \
AttributeError: 'DummyServer' object has no attribute 'shutdown_gracefully'

======================================================================
ERROR: test_handle_write_with_requests (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 60, in test_handle_write_with_requests
    result = inst.handle_write()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 156, in handle_write
    self.check_shutdown_gracefully()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 280, in check_shutdown_gracefully
    if self.server.shutdown_gracefully and self.is_keepalive and \
AttributeError: 'DummyServer' object has no attribute 'shutdown_gracefully'

======================================================================
ERROR: test_handle_write_no_requests_outbuf_gt_send_bytes (waitress.tests.test_channel.TestHTTPChannel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_channel.py", line 138, in test_handle_write_no_requests_outbuf_gt_send_bytes
    result = inst.handle_write()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 156, in handle_write
    self.check_shutdown_gracefully()
  File "/home/chrism/projects/waitress/waitress/channel.py", line 280, in check_shutdown_gracefully
    if self.server.shutdown_gracefully and self.is_keepalive and \
AttributeError: 'DummyServer' object has no attribute 'shutdown_gracefully'

======================================================================
ERROR: test_exits_on_sighup (waitress.tests.test_server.TestWSGIServer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_server.py", line 200, in test_exits_on_sighup
    with self.assertRaises(asyncore.ExitNow):
TypeError: failUnlessRaises() takes at least 3 arguments (2 given)

======================================================================
ERROR: test_it (waitress.tests.test_functional.GracefulShutdownThreadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/waitress/waitress/tests/test_functional.py", line 162, in test_it
    self.assertIsNotNone(self.proc.exitcode)
AttributeError: 'GracefulShutdownThreadTests' object has no attribute 'assertIsNotNone'

----------------------------------------------------------------------
Ran 445 tests in 3.429s

FAILED (errors=13)

@mcdonc
Copy link
Member

mcdonc commented Jul 14, 2014

Note also that running the tests via tox should produce 100% test coverage when the "coverage" part is run. I doubt it will right now (can't tell until the tests are fixed).

@@ -281,6 +290,7 @@ def handle_close(self):
'Unknown exception while trying to close outbuf')
self.connected = False
asyncore.dispatcher.close(self)
self.server.on_channel_close(self)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This indirection is probably unnecessary if on_channel_close doesn't use the channel, and all it does is call check_shutdown_gracefully. It should probably just call check_shutdown_gracefully instead.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, during development it was calling other things. I'll clean it.

@mcdonc
Copy link
Member

mcdonc commented Jul 14, 2014

Note also that the TaskDispatcher.shutdown method already attempts to wait 5 secs for all task threads to finish running; would the application of #56 be enough to prevent the need for this patch?

@pomarec
Copy link
Author

pomarec commented Jul 16, 2014

I'll try to work on this this weekend.

@mcdonc
Copy link
Member

mcdonc commented Jul 16, 2014

Before you spend a lot of time on improving this patch, please do look at #56 and see if there's a way to make the default shutdown "more graceful" as opposed to adding code for graceful shutdown as per this pull request.

@pomarec
Copy link
Author

pomarec commented Aug 25, 2014

#56 only enables the threads death timeout (which already exists) to be configurable. But it only deals with the tasks, not the channels, so we can't be sure that all the HTTP requests are fulfilled.
This PR (#48) is probably not the best way to close the server after having fulfilled all the pending requests but i works. It also provides a way to close the server from "outside" (signal).

@pomarec pomarec closed this Aug 25, 2014
@pomarec pomarec reopened this Aug 25, 2014
@earonesty
Copy link

earonesty commented Dec 1, 2017

What if waitress is running in something other than the main thread... at that point sending a signal won't help, since the main thread would get it. Would be nice if you could simply stop the loop.

Something like this would be useful... and would be an interface people are used to.

server = server(run=False)
Thread(server.serve_forever())
....stuff....
server.shutdown()

Right now the assumption is that waitress is always the only thing running in the process. If the ability to shutdown was added, the user could create their own signal handlers...depending on the vagaries of their current env. This way the ipc mechanism (signals) is not defined by waitress

(Also it simplifies the change to waitress)

@digitalresistor
Copy link
Member

digitalresistor commented Dec 1, 2017

@earonesty
Copy link

earonesty commented Dec 4, 2017

(Neither of those modules are quite suitable for a production server. )

If this patch were merged, I could easily use it by updating the "shutdown_gracefully" flag myself, but maybe better to formalize that (see comment).

@@ -77,6 +80,9 @@ def __init__(self,
self.effective_host, self.effective_port = self.getsockname()
self.server_name = self.get_server_name(self.adj.host)
self.active_channels = {}
self.shutdown_gracefully = False

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make this __shutdown_gracefully, so users don't have direct access to this mechanism, but then add a shutdown() method that sets it, so users can install their own signal handling if needed.

@digitalresistor
Copy link
Member

For me to accept this patch:

  1. All of the review comments left above by @mcdonc would need to be resolved first
  2. Master needs to be merged into it, and any changes necessary need to be added. Since waitress now supports running a single instance of waitress on multiple ports/interfaces, I have a feeling more work needs to be done to add the shutdown gracefully.
  3. Tests need to be added, and coverage needs to be 100%
  4. I'd like to see a shutdown timeout that forces connections to be torn down, shutting down gracefully shouldn't be something that can take hours if there is a long-running connection for example.

I personally haven't had a use case for this yet, running waitress behind a load balancer I simply remove it from the load balancer, once the load balancer notifies of all open requests being complete, I can shut down waitress.

If you are willing to work on this @earonesty, feel free to do so by pulling this PR down, and creating a new PR based upon this. (However, @pomarec would need to sign https://github.com/Pylons/waitress/blob/master/CONTRIBUTORS.txt) or you may create a new patch based upon master instead.

@digitalresistor
Copy link
Member

(Neither of those modules are quite suitable for a production server. )

They are using waitress underneath... you may of course take that source code and use them as a starting point, if you wish.

@pomarec
Copy link
Author

pomarec commented Dec 5, 2017

FYI i don't have time anymore to work on this. I'm sorry.
Cheers

@mmerickel
Copy link
Member

@pomarec can you push your name to CONTRIBUTORS.txt incase anyone wants to fix up this PR otherwise we need to close it

@digitalresistor
Copy link
Member

I am closing this PR as original submitter has not signed contributors.txt.

@renaudl
Copy link

renaudl commented Oct 16, 2019

A use case for this would be to ensure all pending requests are competed before shutdown when used with a hot reloader like hupper.

@unittolabs
Copy link

Any new updates here or future plans to implement this?

@digitalresistor
Copy link
Member

@unitto1 the following comment describes what needs to be done: #269 (comment)

Are there future plans, sure, but I only have so much time and this is not a high priority item for me.

@unittolabs
Copy link

@bertjwregeer clear, ty!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants