Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add --reload option for code reloading #682

Merged
merged 2 commits into from

3 participants

@tilgovi
Collaborator

Fix #526

@tilgovi
Collaborator

Welcome questions / comments / changes.

@tilgovi
Collaborator

Wasn't sure whether to do the callback or pass the log in order to log the message. Also willing to consider a server hook for code change as the callback instead, or as another feature.

@tilgovi
Collaborator

Also, I don't know if the NOTICE change is necessary. I was the principal author of greins, but I didn't write the reloader first (even though I've modified it here, and it's simple, and similar to common code in many places and examples for python module watching). Anyway, it wasn't (c) me, so I included it.

@tilgovi
Collaborator

Still needed:

  • test interaction with paster, which has its own --reload
  • see if i can easily add the config file to the watched files
@tilgovi
Collaborator

Stand by... I will move the reloader support to the base application, and hook in at load_config_from_file

@benoitc
Owner

@tilgovi patch looks good for me. We should indeed see how to handle it with paster. Maybe there is a way to disable paster?

About the config file, what do you mean ? The gunicorn config file? Shouldn't we let this one out of the reloading? Imo HUP is enough for such purpose.

@tilgovi tilgovi merged commit d4f2481 into from
@tilgovi tilgovi deleted the branch
@georgexsh georgexsh commented on the diff
gunicorn/reloader.py
((12 lines not shown))
+
+
+class Reloader(threading.Thread):
+ def __init__(self, extra_files=None, interval=1, callback=None):
+ super(Reloader, self).__init__()
+ self.setDaemon(True)
+ self._extra_files = set(extra_files or ())
+ self._extra_files_lock = threading.RLock()
+ self._interval = interval
+ self._callback = callback
+
+ def add_extra_file(self, filename):
+ with self._extra_files_lock:
+ self._extra_files.add(filename)
+
+ def get_files(self):

This default reloader implementation seems not appeal to me

  • could not deal add new file
  • scan every source file in sys.modules is obvious unnecessary, and there is no guarantee all app modules would be loaded when fire Reloader.get_files, so the reloader scanned extra files but missed some
  • each worker will do file scan and reload(kill), it is a overhead
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@tilgovi
Collaborator

@georgexsh I welcome improvements.

You can add new files. Maybe you mean that the reloader instance is never stored so add_extra_file cannot be called.

I don't see a better way than scanning everything in sys.modules. I can't see why it would be important to limit it to a specific package. I don't know what you mean by "reloader scanned extra files but missed some". Can you say more?

I'm not at all bothered by all workers checking. It's overhead, but this is for development, not production.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 16, 2014
  1. @tilgovi
Commits on Jan 29, 2014
  1. @tilgovi
This page is out of date. Refresh to see the latest.
View
28 NOTICE
@@ -54,6 +54,34 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+gunicorn.reloader
+-----------------
+
+Based on greins.reloader module under MIT license:
+
+2010 (c) Meebo, Inc.
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
doc/sitemap_gen.py
------------------
Under BSD License :
View
35 examples/example_gevent_reloader.py
@@ -1,35 +0,0 @@
-import logging
-import os
-import signal
-import sys
-
-def on_starting(server):
- # use server hook to patch socket to allow worker reloading
- from gevent import monkey
- monkey.patch_socket()
-
-def when_ready(server):
- def monitor():
- modify_times = {}
- while True:
- for module in sys.modules.values():
- path = getattr(module, "__file__", None)
- if not path: continue
- if path.endswith(".pyc") or path.endswith(".pyo"):
- path = path[:-1]
- try:
- modified = os.stat(path).st_mtime
- except:
- continue
- if path not in modify_times:
- modify_times[path] = modified
- continue
- if modify_times[path] != modified:
- logging.info("%s modified; restarting server", path)
- os.kill(os.getpid(), signal.SIGHUP)
- modify_times = {}
- break
- gevent.sleep(1)
-
- import gevent
- gevent.spawn(monitor)
View
19 gunicorn/config.py
@@ -706,6 +706,25 @@ class Debug(Setting):
"""
+class Reload(Setting):
+ name = "reload"
+ section = 'Debugging'
+ cli = ['--reload']
+ validator = validate_bool
+ action = 'store_true'
+ default = False
+ desc = '''\
+ Restart workers when code changes.
+
+ This setting is intended for development. It will cause workers to be
+ restarted whenever application code changes.
+
+ The reloader is incompatible with application preloading. When using a
+ paste configuration be sure that the server block does not import any
+ application code or the reload will not work as designed.
+ '''
+
+
class Spew(Setting):
name = "spew"
section = "Debugging"
View
54 gunicorn/reloader.py
@@ -0,0 +1,54 @@
+# -*- coding: utf-8 -
+#
+# This file is part of gunicorn released under the MIT license.
+# See the NOTICE for more information.
+
+import os
+import re
+import signal
+import sys
+import time
+import threading
+
+
+class Reloader(threading.Thread):
+ def __init__(self, extra_files=None, interval=1, callback=None):
+ super(Reloader, self).__init__()
+ self.setDaemon(True)
+ self._extra_files = set(extra_files or ())
+ self._extra_files_lock = threading.RLock()
+ self._interval = interval
+ self._callback = callback
+
+ def add_extra_file(self, filename):
+ with self._extra_files_lock:
+ self._extra_files.add(filename)
+
+ def get_files(self):

This default reloader implementation seems not appeal to me

  • could not deal add new file
  • scan every source file in sys.modules is obvious unnecessary, and there is no guarantee all app modules would be loaded when fire Reloader.get_files, so the reloader scanned extra files but missed some
  • each worker will do file scan and reload(kill), it is a overhead
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ fnames = [
+ re.sub('py[co]$', 'py', module.__file__)
+ for module in sys.modules.values()
+ if hasattr(module, '__file__')
+ ]
+
+ with self._extra_files_lock:
+ fnames.extend(self._extra_files)
+
+ return fnames
+
+ def run(self):
+ mtimes = {}
+ while True:
+ for filename in self.get_files():
+ try:
+ mtime = os.stat(filename).st_mtime
+ except OSError:
+ continue
+ old_time = mtimes.get(filename)
+ if old_time is None:
+ mtimes[filename] = mtime
+ continue
+ elif mtime > old_time:
+ if self._callback:
+ self._callback(filename)
+ time.sleep(self._interval)
View
9 gunicorn/workers/base.py
@@ -12,6 +12,7 @@
from gunicorn import util
from gunicorn.workers.workertmp import WorkerTmp
+from gunicorn.reloader import Reloader
from gunicorn.http.errors import InvalidHeader, InvalidHeaderName, \
InvalidRequestLine, InvalidRequestMethod, InvalidHTTPVersion, \
LimitRequestLine, LimitRequestHeaders
@@ -79,6 +80,14 @@ def init_process(self):
loop is initiated.
"""
+ # start the reloader
+ if self.cfg.reload:
+ def changed(fname):
+ self.log.info("Worker reloading: %s modified", fname)
+ os.kill(self.pid, signal.SIGTERM)
+ raise SystemExit()
+ Reloader(callback=changed).start()
+
# set enviroment' variables
if self.cfg.env:
for k, v in self.cfg.env.items():
Something went wrong with that request. Please try again.