Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Hook for a setup function when using run_simple and autoreloader #220

Closed
delfick opened this Issue Sep 30, 2012 · 26 comments

Comments

Projects
None yet
3 participants

delfick commented Sep 30, 2012

Hi,

I want to be able to run some arbitrary code everytime the werkzeug reloader decides to reload my application such that if that code fails, all I have to do is change the code and it will auto reload and try it again.

As far as I can tell, the nicest way to achieve this is to run that code before the make_server().serve_forever() call in the inner() function in run_simple.

So, for example, add a setup_func argument to run_simple and call it in the inner() function.

So, in werkzeug/serving.py

def run_simple(hostname, port, application, setup_func=None, [..]):
    [..]

    def inner():
        if setup_func:
            setup_func()
        make_server(hostname, port, application, threaded,
                    processes, request_handler,
                    passthrough_errors, ssl_context).serve_forever()

    [..]
Owner

untitaker commented Dec 5, 2012

Werkzeug's reloading is happening when you modify any source files. From what i understood, you're trying to achieve exactly that.

Moreover, you can run arbitrary python code (such as print statements) in your app code, they'll get executed everytime your application gets loaded. Consider this code:

from werkzeug.wrappers import Request, Response

@Request.application
def application(request):
    return Response('Hello World!')

print "RELOAD"

from werkzeug.serving import run_simple
run_simple('localhost', 4000, application, use_reloader=True)

It would produce this output if i touch the source file multiple times:

RELOAD
 * Running on http://localhost:4000/
 * Restarting with reloader
RELOAD
 * Detected change in 'werkzeug_reloader.py', reloading
 * Restarting with reloader
RELOAD
 * Detected change in 'werkzeug_reloader.py', reloading
 * Restarting with reloader
RELOAD

Where's the +1 button?

delfick commented Dec 5, 2012

@unitaker so that is true and that part works wonderfully.

The problem is that this arbitrary code I want to run isn't inside the application callable and to make it part of the application callable would mean it gets called for every request and this wouldn't be good at all.

But it still needs to be ran when the reload happens.

@delfick, it will not be called every request; the application is loaded once and processes as many requests as it can before it is recycled (crash, reload, etc.); you can even put the tasks to run with the .run part of the application, which will only occur if run in standalone mode, rather than being run by a container (uWSGI, mod_wsgi, etc.):

import flask

app = flask.Flask( __name__ )

if __name__ == '__main__':
    print 'Do something...'
    app.run( use_reloader=True )

delfick commented Dec 6, 2012

I mean the application object that you pass into run_simple. That one does get called every request.

Call the callable from outside, what does the callable look like?

delfick commented Dec 6, 2012

So I'm using it something like this:

from django.core.handlers.wsgi import WSGIHandler
from werkzeug import run_simple
import os

os.environ['DJANGO_SETTINGS_MODULE'] = 'path.to_django_project.settings'
app = WSGIHandler()

run_simple(self.host, self.port, app
    , use_debugger=True, use_reloader=True
    )

If I replace app with something like:

def wrapped(*args, **kwargs):
    print 'one'
    return app(*args, **kwargs)

Then it will print 'one' for every request.

I think your run_simple should wrapped in an if __name__ == '__main__': condition for the requests not to relaunch the server.

delfick commented Dec 6, 2012

The script I'm using already does.

The run_simple only get's called once.

run_simple will create that "inner" function and pass it to use_with_reloader.
use_with_reloader will then call inner everytime it decides to reload everything.
The inner function uses make_server to pass the application object to either ThreadedWSGIServer, ForkingWSGIServer or BaseWSGIServer.
All of those will call the application object for each request.

Hence my change just adds a bit of code before it calls make_server, so I know it only gets called when the reload happens.

delfick commented Dec 6, 2012

actually, it seems my script is running run_simple everytime it reloads.....
Though the application object does get called for every request
edit : and the setup_func function in my pull request only gets called once per reload

delfick commented Dec 6, 2012

It seems even if I execute my script directly, using the if __name__ == "__main__" convention instead of via a pip installed entry_point, it also reloads the entire script.

EDIT : I also don't want to have my setup function running outside of run_simple in case it fails before the reloader starts and hence means I have to manually restart the server, thus defeating the point of the reloader....

Owner

untitaker commented Dec 8, 2012

arbitrary code I want to run isn't inside the application callable

The print statements in my example are neither. I don't get what you want to say and why the code i provided doesn't do the trick for you.

delfick commented Dec 8, 2012

Ok, so that code does work.
Judging by my comment afterwards, maybe I thought I saw the "print 'Reload'" inside the application function.

The problem becomes is that it happens outside the reload.
So if it raises some Exception, the server dies and I have to manually restart it.

So say you have two files: a.py and b.py that look like:

# a.py
from werkzeug.wrappers import Request, Response

@Request.application
def application(request):
    return Response('Hello World!')

# Do our arbritrary setup
from b import arbitrary_setup
arbitrary_setup()

from werkzeug.serving import run_simple
run_simple('localhost', 4000, application, use_reloader=True)

and

#b.py
def arbitrary_setup():
    """Setup things here"""
    print "I am a banana!"

And start the server:

┌─(iambob@iambob-computer, 09 Dec)(example)────(~/Projects/play/setup)─────┐
└─(9:49)Ɣ python a.py
I am a banana!
 * Running on http://localhost:4000/
 * Restarting with reloader
I am a banana!
 * Detected change in 'a.py', reloading
 * Restarting with reloader
I am a banana!

Now edit b.py to look like:

def arbitrary_setup():
    """Setup things here"""
    raise Exception("Monkey ate the banana")

Terminal looks like this

┌─(iambob@iambob-computer, 09 Dec)(example)────(~/Projects/play/setup)─────┐
└─(9:51)Ɣ python a.py
I am a banana!
 * Running on http://localhost:4000/
 * Restarting with reloader
I am a banana!
 * Detected change in 'a.py', reloading
 * Restarting with reloader
I am a banana!
 * Detected change in '/home/iambob/Projects/play/setup/b.py', reloading
 * Restarting with reloader
Traceback (most recent call last):
  File "a.py", line 9, in <module>
    arbitrary_setup()
  File "/home/iambob/Projects/play/setup/b.py", line 3, in arbitrary_setup
    raise Exception("Monkey ate the banana")
Exception: Monkey ate the banana

┌─(iambob@iambob-computer, 09 Dec)(example)────(~/Projects/play/setup)─────┐
└─(9:52)Ɣ 

Note how I now have to restart it again because I broke the setup code.

But, if we edit the run_simple function to take in a setup_func that gets executed inside inner, making a.py look like:

# a.py
from werkzeug.wrappers import Request, Response

@Request.application
def application(request):
    return Response('Hello World!')

# Do our arbritrary setup
def setup_things():
    from b import arbitrary_setup
    arbitrary_setup()

from werkzeug.serving import run_simple
run_simple('localhost', 4000, application, use_reloader=True, setup_func=setup_things)

then the server doesn't crash and I can remove the bug and it looks more like this:

┌─(iambob@iambob-computer, 09 Dec)(example)────(~/Projects/play/setup)─────┐
└─(9:57)Ɣ python a.py
 * Running on http://localhost:4000/
 * Restarting with reloader
Unhandled exception in thread started by <function inner at 0x223b8c0>
Traceback (most recent call last):
  File "/home/iambob/.venv/example/local/lib/python2.7/site-packages/werkzeug/serving.py", line 597, in inner
    setup_func()
  File "a.py", line 10, in setup_things
    arbitrary_setup()
  File "/home/iambob/Projects/play/setup/b.py", line 3, in arbitrary_setup
    raise Exception("Monkey ate the banana")
Exception: Monkey ate the banana
 * Detected change in '/home/iambob/Projects/play/setup/b.py', reloading
 * Restarting with reloader
I am a capsicum
 * Detected change in '/home/iambob/Projects/play/setup/b.py', reloading
 * Restarting with reloader
I am a capsicum
Owner

untitaker commented Dec 9, 2012

So you want exceptions not to hinder the server from starting or reloading. Neither the changes you suggested at the very top does that, nor does my suggestion.

I don't know what the big deal is with your problem. If you really want the behavior that exceptions during the initialization process don't stop the server from starting/reloading (which seems unreasonable to me), you still can add a try-except block around your code. A callback function is unnecessary here and can be avoided, and because of that it should be avoided.

Again, here is the code that does exactly what you want, without any modifications of Werkzeug:

# a.py
from werkzeug.wrappers import Request, Response

@Request.application
def application(request):
    return Response('Hello World!')

# Do our arbritrary setup
from b import arbitrary_setup
try:
    arbitrary_setup()
except Exception, e:
    # duh, somehow print the traceback, who knows

from werkzeug.serving import run_simple
run_simple('localhost', 4000, application, use_reloader=True)

delfick commented Dec 9, 2012

From my point of view it makes it just work and is a whole lot cleaner if I can just pass in a setup function.

It also means that the setup only runs once when I start the development server (without knowing about the WERKZEUG_RUN_MAIN environment variable that get's set)

And it also means it won't start the application itself if the setup doesn't work.
It seems run_simple still has to be ran when WERKZEUG_RUN_MAIN is 'true' otherwise it doesn't see files reload.

As for whether never crashing is acceptable: I only use the reloader during development for the explicit purpose of not having to go back to the terminal for every change, which includes fixing some silly SyntaxError I may have made to the stuff that I run before the application starts. (in production I don't use a reloader)

So you want exceptions not to hinder the server from starting or reloading. Neither the changes you suggested at the very top does that, nor does my suggestion.

??

Owner

untitaker commented Dec 10, 2012

...so do you want the server to start if the setup fails or not?

??

You posted this code at the top:

def run_simple(hostname, port, application, setup_func=None, [..]):
    [..]

    def inner():
        if setup_func:
            setup_func()
        make_server(hostname, port, application, threaded,
                    processes, request_handler,
                    passthrough_errors, ssl_context).serve_forever()

    [..]

If you raise an exception in your setup_func, your suggestion will not catch the exception. But you then said that you want the exception to be catched.

delfick commented Dec 10, 2012

I don't want it to reach the part where it's actually serving http if the setup fails.
But I don't want the application in the cli to end.

With my change, any exception raised by setup_func will get caught by the fact that it gets into main_func and hence run inside thread.start_new_thread
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/serving.py#L593

According to this http://docs.python.org/2/library/thread.html?highlight=start_new_thread#thread.start_new_thread
any exception is printed and execution will continue from where start_new_thread was called.

This means that the reloader_loop will then be called
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/serving.py#L595

which will keep looking at the files until one of them changes, where it will return to run_with_reloader so that restart_with_reloader can be called so the whole process may start again
https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/serving.py#L599

The key part here is that when it fails I don't have to go into my terminal and manually restart everything.
(and as a bonus, it doesn't reach the WSGIServer part if the setup fails)

Owner

untitaker commented Jan 24, 2013

Isn't this related to #14?

delfick commented Jan 25, 2013

Almost, but mine get's run before every start, including the first start.
And also works if you don't have the restarter.

@delfick delfick added a commit to delfick/cwf that referenced this issue Jan 20, 2014

@delfick delfick Make cwf-debugger tell you about pallets/werkzeug#220 dc71014
Owner

untitaker commented Aug 24, 2014

Closing this because of staleness and because the functionality can be achieved by simply running code before starting the server.

@untitaker untitaker closed this Aug 24, 2014

@untitaker untitaker referenced this issue Aug 24, 2014

Closed

Reload handler #14

delfick commented Aug 24, 2014

The code I want to run before the server starts does some black magic that is necessary before my application is importable.

So it must be run in the same context as the application itself each time it's restarted.

Which is why it must be in the inner function.

Owner

untitaker commented Aug 25, 2014

The code I want to run before the server starts does some black magic that is necessary before my application is importable.

  1. Do your magic
  2. Import the application
  3. Run server

delfick commented Aug 25, 2014

That's fine but if I then change something in my application that the magic looks at, then I have to manually stop the server and restart at the cli, rather it "just working".

Owner

untitaker commented Aug 25, 2014

It is "just working". I told you multiple times above that the whole Python interpreter gets reloaded, which means your magic will also get reloaded.

delfick commented Aug 25, 2014

I still get this problem mitsuhiko#220 (comment)

Owner

untitaker commented Aug 25, 2014

And the only good solution i can offer is to wrap a try...except block around it. You can't catch SyntaxErrors this way, but a simple callback couldn't either. This is what flask.cli is trying to solve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment