Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 433 lines (341 sloc) 14.588 kb
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
1 """``tornado.gen`` is a generator-based interface to make it easier to
2 work in an asynchronous environment. Code using the ``gen`` module
3 is technically asynchronous, but it is written as a single generator
4 instead of a collection of separate functions.
5
6 For example, the following asynchronous handler::
7
8 class AsyncHandler(RequestHandler):
9 @asynchronous
10 def get(self):
11 http_client = AsyncHTTPClient()
12 http_client.fetch("http://example.com",
13 callback=self.on_fetch)
14
15 def on_fetch(self, response):
16 do_something_with_response(response)
17 self.render("template.html")
18
19 could be written with ``gen`` as::
20
21 class GenAsyncHandler(RequestHandler):
22 @asynchronous
23 @gen.engine
24 def get(self):
25 http_client = AsyncHTTPClient()
93a0dc8 @bdarnell Fix error in docs
bdarnell authored
26 response = yield gen.Task(http_client.fetch, "http://example.com")
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
27 do_something_with_response(response)
28 self.render("template.html")
29
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
30 `Task` works with any function that takes a ``callback`` keyword
31 argument. You can also yield a list of ``Tasks``, which will be
32 started at the same time and run in parallel; a list of results will
33 be returned when they are all finished::
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
34
35 def get(self):
36 http_client = AsyncHTTPClient()
37 response1, response2 = yield [gen.Task(http_client.fetch, url1),
38 gen.Task(http_client.fetch, url2)]
39
40 For more complicated interfaces, `Task` can be split into two parts:
41 `Callback` and `Wait`::
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
42
43 class GenAsyncHandler2(RequestHandler):
44 @asynchronous
45 @gen.engine
46 def get(self):
47 http_client = AsyncHTTPClient()
48 http_client.fetch("http://example.com",
49 callback=(yield gen.Callback("key"))
50 response = yield gen.Wait("key")
51 do_something_with_response(response)
52 self.render("template.html")
53
54 The ``key`` argument to `Callback` and `Wait` allows for multiple
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
55 asynchronous operations to be started at different times and proceed
56 in parallel: yield several callbacks with different keys, then wait
57 for them once all the async operations have started.
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
58
59 The result of a `Wait` or `Task` yield expression depends on how the callback
60 was run. If it was called with no arguments, the result is ``None``. If
61 it was called with one argument, the result is that argument. If it was
62 called with more than one argument or any keyword arguments, the result
63 is an `Arguments` object, which is a named tuple ``(args, kwargs)``.
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
64 """
58a7ff1 @bdarnell Turn on __future__ division too.
bdarnell authored
65 from __future__ import absolute_import, division, with_statement
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
66
67 import functools
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
68 import operator
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
69 import sys
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
70 import types
71
0cc0df7 @bdarnell Add magic for yielding futures.
bdarnell authored
72 from tornado.concurrent import Future
5587a45 @bdarnell Add gen.YieldFuture.
bdarnell authored
73 from tornado.ioloop import IOLoop
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous function...
bdarnell authored
74 from tornado.stack_context import ExceptionStackContext
75
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
76
77 class KeyReuseError(Exception):
78 pass
79
80
81 class UnknownKeyError(Exception):
82 pass
83
84
85 class LeakedCallbackError(Exception):
86 pass
87
88
89 class BadYieldError(Exception):
90 pass
91
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
92
93 def engine(func):
94 """Decorator for asynchronous generators.
95
96 Any generator that yields objects from this module must be wrapped
97 in this decorator. The decorator only works on functions that are
98 already asynchronous. For `~tornado.web.RequestHandler`
7fd1788 @bdarnell Fix gen.engine docs on decorator order.
bdarnell authored
99 ``get``/``post``/etc methods, this means that both the
100 `tornado.web.asynchronous` and `tornado.gen.engine` decorators
101 must be used (for proper exception handling, ``asynchronous``
102 should come before ``gen.engine``). In most other cases, it means
103 that it doesn't make sense to use ``gen.engine`` on functions that
104 don't already take a callback argument.
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
105 """
106 @functools.wraps(func)
107 def wrapper(*args, **kwargs):
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous function...
bdarnell authored
108 runner = None
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
109
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous function...
bdarnell authored
110 def handle_exception(typ, value, tb):
111 # if the function throws an exception before its first "yield"
112 # (or is not a generator at all), the Runner won't exist yet.
113 # However, in that case we haven't reached anything asynchronous
114 # yet, so we can just let the exception propagate.
115 if runner is not None:
116 return runner.handle_exception(typ, value, tb)
117 return False
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
118 with ExceptionStackContext(handle_exception) as deactivate:
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous function...
bdarnell authored
119 gen = func(*args, **kwargs)
120 if isinstance(gen, types.GeneratorType):
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
121 runner = Runner(gen, deactivate)
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous function...
bdarnell authored
122 runner.run()
123 return
124 assert gen is None, gen
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
125 deactivate()
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous function...
bdarnell authored
126 # no yield, so we're done
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
127 return wrapper
128
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
129
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
130 class YieldPoint(object):
131 """Base class for objects that may be yielded from the generator."""
132 def start(self, runner):
133 """Called by the runner after the generator has yielded.
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
134
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
135 No other methods will be called on this object before ``start``.
136 """
137 raise NotImplementedError()
138
139 def is_ready(self):
140 """Called by the runner to determine whether to resume the generator.
141
2f91460 @bdarnell Allow any sequence in gen.WaitAll, not just lists.
bdarnell authored
142 Returns a boolean; may be called more than once.
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
143 """
144 raise NotImplementedError()
145
146 def get_result(self):
147 """Returns the value to use as the result of the yield expression.
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
148
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
149 This method will only be called once, and only after `is_ready`
150 has returned true.
151 """
152 raise NotImplementedError()
153
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
154
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
155 class Callback(YieldPoint):
156 """Returns a callable object that will allow a matching `Wait` to proceed.
157
158 The key may be any value suitable for use as a dictionary key, and is
159 used to match ``Callbacks`` to their corresponding ``Waits``. The key
160 must be unique among outstanding callbacks within a single run of the
161 generator function, but may be reused across different runs of the same
162 function (so constants generally work fine).
163
164 The callback may be called with zero or one arguments; if an argument
165 is given it will be returned by `Wait`.
166 """
167 def __init__(self, key):
168 self.key = key
169
170 def start(self, runner):
171 self.runner = runner
172 runner.register_callback(self.key)
173
174 def is_ready(self):
175 return True
176
177 def get_result(self):
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
178 return self.runner.result_callback(self.key)
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
179
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
180
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
181 class Wait(YieldPoint):
182 """Returns the argument passed to the result of a previous `Callback`."""
183 def __init__(self, key):
184 self.key = key
185
186 def start(self, runner):
187 self.runner = runner
188
189 def is_ready(self):
190 return self.runner.is_ready(self.key)
191
192 def get_result(self):
193 return self.runner.pop_result(self.key)
194
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
195
0965952 @bdarnell Add gen.WaitAll
bdarnell authored
196 class WaitAll(YieldPoint):
197 """Returns the results of multiple previous `Callbacks`.
198
199 The argument is a sequence of `Callback` keys, and the result is
200 a list of results in the same order.
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
201
202 `WaitAll` is equivalent to yielding a list of `Wait` objects.
0965952 @bdarnell Add gen.WaitAll
bdarnell authored
203 """
204 def __init__(self, keys):
205 self.keys = keys
206
207 def start(self, runner):
208 self.runner = runner
209
210 def is_ready(self):
211 return all(self.runner.is_ready(key) for key in self.keys)
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
212
0965952 @bdarnell Add gen.WaitAll
bdarnell authored
213 def get_result(self):
214 return [self.runner.pop_result(key) for key in self.keys]
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
215
0965952 @bdarnell Add gen.WaitAll
bdarnell authored
216
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
217 class Task(YieldPoint):
218 """Runs a single asynchronous operation.
219
220 Takes a function (and optional additional arguments) and runs it with
221 those arguments plus a ``callback`` keyword argument. The argument passed
222 to the callback is returned as the result of the yield expression.
223
224 A `Task` is equivalent to a `Callback`/`Wait` pair (with a unique
225 key generated automatically)::
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
226
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
227 result = yield gen.Task(func, args)
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
228
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
229 func(args, callback=(yield gen.Callback(key)))
230 result = yield gen.Wait(key)
231 """
232 def __init__(self, func, *args, **kwargs):
233 assert "callback" not in kwargs
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
234 self.args = args
235 self.kwargs = kwargs
236 self.func = func
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
237
238 def start(self, runner):
239 self.runner = runner
240 self.key = object()
241 runner.register_callback(self.key)
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
242 self.kwargs["callback"] = runner.result_callback(self.key)
243 self.func(*self.args, **self.kwargs)
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
244
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
245 def is_ready(self):
246 return self.runner.is_ready(self.key)
247
248 def get_result(self):
249 return self.runner.pop_result(self.key)
250
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
251
5587a45 @bdarnell Add gen.YieldFuture.
bdarnell authored
252 class YieldFuture(YieldPoint):
253 def __init__(self, future, io_loop=None):
254 self.future = future
0cc0df7 @bdarnell Add magic for yielding futures.
bdarnell authored
255 self.io_loop = io_loop or IOLoop.current()
5587a45 @bdarnell Add gen.YieldFuture.
bdarnell authored
256
257 def start(self, runner):
258 self.runner = runner
259 self.key = object()
260 runner.register_callback(self.key)
261 self.io_loop.add_future(self.future, runner.result_callback(self.key))
262
263 def is_ready(self):
264 return self.runner.is_ready(self.key)
265
266 def get_result(self):
267 return self.runner.pop_result(self.key).result()
268
269
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
270 class Multi(YieldPoint):
271 """Runs multiple asynchronous operations in parallel.
272
273 Takes a list of ``Tasks`` or other ``YieldPoints`` and returns a list of
274 their responses. It is not necessary to call `Multi` explicitly,
275 since the engine will do so automatically when the generator yields
276 a list of ``YieldPoints``.
277 """
278 def __init__(self, children):
279 assert all(isinstance(i, YieldPoint) for i in children)
280 self.children = children
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
281
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
282 def start(self, runner):
283 for i in self.children:
284 i.start(runner)
285
286 def is_ready(self):
287 return all(i.is_ready() for i in self.children)
288
289 def get_result(self):
290 return [i.get_result() for i in self.children]
291
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
292
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
293 class _NullYieldPoint(YieldPoint):
294 def start(self, runner):
295 pass
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
296
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
297 def is_ready(self):
298 return True
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
299
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
300 def get_result(self):
301 return None
302
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
303
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
304 class Runner(object):
305 """Internal implementation of `tornado.gen.engine`.
306
307 Maintains information about pending callbacks and their results.
308 """
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
309 def __init__(self, gen, deactivate_stack_context):
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
310 self.gen = gen
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
311 self.deactivate_stack_context = deactivate_stack_context
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
312 self.yield_point = _NullYieldPoint()
313 self.pending_callbacks = set()
314 self.results = {}
315 self.running = False
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
316 self.finished = False
317 self.exc_info = None
008e605 @bdarnell Allow exceptions thrown in the first (synchronous) phase of a gen.Task
bdarnell authored
318 self.had_exception = False
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
319
320 def register_callback(self, key):
321 """Adds ``key`` to the list of callbacks."""
322 if key in self.pending_callbacks:
323 raise KeyReuseError("key %r is already pending" % key)
324 self.pending_callbacks.add(key)
325
326 def is_ready(self, key):
327 """Returns true if a result is available for ``key``."""
328 if key not in self.pending_callbacks:
329 raise UnknownKeyError("key %r is not pending" % key)
330 return key in self.results
331
332 def set_result(self, key, result):
333 """Sets the result for ``key`` and attempts to resume the generator."""
334 self.results[key] = result
335 self.run()
336
337 def pop_result(self, key):
338 """Returns the result for ``key`` and unregisters it."""
339 self.pending_callbacks.remove(key)
340 return self.results.pop(key)
341
342 def run(self):
343 """Starts or resumes the generator, running until it reaches a
344 yield point that is not ready.
345 """
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
346 if self.running or self.finished:
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
347 return
348 try:
349 self.running = True
350 while True:
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
351 if self.exc_info is None:
352 try:
353 if not self.yield_point.is_ready():
354 return
355 next = self.yield_point.get_result()
356 except Exception:
357 self.exc_info = sys.exc_info()
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
358 try:
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
359 if self.exc_info is not None:
008e605 @bdarnell Allow exceptions thrown in the first (synchronous) phase of a gen.Task
bdarnell authored
360 self.had_exception = True
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
361 exc_info = self.exc_info
362 self.exc_info = None
363 yielded = self.gen.throw(*exc_info)
364 else:
365 yielded = self.gen.send(next)
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
366 except StopIteration:
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
367 self.finished = True
008e605 @bdarnell Allow exceptions thrown in the first (synchronous) phase of a gen.Task
bdarnell authored
368 if self.pending_callbacks and not self.had_exception:
369 # If we ran cleanly without waiting on all callbacks
370 # raise an error (really more of a warning). If we
371 # had an exception then some callbacks may have been
372 # orphaned, so skip the check in that case.
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
373 raise LeakedCallbackError(
374 "finished without waiting for callbacks %r" %
375 self.pending_callbacks)
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
376 self.deactivate_stack_context()
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
377 return
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
378 except Exception:
379 self.finished = True
380 raise
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
381 if isinstance(yielded, list):
382 yielded = Multi(yielded)
0cc0df7 @bdarnell Add magic for yielding futures.
bdarnell authored
383 if isinstance(yielded, Future):
384 # TODO: lists of futures
385 yielded = YieldFuture(yielded)
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
386 if isinstance(yielded, YieldPoint):
387 self.yield_point = yielded
008e605 @bdarnell Allow exceptions thrown in the first (synchronous) phase of a gen.Task
bdarnell authored
388 try:
389 self.yield_point.start(self)
390 except Exception:
391 self.exc_info = sys.exc_info()
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
392 else:
393 self.exc_info = (BadYieldError("yielded unknown object %r" % yielded),)
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
394 finally:
395 self.running = False
396
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
397 def result_callback(self, key):
398 def inner(*args, **kwargs):
399 if kwargs or len(args) > 1:
400 result = Arguments(args, kwargs)
401 elif args:
402 result = args[0]
403 else:
404 result = None
405 self.set_result(key, result)
406 return inner
407
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous function...
bdarnell authored
408 def handle_exception(self, typ, value, tb):
409 if not self.running and not self.finished:
410 self.exc_info = (typ, value, tb)
411 self.run()
412 return True
413 else:
414 return False
415
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
416 # in python 2.6+ this could be a collections.namedtuple
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
417
418
5872db2 @bdarnell Add support for callbacks that take more than a single positional argume...
bdarnell authored
419 class Arguments(tuple):
420 """The result of a yield expression whose callback had more than one
421 argument (or keyword arguments).
422
423 The `Arguments` object can be used as a tuple ``(args, kwargs)``
424 or an object with attributes ``args`` and ``kwargs``.
425 """
426 __slots__ = ()
427
428 def __new__(cls, args, kwargs):
429 return tuple.__new__(cls, (args, kwargs))
430
431 args = property(operator.itemgetter(0))
432 kwargs = property(operator.itemgetter(1))
Something went wrong with that request. Please try again.