Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 411 lines (324 sloc) 13.906 kb
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
1 """``tornado.gen`` is a generator-based interface to make it easier to
2 work in an asynchronous environment. Code using the ``gen`` module
3 is technically asynchronous, but it is written as a single generator
4 instead of a collection of separate functions.
5
6 For example, the following asynchronous handler::
7
8 class AsyncHandler(RequestHandler):
9 @asynchronous
10 def get(self):
11 http_client = AsyncHTTPClient()
12 http_client.fetch("http://example.com",
13 callback=self.on_fetch)
14
15 def on_fetch(self, response):
16 do_something_with_response(response)
17 self.render("template.html")
18
19 could be written with ``gen`` as::
20
21 class GenAsyncHandler(RequestHandler):
22 @asynchronous
23 @gen.engine
24 def get(self):
25 http_client = AsyncHTTPClient()
93a0dc8 @bdarnell Fix error in docs
bdarnell authored
26 response = yield gen.Task(http_client.fetch, "http://example.com")
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
27 do_something_with_response(response)
28 self.render("template.html")
29
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
30 `Task` works with any function that takes a ``callback`` keyword
31 argument. You can also yield a list of ``Tasks``, which will be
32 started at the same time and run in parallel; a list of results will
33 be returned when they are all finished::
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
34
35 def get(self):
36 http_client = AsyncHTTPClient()
37 response1, response2 = yield [gen.Task(http_client.fetch, url1),
38 gen.Task(http_client.fetch, url2)]
39
40 For more complicated interfaces, `Task` can be split into two parts:
41 `Callback` and `Wait`::
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
42
43 class GenAsyncHandler2(RequestHandler):
44 @asynchronous
45 @gen.engine
46 def get(self):
47 http_client = AsyncHTTPClient()
48 http_client.fetch("http://example.com",
49 callback=(yield gen.Callback("key"))
50 response = yield gen.Wait("key")
51 do_something_with_response(response)
52 self.render("template.html")
53
54 The ``key`` argument to `Callback` and `Wait` allows for multiple
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
55 asynchronous operations to be started at different times and proceed
56 in parallel: yield several callbacks with different keys, then wait
57 for them once all the async operations have started.
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
58
59 The result of a `Wait` or `Task` yield expression depends on how the callback
60 was run. If it was called with no arguments, the result is ``None``. If
61 it was called with one argument, the result is that argument. If it was
62 called with more than one argument or any keyword arguments, the result
63 is an `Arguments` object, which is a named tuple ``(args, kwargs)``.
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
64 """
58a7ff1 @bdarnell Turn on __future__ division too.
bdarnell authored
65 from __future__ import absolute_import, division, with_statement
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
66
67 import functools
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
68 import operator
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
69 import sys
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
70 import types
71
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous funct…
bdarnell authored
72 from tornado.stack_context import ExceptionStackContext
73
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
74
75 class KeyReuseError(Exception):
76 pass
77
78
79 class UnknownKeyError(Exception):
80 pass
81
82
83 class LeakedCallbackError(Exception):
84 pass
85
86
87 class BadYieldError(Exception):
88 pass
89
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
90
91 def engine(func):
92 """Decorator for asynchronous generators.
93
94 Any generator that yields objects from this module must be wrapped
95 in this decorator. The decorator only works on functions that are
96 already asynchronous. For `~tornado.web.RequestHandler`
7fd1788 @bdarnell Fix gen.engine docs on decorator order.
bdarnell authored
97 ``get``/``post``/etc methods, this means that both the
98 `tornado.web.asynchronous` and `tornado.gen.engine` decorators
99 must be used (for proper exception handling, ``asynchronous``
100 should come before ``gen.engine``). In most other cases, it means
101 that it doesn't make sense to use ``gen.engine`` on functions that
102 don't already take a callback argument.
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
103 """
104 @functools.wraps(func)
105 def wrapper(*args, **kwargs):
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous funct…
bdarnell authored
106 runner = None
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
107
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous funct…
bdarnell authored
108 def handle_exception(typ, value, tb):
109 # if the function throws an exception before its first "yield"
110 # (or is not a generator at all), the Runner won't exist yet.
111 # However, in that case we haven't reached anything asynchronous
112 # yet, so we can just let the exception propagate.
113 if runner is not None:
114 return runner.handle_exception(typ, value, tb)
115 return False
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
116 with ExceptionStackContext(handle_exception) as deactivate:
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous funct…
bdarnell authored
117 gen = func(*args, **kwargs)
118 if isinstance(gen, types.GeneratorType):
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
119 runner = Runner(gen, deactivate)
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous funct…
bdarnell authored
120 runner.run()
121 return
122 assert gen is None, gen
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
123 deactivate()
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous funct…
bdarnell authored
124 # no yield, so we're done
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
125 return wrapper
126
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
127
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
128 class YieldPoint(object):
129 """Base class for objects that may be yielded from the generator."""
130 def start(self, runner):
131 """Called by the runner after the generator has yielded.
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
132
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
133 No other methods will be called on this object before ``start``.
134 """
135 raise NotImplementedError()
136
137 def is_ready(self):
138 """Called by the runner to determine whether to resume the generator.
139
2f91460 @bdarnell Allow any sequence in gen.WaitAll, not just lists.
bdarnell authored
140 Returns a boolean; may be called more than once.
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
141 """
142 raise NotImplementedError()
143
144 def get_result(self):
145 """Returns the value to use as the result of the yield expression.
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
146
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
147 This method will only be called once, and only after `is_ready`
148 has returned true.
149 """
150 raise NotImplementedError()
151
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
152
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
153 class Callback(YieldPoint):
154 """Returns a callable object that will allow a matching `Wait` to proceed.
155
156 The key may be any value suitable for use as a dictionary key, and is
157 used to match ``Callbacks`` to their corresponding ``Waits``. The key
158 must be unique among outstanding callbacks within a single run of the
159 generator function, but may be reused across different runs of the same
160 function (so constants generally work fine).
161
162 The callback may be called with zero or one arguments; if an argument
163 is given it will be returned by `Wait`.
164 """
165 def __init__(self, key):
166 self.key = key
167
168 def start(self, runner):
169 self.runner = runner
170 runner.register_callback(self.key)
171
172 def is_ready(self):
173 return True
174
175 def get_result(self):
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
176 return self.runner.result_callback(self.key)
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
177
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
178
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
179 class Wait(YieldPoint):
180 """Returns the argument passed to the result of a previous `Callback`."""
181 def __init__(self, key):
182 self.key = key
183
184 def start(self, runner):
185 self.runner = runner
186
187 def is_ready(self):
188 return self.runner.is_ready(self.key)
189
190 def get_result(self):
191 return self.runner.pop_result(self.key)
192
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
193
0965952 @bdarnell Add gen.WaitAll
bdarnell authored
194 class WaitAll(YieldPoint):
195 """Returns the results of multiple previous `Callbacks`.
196
197 The argument is a sequence of `Callback` keys, and the result is
198 a list of results in the same order.
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
199
200 `WaitAll` is equivalent to yielding a list of `Wait` objects.
0965952 @bdarnell Add gen.WaitAll
bdarnell authored
201 """
202 def __init__(self, keys):
203 self.keys = keys
204
205 def start(self, runner):
206 self.runner = runner
207
208 def is_ready(self):
209 return all(self.runner.is_ready(key) for key in self.keys)
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
210
0965952 @bdarnell Add gen.WaitAll
bdarnell authored
211 def get_result(self):
212 return [self.runner.pop_result(key) for key in self.keys]
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
213
0965952 @bdarnell Add gen.WaitAll
bdarnell authored
214
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
215 class Task(YieldPoint):
216 """Runs a single asynchronous operation.
217
218 Takes a function (and optional additional arguments) and runs it with
219 those arguments plus a ``callback`` keyword argument. The argument passed
220 to the callback is returned as the result of the yield expression.
221
222 A `Task` is equivalent to a `Callback`/`Wait` pair (with a unique
223 key generated automatically)::
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
224
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
225 result = yield gen.Task(func, args)
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
226
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
227 func(args, callback=(yield gen.Callback(key)))
228 result = yield gen.Wait(key)
229 """
230 def __init__(self, func, *args, **kwargs):
231 assert "callback" not in kwargs
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
232 self.args = args
233 self.kwargs = kwargs
234 self.func = func
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
235
236 def start(self, runner):
237 self.runner = runner
238 self.key = object()
239 runner.register_callback(self.key)
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
240 self.kwargs["callback"] = runner.result_callback(self.key)
241 self.func(*self.args, **self.kwargs)
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
242
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
243 def is_ready(self):
244 return self.runner.is_ready(self.key)
245
246 def get_result(self):
247 return self.runner.pop_result(self.key)
248
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
249
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
250 class Multi(YieldPoint):
251 """Runs multiple asynchronous operations in parallel.
252
253 Takes a list of ``Tasks`` or other ``YieldPoints`` and returns a list of
254 their responses. It is not necessary to call `Multi` explicitly,
255 since the engine will do so automatically when the generator yields
256 a list of ``YieldPoints``.
257 """
258 def __init__(self, children):
259 assert all(isinstance(i, YieldPoint) for i in children)
260 self.children = children
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
261
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
262 def start(self, runner):
263 for i in self.children:
264 i.start(runner)
265
266 def is_ready(self):
267 return all(i.is_ready() for i in self.children)
268
269 def get_result(self):
270 return [i.get_result() for i in self.children]
271
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
272
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
273 class _NullYieldPoint(YieldPoint):
274 def start(self, runner):
275 pass
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
276
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
277 def is_ready(self):
278 return True
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
279
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
280 def get_result(self):
281 return None
282
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
283
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
284 class Runner(object):
285 """Internal implementation of `tornado.gen.engine`.
286
287 Maintains information about pending callbacks and their results.
288 """
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
289 def __init__(self, gen, deactivate_stack_context):
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
290 self.gen = gen
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
291 self.deactivate_stack_context = deactivate_stack_context
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
292 self.yield_point = _NullYieldPoint()
293 self.pending_callbacks = set()
294 self.results = {}
295 self.running = False
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
296 self.finished = False
297 self.exc_info = None
008e605 @bdarnell Allow exceptions thrown in the first (synchronous) phase of a gen.Task
bdarnell authored
298 self.had_exception = False
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
299
300 def register_callback(self, key):
301 """Adds ``key`` to the list of callbacks."""
302 if key in self.pending_callbacks:
303 raise KeyReuseError("key %r is already pending" % key)
304 self.pending_callbacks.add(key)
305
306 def is_ready(self, key):
307 """Returns true if a result is available for ``key``."""
308 if key not in self.pending_callbacks:
309 raise UnknownKeyError("key %r is not pending" % key)
310 return key in self.results
311
312 def set_result(self, key, result):
313 """Sets the result for ``key`` and attempts to resume the generator."""
314 self.results[key] = result
315 self.run()
316
317 def pop_result(self, key):
318 """Returns the result for ``key`` and unregisters it."""
319 self.pending_callbacks.remove(key)
320 return self.results.pop(key)
321
322 def run(self):
323 """Starts or resumes the generator, running until it reaches a
324 yield point that is not ready.
325 """
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
326 if self.running or self.finished:
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
327 return
328 try:
329 self.running = True
330 while True:
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
331 if self.exc_info is None:
332 try:
333 if not self.yield_point.is_ready():
334 return
335 next = self.yield_point.get_result()
336 except Exception:
337 self.exc_info = sys.exc_info()
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
338 try:
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
339 if self.exc_info is not None:
008e605 @bdarnell Allow exceptions thrown in the first (synchronous) phase of a gen.Task
bdarnell authored
340 self.had_exception = True
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
341 exc_info = self.exc_info
342 self.exc_info = None
343 yielded = self.gen.throw(*exc_info)
344 else:
345 yielded = self.gen.send(next)
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
346 except StopIteration:
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
347 self.finished = True
008e605 @bdarnell Allow exceptions thrown in the first (synchronous) phase of a gen.Task
bdarnell authored
348 if self.pending_callbacks and not self.had_exception:
349 # If we ran cleanly without waiting on all callbacks
350 # raise an error (really more of a warning). If we
351 # had an exception then some callbacks may have been
352 # orphaned, so skip the check in that case.
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
353 raise LeakedCallbackError(
354 "finished without waiting for callbacks %r" %
355 self.pending_callbacks)
57a3f83 @bdarnell Prevent leak of StackContexts in repeated gen.engine functions.
bdarnell authored
356 self.deactivate_stack_context()
18a66be @bdarnell Fix a memory leak in stack_context.
bdarnell authored
357 self.deactivate_stack_context = None
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
358 return
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
359 except Exception:
360 self.finished = True
361 raise
94b483e @bdarnell Add support for lists of YieldPoints in the gen framework.
bdarnell authored
362 if isinstance(yielded, list):
363 yielded = Multi(yielded)
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
364 if isinstance(yielded, YieldPoint):
365 self.yield_point = yielded
008e605 @bdarnell Allow exceptions thrown in the first (synchronous) phase of a gen.Task
bdarnell authored
366 try:
367 self.yield_point.start(self)
368 except Exception:
369 self.exc_info = sys.exc_info()
5997f41 @bdarnell Improve exception handling for gen module.
bdarnell authored
370 else:
371 self.exc_info = (BadYieldError("yielded unknown object %r" % yielded),)
e4ead59 @bdarnell Add tornado.gen module for simpler generator-based async code.
bdarnell authored
372 finally:
373 self.running = False
374
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
375 def result_callback(self, key):
376 def inner(*args, **kwargs):
377 if kwargs or len(args) > 1:
378 result = Arguments(args, kwargs)
379 elif args:
380 result = args[0]
381 else:
382 result = None
383 self.set_result(key, result)
384 return inner
385
ac9902c @bdarnell Use a StackContext to allow exceptions thrown from asynchronous funct…
bdarnell authored
386 def handle_exception(self, typ, value, tb):
387 if not self.running and not self.finished:
388 self.exc_info = (typ, value, tb)
389 self.run()
390 return True
391 else:
392 return False
393
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
394 # in python 2.6+ this could be a collections.namedtuple
c152b78 @bdarnell While I'm touching every file, run autopep8 too.
bdarnell authored
395
396
5872db2 @bdarnell Add support for callbacks that take more than a single positional arg…
bdarnell authored
397 class Arguments(tuple):
398 """The result of a yield expression whose callback had more than one
399 argument (or keyword arguments).
400
401 The `Arguments` object can be used as a tuple ``(args, kwargs)``
402 or an object with attributes ``args`` and ``kwargs``.
403 """
404 __slots__ = ()
405
406 def __new__(cls, args, kwargs):
407 return tuple.__new__(cls, (args, kwargs))
408
409 args = property(operator.itemgetter(0))
410 kwargs = property(operator.itemgetter(1))
Something went wrong with that request. Please try again.