Skip to content
This repository
Newer
Older
100644 407 lines (320 sloc) 13.655 kb
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
1 """``tornado.gen`` is a generator-based interface to make it easier to
2 work in an asynchronous environment. Code using the ``gen`` module
3 is technically asynchronous, but it is written as a single generator
4 instead of a collection of separate functions.
5
6 For example, the following asynchronous handler::
7
8 class AsyncHandler(RequestHandler):
9 @asynchronous
10 def get(self):
11 http_client = AsyncHTTPClient()
12 http_client.fetch("http://example.com",
13 callback=self.on_fetch)
14
15 def on_fetch(self, response):
16 do_something_with_response(response)
17 self.render("template.html")
18
19 could be written with ``gen`` as::
20
21 class GenAsyncHandler(RequestHandler):
22 @asynchronous
23 @gen.engine
24 def get(self):
25 http_client = AsyncHTTPClient()
93a0dc82 »
2011-09-04 Fix error in docs
26 response = yield gen.Task(http_client.fetch, "http://example.com")
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
27 do_something_with_response(response)
28 self.render("template.html")
29
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
30 `Task` works with any function that takes a ``callback`` keyword
31 argument. You can also yield a list of ``Tasks``, which will be
32 started at the same time and run in parallel; a list of results will
33 be returned when they are all finished::
94b483ef »
2011-09-11 Add support for lists of YieldPoints in the gen framework.
34
35 def get(self):
36 http_client = AsyncHTTPClient()
37 response1, response2 = yield [gen.Task(http_client.fetch, url1),
38 gen.Task(http_client.fetch, url2)]
39
40 For more complicated interfaces, `Task` can be split into two parts:
41 `Callback` and `Wait`::
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
42
43 class GenAsyncHandler2(RequestHandler):
44 @asynchronous
45 @gen.engine
46 def get(self):
47 http_client = AsyncHTTPClient()
48 http_client.fetch("http://example.com",
49 callback=(yield gen.Callback("key"))
50 response = yield gen.Wait("key")
51 do_something_with_response(response)
52 self.render("template.html")
53
54 The ``key`` argument to `Callback` and `Wait` allows for multiple
94b483ef »
2011-09-11 Add support for lists of YieldPoints in the gen framework.
55 asynchronous operations to be started at different times and proceed
56 in parallel: yield several callbacks with different keys, then wait
57 for them once all the async operations have started.
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
58
59 The result of a `Wait` or `Task` yield expression depends on how the callback
60 was run. If it was called with no arguments, the result is ``None``. If
61 it was called with one argument, the result is that argument. If it was
62 called with more than one argument or any keyword arguments, the result
63 is an `Arguments` object, which is a named tuple ``(args, kwargs)``.
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
64 """
58a7ff10 »
2012-02-09 Turn on __future__ division too.
65 from __future__ import absolute_import, division, with_statement
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
66
67 import functools
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
68 import operator
5997f411 »
2011-09-05 Improve exception handling for gen module.
69 import sys
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
70 import types
71
ac9902cb »
2012-01-01 Use a StackContext to allow exceptions thrown from asynchronous funct…
72 from tornado.stack_context import ExceptionStackContext
73
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
74
75 class KeyReuseError(Exception):
76 pass
77
78
79 class UnknownKeyError(Exception):
80 pass
81
82
83 class LeakedCallbackError(Exception):
84 pass
85
86
87 class BadYieldError(Exception):
88 pass
89
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
90
91 def engine(func):
92 """Decorator for asynchronous generators.
93
94 Any generator that yields objects from this module must be wrapped
95 in this decorator. The decorator only works on functions that are
96 already asynchronous. For `~tornado.web.RequestHandler`
7fd17884 »
2012-01-23 Fix gen.engine docs on decorator order.
97 ``get``/``post``/etc methods, this means that both the
98 `tornado.web.asynchronous` and `tornado.gen.engine` decorators
99 must be used (for proper exception handling, ``asynchronous``
100 should come before ``gen.engine``). In most other cases, it means
101 that it doesn't make sense to use ``gen.engine`` on functions that
102 don't already take a callback argument.
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
103 """
104 @functools.wraps(func)
105 def wrapper(*args, **kwargs):
ac9902cb »
2012-01-01 Use a StackContext to allow exceptions thrown from asynchronous funct…
106 runner = None
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
107
ac9902cb »
2012-01-01 Use a StackContext to allow exceptions thrown from asynchronous funct…
108 def handle_exception(typ, value, tb):
109 # if the function throws an exception before its first "yield"
110 # (or is not a generator at all), the Runner won't exist yet.
111 # However, in that case we haven't reached anything asynchronous
112 # yet, so we can just let the exception propagate.
113 if runner is not None:
114 return runner.handle_exception(typ, value, tb)
115 return False
116 with ExceptionStackContext(handle_exception):
117 gen = func(*args, **kwargs)
118 if isinstance(gen, types.GeneratorType):
119 runner = Runner(gen)
120 runner.run()
121 return
122 assert gen is None, gen
123 # no yield, so we're done
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
124 return wrapper
125
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
126
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
127 class YieldPoint(object):
128 """Base class for objects that may be yielded from the generator."""
129 def start(self, runner):
130 """Called by the runner after the generator has yielded.
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
131
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
132 No other methods will be called on this object before ``start``.
133 """
134 raise NotImplementedError()
135
136 def is_ready(self):
137 """Called by the runner to determine whether to resume the generator.
138
2f914601 »
2011-09-05 Allow any sequence in gen.WaitAll, not just lists.
139 Returns a boolean; may be called more than once.
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
140 """
141 raise NotImplementedError()
142
143 def get_result(self):
144 """Returns the value to use as the result of the yield expression.
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
145
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
146 This method will only be called once, and only after `is_ready`
147 has returned true.
148 """
149 raise NotImplementedError()
150
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
151
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
152 class Callback(YieldPoint):
153 """Returns a callable object that will allow a matching `Wait` to proceed.
154
155 The key may be any value suitable for use as a dictionary key, and is
156 used to match ``Callbacks`` to their corresponding ``Waits``. The key
157 must be unique among outstanding callbacks within a single run of the
158 generator function, but may be reused across different runs of the same
159 function (so constants generally work fine).
160
161 The callback may be called with zero or one arguments; if an argument
162 is given it will be returned by `Wait`.
163 """
164 def __init__(self, key):
165 self.key = key
166
167 def start(self, runner):
168 self.runner = runner
169 runner.register_callback(self.key)
170
171 def is_ready(self):
172 return True
173
174 def get_result(self):
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
175 return self.runner.result_callback(self.key)
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
176
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
177
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
178 class Wait(YieldPoint):
179 """Returns the argument passed to the result of a previous `Callback`."""
180 def __init__(self, key):
181 self.key = key
182
183 def start(self, runner):
184 self.runner = runner
185
186 def is_ready(self):
187 return self.runner.is_ready(self.key)
188
189 def get_result(self):
190 return self.runner.pop_result(self.key)
191
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
192
09659523 »
2011-09-04 Add gen.WaitAll
193 class WaitAll(YieldPoint):
194 """Returns the results of multiple previous `Callbacks`.
195
196 The argument is a sequence of `Callback` keys, and the result is
197 a list of results in the same order.
94b483ef »
2011-09-11 Add support for lists of YieldPoints in the gen framework.
198
199 `WaitAll` is equivalent to yielding a list of `Wait` objects.
09659523 »
2011-09-04 Add gen.WaitAll
200 """
201 def __init__(self, keys):
202 self.keys = keys
203
204 def start(self, runner):
205 self.runner = runner
206
207 def is_ready(self):
208 return all(self.runner.is_ready(key) for key in self.keys)
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
209
09659523 »
2011-09-04 Add gen.WaitAll
210 def get_result(self):
211 return [self.runner.pop_result(key) for key in self.keys]
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
212
09659523 »
2011-09-04 Add gen.WaitAll
213
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
214 class Task(YieldPoint):
215 """Runs a single asynchronous operation.
216
217 Takes a function (and optional additional arguments) and runs it with
218 those arguments plus a ``callback`` keyword argument. The argument passed
219 to the callback is returned as the result of the yield expression.
220
221 A `Task` is equivalent to a `Callback`/`Wait` pair (with a unique
222 key generated automatically)::
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
223
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
224 result = yield gen.Task(func, args)
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
225
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
226 func(args, callback=(yield gen.Callback(key)))
227 result = yield gen.Wait(key)
228 """
229 def __init__(self, func, *args, **kwargs):
230 assert "callback" not in kwargs
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
231 self.args = args
232 self.kwargs = kwargs
233 self.func = func
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
234
235 def start(self, runner):
236 self.runner = runner
237 self.key = object()
238 runner.register_callback(self.key)
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
239 self.kwargs["callback"] = runner.result_callback(self.key)
240 self.func(*self.args, **self.kwargs)
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
241
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
242 def is_ready(self):
243 return self.runner.is_ready(self.key)
244
245 def get_result(self):
246 return self.runner.pop_result(self.key)
247
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
248
94b483ef »
2011-09-11 Add support for lists of YieldPoints in the gen framework.
249 class Multi(YieldPoint):
250 """Runs multiple asynchronous operations in parallel.
251
252 Takes a list of ``Tasks`` or other ``YieldPoints`` and returns a list of
253 their responses. It is not necessary to call `Multi` explicitly,
254 since the engine will do so automatically when the generator yields
255 a list of ``YieldPoints``.
256 """
257 def __init__(self, children):
258 assert all(isinstance(i, YieldPoint) for i in children)
259 self.children = children
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
260
94b483ef »
2011-09-11 Add support for lists of YieldPoints in the gen framework.
261 def start(self, runner):
262 for i in self.children:
263 i.start(runner)
264
265 def is_ready(self):
266 return all(i.is_ready() for i in self.children)
267
268 def get_result(self):
269 return [i.get_result() for i in self.children]
270
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
271
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
272 class _NullYieldPoint(YieldPoint):
273 def start(self, runner):
274 pass
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
275
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
276 def is_ready(self):
277 return True
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
278
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
279 def get_result(self):
280 return None
281
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
282
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
283 class Runner(object):
284 """Internal implementation of `tornado.gen.engine`.
285
286 Maintains information about pending callbacks and their results.
287 """
288 def __init__(self, gen):
289 self.gen = gen
290 self.yield_point = _NullYieldPoint()
291 self.pending_callbacks = set()
292 self.results = {}
293 self.running = False
5997f411 »
2011-09-05 Improve exception handling for gen module.
294 self.finished = False
295 self.exc_info = None
008e605e »
2012-01-01 Allow exceptions thrown in the first (synchronous) phase of a gen.Task
296 self.had_exception = False
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
297
298 def register_callback(self, key):
299 """Adds ``key`` to the list of callbacks."""
300 if key in self.pending_callbacks:
301 raise KeyReuseError("key %r is already pending" % key)
302 self.pending_callbacks.add(key)
303
304 def is_ready(self, key):
305 """Returns true if a result is available for ``key``."""
306 if key not in self.pending_callbacks:
307 raise UnknownKeyError("key %r is not pending" % key)
308 return key in self.results
309
310 def set_result(self, key, result):
311 """Sets the result for ``key`` and attempts to resume the generator."""
312 self.results[key] = result
313 self.run()
314
315 def pop_result(self, key):
316 """Returns the result for ``key`` and unregisters it."""
317 self.pending_callbacks.remove(key)
318 return self.results.pop(key)
319
320 def run(self):
321 """Starts or resumes the generator, running until it reaches a
322 yield point that is not ready.
323 """
5997f411 »
2011-09-05 Improve exception handling for gen module.
324 if self.running or self.finished:
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
325 return
326 try:
327 self.running = True
328 while True:
5997f411 »
2011-09-05 Improve exception handling for gen module.
329 if self.exc_info is None:
330 try:
331 if not self.yield_point.is_ready():
332 return
333 next = self.yield_point.get_result()
334 except Exception:
335 self.exc_info = sys.exc_info()
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
336 try:
5997f411 »
2011-09-05 Improve exception handling for gen module.
337 if self.exc_info is not None:
008e605e »
2012-01-01 Allow exceptions thrown in the first (synchronous) phase of a gen.Task
338 self.had_exception = True
5997f411 »
2011-09-05 Improve exception handling for gen module.
339 exc_info = self.exc_info
340 self.exc_info = None
341 yielded = self.gen.throw(*exc_info)
342 else:
343 yielded = self.gen.send(next)
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
344 except StopIteration:
5997f411 »
2011-09-05 Improve exception handling for gen module.
345 self.finished = True
008e605e »
2012-01-01 Allow exceptions thrown in the first (synchronous) phase of a gen.Task
346 if self.pending_callbacks and not self.had_exception:
347 # If we ran cleanly without waiting on all callbacks
348 # raise an error (really more of a warning). If we
349 # had an exception then some callbacks may have been
350 # orphaned, so skip the check in that case.
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
351 raise LeakedCallbackError(
352 "finished without waiting for callbacks %r" %
353 self.pending_callbacks)
354 return
5997f411 »
2011-09-05 Improve exception handling for gen module.
355 except Exception:
356 self.finished = True
357 raise
94b483ef »
2011-09-11 Add support for lists of YieldPoints in the gen framework.
358 if isinstance(yielded, list):
359 yielded = Multi(yielded)
5997f411 »
2011-09-05 Improve exception handling for gen module.
360 if isinstance(yielded, YieldPoint):
361 self.yield_point = yielded
008e605e »
2012-01-01 Allow exceptions thrown in the first (synchronous) phase of a gen.Task
362 try:
363 self.yield_point.start(self)
364 except Exception:
365 self.exc_info = sys.exc_info()
5997f411 »
2011-09-05 Improve exception handling for gen module.
366 else:
367 self.exc_info = (BadYieldError("yielded unknown object %r" % yielded),)
e4ead597 »
2011-09-04 Add tornado.gen module for simpler generator-based async code.
368 finally:
369 self.running = False
370
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
371 def result_callback(self, key):
372 def inner(*args, **kwargs):
373 if kwargs or len(args) > 1:
374 result = Arguments(args, kwargs)
375 elif args:
376 result = args[0]
377 else:
378 result = None
379 self.set_result(key, result)
380 return inner
381
ac9902cb »
2012-01-01 Use a StackContext to allow exceptions thrown from asynchronous funct…
382 def handle_exception(self, typ, value, tb):
383 if not self.running and not self.finished:
384 self.exc_info = (typ, value, tb)
385 self.run()
386 return True
387 else:
388 return False
389
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
390 # in python 2.6+ this could be a collections.namedtuple
c152b784 »
2012-02-09 While I'm touching every file, run autopep8 too.
391
392
5872db2e »
2011-09-11 Add support for callbacks that take more than a single positional arg…
393 class Arguments(tuple):
394 """The result of a yield expression whose callback had more than one
395 argument (or keyword arguments).
396
397 The `Arguments` object can be used as a tuple ``(args, kwargs)``
398 or an object with attributes ``args`` and ``kwargs``.
399 """
400 __slots__ = ()
401
402 def __new__(cls, args, kwargs):
403 return tuple.__new__(cls, (args, kwargs))
404
405 args = property(operator.itemgetter(0))
406 kwargs = property(operator.itemgetter(1))
Something went wrong with that request. Please try again.