Skip to content

Commit 2bcdd29

Browse files
Phil9lPhil9l
committed
Finish chapter 7
1 parent 6b9217e commit 2bcdd29

File tree

2 files changed

+646
-0
lines changed

2 files changed

+646
-0
lines changed

‎07. Functions/11. Inlining Callback Functions.ipynb‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
{
2+
"metadata": {
3+
"name": ""
4+
},
5+
"nbformat": 3,
6+
"nbformat_minor": 0,
7+
"worksheets": [
8+
{
9+
"cells": [
10+
{
11+
"cell_type": "heading",
12+
"level": 1,
13+
"metadata": {},
14+
"source": [
15+
"7.11. Inlining Callback Functions"
16+
]
17+
},
18+
{
19+
"cell_type": "markdown",
20+
"metadata": {},
21+
"source": [
22+
"# Problem\n",
23+
"You\u2019re writing code that uses callback functions, but you\u2019re concerned about the pro\u2010\n",
24+
"liferation of small functions and mind boggling control flow. You would like some way\n",
25+
"to make the code look more like a normal sequence of procedural steps.\n",
26+
"\n",
27+
"# Solution\n",
28+
"Callback functions can be inlined into a function using generators and coroutines. To\n",
29+
"illustrate, suppose you have a function that performs work and invokes a callback as\n",
30+
"follows (see Recipe 7.10):"
31+
]
32+
},
33+
{
34+
"cell_type": "code",
35+
"collapsed": false,
36+
"input": [
37+
"def apply_async(func, args, *, callback):\n",
38+
" # Compute the result\n",
39+
" result = func(*args)\n",
40+
" # Invoke the callback with the result\n",
41+
" callback(result)"
42+
],
43+
"language": "python",
44+
"metadata": {},
45+
"outputs": [],
46+
"prompt_number": 1
47+
},
48+
{
49+
"cell_type": "markdown",
50+
"metadata": {},
51+
"source": [
52+
"Now take a look at the following supporting code, which involves an `Async` class and\n",
53+
"an `inlined_async` decorator:"
54+
]
55+
},
56+
{
57+
"cell_type": "code",
58+
"collapsed": false,
59+
"input": [
60+
"from queue import Queue\n",
61+
"from functools import wraps\n",
62+
"\n",
63+
"class Async:\n",
64+
" def __init__(self, func, args):\n",
65+
" self.func = func\n",
66+
" self.args = args\n",
67+
"\n",
68+
" def inlined_async(func):\n",
69+
" @wraps(func)\n",
70+
" def wrapper(*args):\n",
71+
" f = func(*args)\n",
72+
" result_queue = Queue()\n",
73+
" result_queue.put(None)\n",
74+
" while True:\n",
75+
" result = result_queue.get()\n",
76+
" try:\n",
77+
" a = f.send(result)\n",
78+
" apply_async(a.func, a.args, callback=result_queue.put)\n",
79+
" except StopIteration:\n",
80+
" break\n",
81+
" return wrapper"
82+
],
83+
"language": "python",
84+
"metadata": {},
85+
"outputs": [],
86+
"prompt_number": 2
87+
},
88+
{
89+
"cell_type": "markdown",
90+
"metadata": {},
91+
"source": [
92+
"These two fragments of code will allow you to inline the callback steps using `yield`\n",
93+
"statements. For example:"
94+
]
95+
},
96+
{
97+
"cell_type": "code",
98+
"collapsed": false,
99+
"input": [
100+
"def add(x, y):\n",
101+
" return x + y\n",
102+
"\n",
103+
"@Async.inlined_async\n",
104+
"def test():\n",
105+
" r = yield Async(add, (2, 3))\n",
106+
" print(r)\n",
107+
" r = yield Async(add, ('hello', 'world'))\n",
108+
" print(r)\n",
109+
" for n in range(10):\n",
110+
" r = yield Async(add, (n, n))\n",
111+
" print(r)\n",
112+
" print('Goodbye')"
113+
],
114+
"language": "python",
115+
"metadata": {},
116+
"outputs": [],
117+
"prompt_number": 3
118+
},
119+
{
120+
"cell_type": "markdown",
121+
"metadata": {},
122+
"source": [
123+
"If you call `test()`, you\u2019ll get output like this:\n",
124+
"```\n",
125+
"5\n",
126+
"helloworld\n",
127+
"0\n",
128+
"2\n",
129+
"4\n",
130+
"6\n",
131+
"8\n",
132+
"10\n",
133+
"12\n",
134+
"14\n",
135+
"16\n",
136+
"18\n",
137+
"Goodbye\n",
138+
"```"
139+
]
140+
},
141+
{
142+
"cell_type": "code",
143+
"collapsed": false,
144+
"input": [
145+
"test()"
146+
],
147+
"language": "python",
148+
"metadata": {},
149+
"outputs": [
150+
{
151+
"output_type": "stream",
152+
"stream": "stdout",
153+
"text": [
154+
"5\n",
155+
"helloworld\n",
156+
"0\n",
157+
"2\n",
158+
"4\n",
159+
"6\n",
160+
"8\n",
161+
"10\n",
162+
"12\n",
163+
"14\n",
164+
"16\n",
165+
"18\n",
166+
"Goodbye\n"
167+
]
168+
}
169+
],
170+
"prompt_number": 5
171+
},
172+
{
173+
"cell_type": "markdown",
174+
"metadata": {},
175+
"source": [
176+
"Aside from the special decorator and use of `yield`, you will notice that no callback\n",
177+
"functions appear anywhere (except behind the scenes)."
178+
]
179+
},
180+
{
181+
"cell_type": "markdown",
182+
"metadata": {},
183+
"source": [
184+
"# Discussion\n",
185+
"This recipe will really test your knowledge of callback functions, generators, and control\n",
186+
"flow.\n",
187+
"\n",
188+
"First, in code involving callbacks, the whole point is that the current calculation will\n",
189+
"suspend and resume at some later point in time (e.g., asynchronously). When the \n",
190+
"calculation resumes, the callback will get executed to continue the processing. The \n",
191+
"`apply_async()` function illustrates the essential parts of executing the callback, although\n",
192+
"in reality it might be much more complicated (involving threads, processes, event \n",
193+
"handlers, etc.).\n",
194+
"\n",
195+
"The idea that a calculation will suspend and resume naturally maps to the execution\n",
196+
"model of a generator function. Specifically, the `yield` operation makes a generator\n",
197+
"function emit a value and suspend. Subsequent calls to the `__next__()` or `send()`\n",
198+
"method of a generator will make it start again.\n",
199+
"\n",
200+
"With this in mind, the core of this recipe is found in the `inline_async()` decorator\n",
201+
"function. The key idea is that the decorator will step the generator function through all\n",
202+
"of its `yield` statements, one at a time. To do this, a result queue is created and initially\n",
203+
"populated with a value of `None`. A loop is then initiated in which a result is popped off\n",
204+
"the queue and sent into the generator. This advances to the next yield, at which point\n",
205+
"an instance of `Async` is received. The loop then looks at the function and arguments,\n",
206+
"and initiates the asynchronous calculation `apply_async()`. However, the sneakiest part\n",
207+
"of this calculation is that instead of using a normal callback function, the callback is set\n",
208+
"to the queue `put()` method.\n",
209+
"\n",
210+
"At this point, it is left somewhat open as to precisely what happens. The main loop\n",
211+
"immediately goes back to the top and simply executes a `get()` operation on the queue.\n",
212+
"If data is present, it must be the result placed there by the `put()` callback. If nothing is\n",
213+
"there, the operation blocks, waiting for a result to arrive at some future time. How that\n",
214+
"might happen depends on the precise implementation of the `apply_async()` function.\n",
215+
"\n",
216+
"If you\u2019re doubtful that anything this crazy would work, you can try it with the \n",
217+
"multiprocessing library and have async operations executed in separate processes:\n",
218+
"```python\n",
219+
"if __name__ == '__main__':\n",
220+
" import multiprocessing\n",
221+
" pool = multiprocessing.Pool()\n",
222+
" apply_async = pool.apply_async\n",
223+
"\n",
224+
" # Run the test function\n",
225+
" test()\n",
226+
"```\n",
227+
"Indeed, you\u2019ll find that it works, but unraveling the control flow might require more\n",
228+
"coffee.\n",
229+
"\n",
230+
"Hiding tricky control flow behind generator functions is found elsewhere in the \n",
231+
"standard library and third-party packages. For example, the `@contextmanager` decorator in\n",
232+
"the `contextlib` performs a similar mind-bending trick that glues the entry and exit\n",
233+
"from a context manager together across a yield statement. The popular \n",
234+
"[Twisted package](http://twistedmatrix.com/) has inlined callbacks that are also similar."
235+
]
236+
}
237+
],
238+
"metadata": {}
239+
}
240+
]
241+
}

0 commit comments

Comments
 (0)