-
Notifications
You must be signed in to change notification settings - Fork 58
/
chap2.html
1202 lines (1169 loc) · 117 KB
/
chap2.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
<!DOCTYPE html>
<html lang="en">
<!-- Produced from a LaTeX source file. Note that the production is done -->
<!-- by a very rough-and-ready (and buggy) script, so the HTML and other -->
<!-- code is quite ugly! Later versions should be better. -->
<head>
<meta charset="utf-8">
<meta name="citation_title" content="Neural Networks and Deep Learning">
<meta name="citation_author" content="Nielsen, Michael A.">
<meta name="citation_publication_date" content="2015">
<meta name="citation_fulltext_html_url" content="http://neuralnetworksanddeeplearning.com">
<meta name="citation_publisher" content="Determination Press">
<link rel="icon" href="nnadl_favicon.ICO" />
<title>Neural networks and deep learning</title>
<script src="assets/jquery.min.js"></script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
tex2jax: {inlineMath: [['$','$']]},
"HTML-CSS":
{scale: 92},
TeX: { equationNumbers: { autoNumber: "AMS" }}});
</script>
<script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<link href="assets/style.css" rel="stylesheet">
<link href="assets/pygments.css" rel="stylesheet">
<link rel="stylesheet" href="https://code.jquery.com/ui/1.11.2/themes/smoothness/jquery-ui.css">
<style>
/* Adapted from */
/* https://groups.google.com/d/msg/mathjax-users/jqQxrmeG48o/oAaivLgLN90J, */
/* by David Cervone */
@font-face {
font-family: 'MJX_Math';
src: url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); /* IE9 Compat Modes */
src: url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot?iefix') format('eot'),
url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'),
url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype'),
url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/svg/MathJax_Math-Italic.svg#MathJax_Math-Italic') format('svg');
}
@font-face {
font-family: 'MJX_Main';
src: url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); /* IE9 Compat Modes */
src: url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot?iefix') format('eot'),
url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'),
url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype'),
url('https://cdn.mathjax.org/mathjax/latest/fonts/HTML-CSS/TeX/svg/MathJax_Main-Regular.svg#MathJax_Main-Regular') format('svg');
}
</style>
</head>
<body><div class="header"><h1 class="chapter_number">
<a href="">CHAPTER 2</a></h1>
<h1 class="chapter_title"><a href="">How the backpropagation algorithm works</a></h1></div><div class="section"><div id="toc">
<p class="toc_title"><a href="index.html">Neural Networks and Deep Learning</a></p><p class="toc_not_mainchapter"><a href="about.html">What this book is about</a></p><p class="toc_not_mainchapter"><a href="exercises_and_problems.html">On the exercises and problems</a></p><p class='toc_mainchapter'><a id="toc_using_neural_nets_to_recognize_handwritten_digits_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_using_neural_nets_to_recognize_handwritten_digits" src="images/arrow.png" width="15px"></a><a href="chap1.html">Using neural nets to recognize handwritten digits</a><div id="toc_using_neural_nets_to_recognize_handwritten_digits" style="display: none;"><p class="toc_section"><ul><a href="chap1.html#perceptrons"><li>Perceptrons</li></a><a href="chap1.html#sigmoid_neurons"><li>Sigmoid neurons</li></a><a href="chap1.html#the_architecture_of_neural_networks"><li>The architecture of neural networks</li></a><a href="chap1.html#a_simple_network_to_classify_handwritten_digits"><li>A simple network to classify handwritten digits</li></a><a href="chap1.html#learning_with_gradient_descent"><li>Learning with gradient descent</li></a><a href="chap1.html#implementing_our_network_to_classify_digits"><li>Implementing our network to classify digits</li></a><a href="chap1.html#toward_deep_learning"><li>Toward deep learning</li></a></ul></p></div>
<script>
$('#toc_using_neural_nets_to_recognize_handwritten_digits_reveal').click(function() {
var src = $('#toc_img_using_neural_nets_to_recognize_handwritten_digits').attr('src');
if(src == 'images/arrow.png') {
$("#toc_img_using_neural_nets_to_recognize_handwritten_digits").attr('src', 'images/arrow_down.png');
} else {
$("#toc_img_using_neural_nets_to_recognize_handwritten_digits").attr('src', 'images/arrow.png');
};
$('#toc_using_neural_nets_to_recognize_handwritten_digits').toggle('fast', function() {});
});</script><p class='toc_mainchapter'><a id="toc_how_the_backpropagation_algorithm_works_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_how_the_backpropagation_algorithm_works" src="images/arrow.png" width="15px"></a><a href="chap2.html">How the backpropagation algorithm works</a><div id="toc_how_the_backpropagation_algorithm_works" style="display: none;"><p class="toc_section"><ul><a href="chap2.html#warm_up_a_fast_matrix-based_approach_to_computing_the_output_from_a_neural_network"><li>Warm up: a fast matrix-based approach to computing the output from a neural network</li></a><a href="chap2.html#the_two_assumptions_we_need_about_the_cost_function"><li>The two assumptions we need about the cost function</li></a><a href="chap2.html#the_hadamard_product_$s_\odot_t$"><li>The Hadamard product, $s \odot t$</li></a><a href="chap2.html#the_four_fundamental_equations_behind_backpropagation"><li>The four fundamental equations behind backpropagation</li></a><a href="chap2.html#proof_of_the_four_fundamental_equations_(optional)"><li>Proof of the four fundamental equations (optional)</li></a><a href="chap2.html#the_backpropagation_algorithm"><li>The backpropagation algorithm</li></a><a href="chap2.html#the_code_for_backpropagation"><li>The code for backpropagation</li></a><a href="chap2.html#in_what_sense_is_backpropagation_a_fast_algorithm"><li>In what sense is backpropagation a fast algorithm?</li></a><a href="chap2.html#backpropagation_the_big_picture"><li>Backpropagation: the big picture</li></a></ul></p></div>
<script>
$('#toc_how_the_backpropagation_algorithm_works_reveal').click(function() {
var src = $('#toc_img_how_the_backpropagation_algorithm_works').attr('src');
if(src == 'images/arrow.png') {
$("#toc_img_how_the_backpropagation_algorithm_works").attr('src', 'images/arrow_down.png');
} else {
$("#toc_img_how_the_backpropagation_algorithm_works").attr('src', 'images/arrow.png');
};
$('#toc_how_the_backpropagation_algorithm_works').toggle('fast', function() {});
});</script><p class='toc_mainchapter'><a id="toc_improving_the_way_neural_networks_learn_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_improving_the_way_neural_networks_learn" src="images/arrow.png" width="15px"></a><a href="chap3.html">Improving the way neural networks learn</a><div id="toc_improving_the_way_neural_networks_learn" style="display: none;"><p class="toc_section"><ul><a href="chap3.html#the_cross-entropy_cost_function"><li>The cross-entropy cost function</li></a><a href="chap3.html#overfitting_and_regularization"><li>Overfitting and regularization</li></a><a href="chap3.html#weight_initialization"><li>Weight initialization</li></a><a href="chap3.html#handwriting_recognition_revisited_the_code"><li>Handwriting recognition revisited: the code</li></a><a href="chap3.html#how_to_choose_a_neural_network's_hyper-parameters"><li>How to choose a neural network's hyper-parameters?</li></a><a href="chap3.html#other_techniques"><li>Other techniques</li></a></ul></p></div>
<script>
$('#toc_improving_the_way_neural_networks_learn_reveal').click(function() {
var src = $('#toc_img_improving_the_way_neural_networks_learn').attr('src');
if(src == 'images/arrow.png') {
$("#toc_img_improving_the_way_neural_networks_learn").attr('src', 'images/arrow_down.png');
} else {
$("#toc_img_improving_the_way_neural_networks_learn").attr('src', 'images/arrow.png');
};
$('#toc_improving_the_way_neural_networks_learn').toggle('fast', function() {});
});</script><p class='toc_mainchapter'><a id="toc_a_visual_proof_that_neural_nets_can_compute_any_function_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_a_visual_proof_that_neural_nets_can_compute_any_function" src="images/arrow.png" width="15px"></a><a href="chap4.html">A visual proof that neural nets can compute any function</a><div id="toc_a_visual_proof_that_neural_nets_can_compute_any_function" style="display: none;"><p class="toc_section"><ul><a href="chap4.html#two_caveats"><li>Two caveats</li></a><a href="chap4.html#universality_with_one_input_and_one_output"><li>Universality with one input and one output</li></a><a href="chap4.html#many_input_variables"><li>Many input variables</li></a><a href="chap4.html#extension_beyond_sigmoid_neurons"><li>Extension beyond sigmoid neurons</li></a><a href="chap4.html#fixing_up_the_step_functions"><li>Fixing up the step functions</li></a><a href="chap4.html#conclusion"><li>Conclusion</li></a></ul></p></div>
<script>
$('#toc_a_visual_proof_that_neural_nets_can_compute_any_function_reveal').click(function() {
var src = $('#toc_img_a_visual_proof_that_neural_nets_can_compute_any_function').attr('src');
if(src == 'images/arrow.png') {
$("#toc_img_a_visual_proof_that_neural_nets_can_compute_any_function").attr('src', 'images/arrow_down.png');
} else {
$("#toc_img_a_visual_proof_that_neural_nets_can_compute_any_function").attr('src', 'images/arrow.png');
};
$('#toc_a_visual_proof_that_neural_nets_can_compute_any_function').toggle('fast', function() {});
});</script><p class='toc_mainchapter'><a id="toc_why_are_deep_neural_networks_hard_to_train_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_why_are_deep_neural_networks_hard_to_train" src="images/arrow.png" width="15px"></a><a href="chap5.html">Why are deep neural networks hard to train?</a><div id="toc_why_are_deep_neural_networks_hard_to_train" style="display: none;"><p class="toc_section"><ul><a href="chap5.html#the_vanishing_gradient_problem"><li>The vanishing gradient problem</li></a><a href="chap5.html#what's_causing_the_vanishing_gradient_problem_unstable_gradients_in_deep_neural_nets"><li>What's causing the vanishing gradient problem? Unstable gradients in deep neural nets</li></a><a href="chap5.html#unstable_gradients_in_more_complex_networks"><li>Unstable gradients in more complex networks</li></a><a href="chap5.html#other_obstacles_to_deep_learning"><li>Other obstacles to deep learning</li></a></ul></p></div>
<script>
$('#toc_why_are_deep_neural_networks_hard_to_train_reveal').click(function() {
var src = $('#toc_img_why_are_deep_neural_networks_hard_to_train').attr('src');
if(src == 'images/arrow.png') {
$("#toc_img_why_are_deep_neural_networks_hard_to_train").attr('src', 'images/arrow_down.png');
} else {
$("#toc_img_why_are_deep_neural_networks_hard_to_train").attr('src', 'images/arrow.png');
};
$('#toc_why_are_deep_neural_networks_hard_to_train').toggle('fast', function() {});
});</script><p class='toc_mainchapter'><a id="toc_deep_learning_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_deep_learning" src="images/arrow.png" width="15px"></a><a href="chap6.html">Deep learning</a><div id="toc_deep_learning" style="display: none;"><p class="toc_section"><ul><a href="chap6.html#introducing_convolutional_networks"><li>Introducing convolutional networks</li></a><a href="chap6.html#convolutional_neural_networks_in_practice"><li>Convolutional neural networks in practice</li></a><a href="chap6.html#the_code_for_our_convolutional_networks"><li>The code for our convolutional networks</li></a><a href="chap6.html#recent_progress_in_image_recognition"><li>Recent progress in image recognition</li></a><a href="chap6.html#other_approaches_to_deep_neural_nets"><li>Other approaches to deep neural nets</li></a><a href="chap6.html#on_the_future_of_neural_networks"><li>On the future of neural networks</li></a></ul></p></div>
<script>
$('#toc_deep_learning_reveal').click(function() {
var src = $('#toc_img_deep_learning').attr('src');
if(src == 'images/arrow.png') {
$("#toc_img_deep_learning").attr('src', 'images/arrow_down.png');
} else {
$("#toc_img_deep_learning").attr('src', 'images/arrow.png');
};
$('#toc_deep_learning').toggle('fast', function() {});
});</script><p class="toc_not_mainchapter"><a href="sai.html">Appendix: Is there a <em>simple</em> algorithm for intelligence?</a></p><p class="toc_not_mainchapter"><a href="acknowledgements.html">Acknowledgements</a></p><p class="toc_not_mainchapter"><a href="faq.html">Frequently Asked Questions</a></p>
<!--
<hr>
<p class="sidebar"> If you benefit from the book, please make a small
donation. I suggest $3, but you can choose the amount.</p>
<form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_top">
<input type="hidden" name="cmd" value="_s-xclick">
<input type="hidden" name="encrypted" value="-----BEGIN PKCS7-----MIIHTwYJKoZIhvcNAQcEoIIHQDCCBzwCAQExggEwMIIBLAIBADCBlDCBjjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQKEwtQYXlQYWwgSW5jLjETMBEGA1UECxQKbGl2ZV9jZXJ0czERMA8GA1UEAxQIbGl2ZV9hcGkxHDAaBgkqhkiG9w0BCQEWDXJlQHBheXBhbC5jb20CAQAwDQYJKoZIhvcNAQEBBQAEgYAtusFIFTgWVpgZsMgI9zMrWRAFFKQqeFiE6ay1nbmP360YzPtR+vvCXwn214Az9+F9g7mFxe0L+m9zOCdjzgRROZdTu1oIuS78i0TTbcbD/Vs/U/f9xcmwsdX9KYlhimfsya0ydPQ2xvr4iSGbwfNemIPVRCTadp/Y4OQWWRFKGTELMAkGBSsOAwIaBQAwgcwGCSqGSIb3DQEHATAUBggqhkiG9w0DBwQIK5obVTaqzmyAgajgc4w5t7l6DjTGVI7k+4UyO3uafxPac23jOyBGmxSnVRPONB9I+/Q6OqpXZtn8JpTuzFmuIgkNUf1nldv/DA1mhPOeeVxeuSGL8KpWxpJboKZ0mEu9b+0FJXvZW+snv0jodnRDtI4g0AXDZNPyRWIdJ3m+tlYfsXu4mQAe0q+CyT+QrSRhPGI/llicF4x3rMbRBNqlDze/tFqp/jbgW84Puzz6KyxAez6gggOHMIIDgzCCAuygAwIBAgIBADANBgkqhkiG9w0BAQUFADCBjjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQKEwtQYXlQYWwgSW5jLjETMBEGA1UECxQKbGl2ZV9jZXJ0czERMA8GA1UEAxQIbGl2ZV9hcGkxHDAaBgkqhkiG9w0BCQEWDXJlQHBheXBhbC5jb20wHhcNMDQwMjEzMTAxMzE1WhcNMzUwMjEzMTAxMzE1WjCBjjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQKEwtQYXlQYWwgSW5jLjETMBEGA1UECxQKbGl2ZV9jZXJ0czERMA8GA1UEAxQIbGl2ZV9hcGkxHDAaBgkqhkiG9w0BCQEWDXJlQHBheXBhbC5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMFHTt38RMxLXJyO2SmS+Ndl72T7oKJ4u4uw+6awntALWh03PewmIJuzbALScsTS4sZoS1fKciBGoh11gIfHzylvkdNe/hJl66/RGqrj5rFb08sAABNTzDTiqqNpJeBsYs/c2aiGozptX2RlnBktH+SUNpAajW724Nv2Wvhif6sFAgMBAAGjge4wgeswHQYDVR0OBBYEFJaffLvGbxe9WT9S1wob7BDWZJRrMIG7BgNVHSMEgbMwgbCAFJaffLvGbxe9WT9S1wob7BDWZJRroYGUpIGRMIGOMQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDU1vdW50YWluIFZpZXcxFDASBgNVBAoTC1BheVBhbCBJbmMuMRMwEQYDVQQLFApsaXZlX2NlcnRzMREwDwYDVQQDFAhsaXZlX2FwaTEcMBoGCSqGSIb3DQEJARYNcmVAcGF5cGFsLmNvbYIBADAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4GBAIFfOlaagFrl71+jq6OKidbWFSE+Q4FqROvdgIONth+8kSK//Y/4ihuE4Ymvzn5ceE3S/iBSQQMjyvb+s2TWbQYDwcp129OPIbD9epdr4tJOUNiSojw7BHwYRiPh58S1xGlFgHFXwrEBb3dgNbMUa+u4qectsMAXpVHnD9wIyfmHMYIBmjCCAZYCAQEwgZQwgY4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEWMBQGA1UEBxMNTW91bnRhaW4gVmlldzEUMBIGA1UEChMLUGF5UGFsIEluYy4xEzARBgNVBAsUCmxpdmVfY2VydHMxETAPBgNVBAMUCGxpdmVfYXBpMRwwGgYJKoZIhvcNAQkBFg1yZUBwYXlwYWwuY29tAgEAMAkGBSsOAwIaBQCgXTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTA4MDUxMzMyMTRaMCMGCSqGSIb3DQEJBDEWBBRtGLYvbZ45sWVegWVP2CuXTHPmJTANBgkqhkiG9w0BAQEFAASBgKgrMHMINfV7yVuZgcTjp8gUzejPF2x2zRPU/G8pKUvYIl1F38TjV2pe4w0QXcGMJRT8mQfxHCy9UmF3LfblH8F0NSMMDrZqu3M0eLk96old+L0Xl6ING8l3idFDkLagE+lZK4A0rNV35aMci3VLvjQ34CvEj7jaHeLpbkgk/l6v-----END PKCS7-----
">
<input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
<img alt="" border="0" src="https://www.paypalobjects.com/en_US/i/scr/pixel.gif" width="1" height="1">
</form>
-->
<hr>
<span class="sidebar_title">Sponsors</span>
<br/>
<a href='http://www.ersatz1.com/'><img src='assets/ersatz.png' width='140px' style="padding: 0px 0px 10px 8px; border-style: none;"></a>
<a href='http://gsquaredcapital.com/'><img src='assets/gsquared.png' width='150px' style="padding: 0px 0px 10px 10px; border-style: none;"></a>
<a href='http://www.tineye.com'><img src='assets/tineye.png' width='150px'
style="padding: 0px 0px 10px 8px; border-style: none;"></a>
<a href='http://www.visionsmarts.com'><img
src='assets/visionsmarts.png' width='160px' style="padding: 0px 0px
0px 0px; border-style: none;"></a> <br/>
<p class="sidebar">Thanks to all the <a
href="supporters.html">supporters</a> who made the book possible, with
especial thanks to Pavel Dudrenov. Thanks also to all the
contributors to the <a href="bugfinder.html">Bugfinder Hall of
Fame</a>. </p>
<hr>
<span class="sidebar_title">Resources</span>
<p class="sidebar"><a href="https://twitter.com/michael_nielsen">Michael Nielsen on Twitter</a></p>
<p class="sidebar"><a href="faq.html">Book FAQ</a></p>
<p class="sidebar">
<a href="https://github.com/mnielsen/neural-networks-and-deep-learning">Code repository</a></p>
<p class="sidebar">
<a href="http://eepurl.com/0Xxjb">Michael Nielsen's project announcement mailing list</a>
</p>
<p class="sidebar"> <a href="http://www.deeplearningbook.org/">Deep Learning</a>, book by Ian
Goodfellow, Yoshua Bengio, and Aaron Courville</p>
<p class="sidebar"><a href="http://cognitivemedium.com">cognitivemedium.com</a></p>
<hr>
<a href="http://michaelnielsen.org"><img src="assets/Michael_Nielsen_Web_Small.jpg" width="160px" style="border-style: none;"/></a>
<p class="sidebar">
By <a href="http://michaelnielsen.org">Michael Nielsen</a> / Jan 2017
</p>
</div>
</p><p>In the <a href="chap1.html">last chapter</a> we saw how neural networks canlearn their weights and biases using the gradient descent algorithm.There was, however, a gap in our explanation: we didn't discuss how tocompute the gradient of the cost function. That's quite a gap! Inthis chapter I'll explain a fast algorithm for computing suchgradients, an algorithm known as <em>backpropagation</em>. </p><p>The backpropagation algorithm was originally introduced in the 1970s,but its importance wasn't fully appreciated until a<a href="http://www.nature.com/nature/journal/v323/n6088/pdf/323533a0.pdf">famous 1986 paper</a> by<a href="http://en.wikipedia.org/wiki/David_Rumelhart">David Rumelhart</a>,<a href="http://www.cs.toronto.edu/~hinton/">Geoffrey Hinton</a>, and<a href="http://en.wikipedia.org/wiki/Ronald_J._Williams">Ronald Williams</a>. That paper describes severalneural networks where backpropagation works far faster than earlierapproaches to learning, making it possible to use neural nets to solveproblems which had previously been insoluble. Today, thebackpropagation algorithm is the workhorse of learning in neuralnetworks.</p><p>This chapter is more mathematically involved than the rest of thebook. If you're not crazy about mathematics you may be tempted toskip the chapter, and to treat backpropagation as a black box whosedetails you're willing to ignore. Why take the time to study thosedetails?</p><p>The reason, of course, is understanding. At the heart ofbackpropagation is an expression for the partial derivative $\partialC / \partial w$ of the cost function $C$ with respect to any weight$w$ (or bias $b$) in the network. The expression tells us how quicklythe cost changes when we change the weights and biases. And while theexpression is somewhat complex, it also has a beauty to it, with eachelement having a natural, intuitive interpretation. And sobackpropagation isn't just a fast algorithm for learning. It actuallygives us detailed insights into how changing the weights and biaseschanges the overall behaviour of the network. That's well worthstudying in detail.</p><p>With that said, if you want to skim the chapter, or jump straight tothe next chapter, that's fine. I've written the rest of the book tobe accessible even if you treat backpropagation as a black box. Thereare, of course, points later in the book where I refer back to resultsfrom this chapter. But at those points you should still be able tounderstand the main conclusions, even if you don't follow all thereasoning.</p><p><h3><a name="warm_up_a_fast_matrix-based_approach_to_computing_the_output_from_a_neural_network"></a><a href="#warm_up_a_fast_matrix-based_approach_to_computing_the_output_from_a_neural_network">Warm up: a fast matrix-based approach to computing the output from a neural network</a></h3></p><p>Before discussing backpropagation, let's warm up with a fastmatrix-based algorithm to compute the output from a neural network.We actually already briefly saw this algorithm<a href="chap1.html#implementing_our_network_to_classify_digits">near the end of the last chapter</a>, but I described it quickly, so it'sworth revisiting in detail. In particular, this is a good way ofgetting comfortable with the notation used in backpropagation, in afamiliar context.</p><p>Let's begin with a notation which lets us refer to weights in thenetwork in an unambiguous way. We'll use $w^l_{jk}$ to denote theweight for the connection from the $k^{\rm th}$ neuron in the$(l-1)^{\rm th}$ layer to the $j^{\rm th}$ neuron in the $l^{\rm th}$layer. So, for example, the diagram below shows the weight on aconnection from the fourth neuron in the second layer to the secondneuron in the third layer of a network:<center><img src="images/tikz16.png"/></center>This notation is cumbersome at first, and it does take some work tomaster. But with a little effort you'll find the notation becomeseasy and natural. One quirk of the notation is the ordering of the$j$ and $k$ indices. You might think that it makes more sense to use$j$ to refer to the input neuron, and $k$ to the output neuron, notvice versa, as is actually done. I'll explain the reason for thisquirk below.</p><p>We use a similar notation for the network's biases and activations.Explicitly, we use $b^l_j$ for the bias of the $j^{\rm th}$ neuron inthe $l^{\rm th}$ layer. And we use $a^l_j$ for the activation of the$j^{\rm th}$ neuron in the $l^{\rm th}$ layer. The following diagramshows examples of these notations in use:<center><img src="images/tikz17.png"/></center>With these notations, the activation $a^{l}_j$ of the $j^{\rm th}$neuron in the $l^{\rm th}$ layer is related to the activations in the$(l-1)^{\rm th}$ layer by the equation (compareEquation <span id="margin_788574725314_reveal" class="equation_link">(4)</span><span id="margin_788574725314" class="marginequation" style="display: none;"><a href="chap1.html#eqtn4" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{1}{1+\exp(-\sum_j w_j x_j-b)} \nonumber\end{eqnarray}</a></span><script>$('#margin_788574725314_reveal').click(function() {$('#margin_788574725314').toggle('slow', function() {});});</script> and surroundingdiscussion in the last chapter)<a class="displaced_anchor" name="eqtn23"></a>\begin{eqnarray} a^{l}_j = \sigma\left( \sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right),\tag{23}\end{eqnarray}where the sum is over all neurons $k$ in the $(l-1)^{\rm th}$ layer. Torewrite this expression in a matrix form we define a <em>weight matrix</em> $w^l$ for each layer, $l$. The entries of the weight matrix$w^l$ are just the weights connecting to the $l^{\rm th}$ layer of neurons,that is, the entry in the $j^{\rm th}$ row and $k^{\rm th}$ column is $w^l_{jk}$.Similarly, for each layer $l$ we define a <em>bias vector</em>, $b^l$.You can probably guess how this works - the components of the biasvector are just the values $b^l_j$, one component for each neuron inthe $l^{\rm th}$ layer. And finally, we define an activation vector $a^l$whose components are the activations $a^l_j$.</p><p>The last ingredient we need to rewrite <span id="margin_735534017585_reveal" class="equation_link">(23)</span><span id="margin_735534017585" class="marginequation" style="display: none;"><a href="chap2.html#eqtn23" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} a^{l}_j = \sigma\left( \sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right) \nonumber\end{eqnarray}</a></span><script>$('#margin_735534017585_reveal').click(function() {$('#margin_735534017585').toggle('slow', function() {});});</script> in amatrix form is the idea of vectorizing a function such as $\sigma$.We met vectorization briefly in the last chapter, but to recap, theidea is that we want to apply a function such as $\sigma$ to everyelement in a vector $v$. We use the obvious notation $\sigma(v)$ todenote this kind of elementwise application of a function. That is,the components of $\sigma(v)$ are just $\sigma(v)_j = \sigma(v_j)$.As an example, if we have the function $f(x) = x^2$ then thevectorized form of $f$ has the effect<a class="displaced_anchor" name="eqtn24"></a>\begin{eqnarray} f\left(\left[ \begin{array}{c} 2 \\ 3 \end{array} \right] \right) = \left[ \begin{array}{c} f(2) \\ f(3) \end{array} \right] = \left[ \begin{array}{c} 4 \\ 9 \end{array} \right],\tag{24}\end{eqnarray}that is, the vectorized $f$ just squares every element of the vector.</p><p>With these notations in mind, Equation <span id="margin_114817738681_reveal" class="equation_link">(23)</span><span id="margin_114817738681" class="marginequation" style="display: none;"><a href="chap2.html#eqtn23" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} a^{l}_j = \sigma\left( \sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right) \nonumber\end{eqnarray}</a></span><script>$('#margin_114817738681_reveal').click(function() {$('#margin_114817738681').toggle('slow', function() {});});</script> canbe rewritten in the beautiful and compact vectorized form<a class="displaced_anchor" name="eqtn25"></a>\begin{eqnarray} a^{l} = \sigma(w^l a^{l-1}+b^l).\tag{25}\end{eqnarray}This expression gives us a much more global way of thinking about howthe activations in one layer relate to activations in the previouslayer: we just apply the weight matrix to the activations, then addthe bias vector, and finally apply the $\sigma$ function<a id="quirk"></a>*<span class="marginnote">*By the way, it's this expression that motivates the quirk in the $w^l_{jk}$ notation mentioned earlier. If we used $j$ to index the input neuron, and $k$ to index the output neuron, then we'd need to replace the weight matrix in Equation <span id="margin_984031576396_reveal" class="equation_link">(25)</span><span id="margin_984031576396" class="marginequation" style="display: none;"><a href="chap2.html#eqtn25" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} a^{l} = \sigma(w^l a^{l-1}+b^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_984031576396_reveal').click(function() {$('#margin_984031576396').toggle('slow', function() {});});</script> by the transpose of the weight matrix. That's a small change, but annoying, and we'd lose the easy simplicity of saying (and thinking) "apply the weight matrix to the activations".</span>. That global view is often easier andmore succinct (and involves fewer indices!) than the neuron-by-neuronview we've taken to now. Think of it as a way of escaping index hell,while remaining precise about what's going on. The expression is alsouseful in practice, because most matrix libraries provide fast ways ofimplementing matrix multiplication, vector addition, andvectorization. Indeed, the<a href="chap1.html#implementing_our_network_to_classify_digits">code</a>in the last chapter made implicit use of this expression to computethe behaviour of the network.</p><p>When using Equation <span id="margin_709882465331_reveal" class="equation_link">(25)</span><span id="margin_709882465331" class="marginequation" style="display: none;"><a href="chap2.html#eqtn25" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} a^{l} = \sigma(w^l a^{l-1}+b^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_709882465331_reveal').click(function() {$('#margin_709882465331').toggle('slow', function() {});});</script> to compute $a^l$,we compute the intermediate quantity $z^l \equiv w^l a^{l-1}+b^l$along the way. This quantity turns out to be useful enough to beworth naming: we call $z^l$ the <em>weighted input</em> to the neuronsin layer $l$. We'll make considerable use of the weighted input $z^l$later in the chapter. Equation <span id="margin_383003775830_reveal" class="equation_link">(25)</span><span id="margin_383003775830" class="marginequation" style="display: none;"><a href="chap2.html#eqtn25" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} a^{l} = \sigma(w^l a^{l-1}+b^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_383003775830_reveal').click(function() {$('#margin_383003775830').toggle('slow', function() {});});</script> issometimes written in terms of the weighted input, as $a^l =\sigma(z^l)$. It's also worth noting that $z^l$ has components $z^l_j= \sum_k w^l_{jk} a^{l-1}_k+b^l_j$, that is, $z^l_j$ is just theweighted input to the activation function for neuron $j$ in layer $l$.</p><p><h3><a name="the_two_assumptions_we_need_about_the_cost_function"></a><a href="#the_two_assumptions_we_need_about_the_cost_function">The two assumptions we need about the cost function</a></h3></p><p>The goal of backpropagation is to compute the partial derivatives$\partial C / \partial w$ and $\partial C / \partial b$ of the costfunction $C$ with respect to any weight $w$ or bias $b$ in thenetwork. For backpropagation to work we need to make two mainassumptions about the form of the cost function. Before stating thoseassumptions, though, it's useful to have an example cost function inmind. We'll use the quadratic cost function from last chapter(c.f. Equation <span id="margin_59913254684_reveal" class="equation_link">(6)</span><span id="margin_59913254684" class="marginequation" style="display: none;"><a href="chap1.html#eqtn6" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} C(w,b) \equiv \frac{1}{2n} \sum_x \| y(x) - a\|^2 \nonumber\end{eqnarray}</a></span><script>$('#margin_59913254684_reveal').click(function() {$('#margin_59913254684').toggle('slow', function() {});});</script>). In the notation ofthe last section, the quadratic cost has the form<a class="displaced_anchor" name="eqtn26"></a>\begin{eqnarray} C = \frac{1}{2n} \sum_x \|y(x)-a^L(x)\|^2,\tag{26}\end{eqnarray}where: $n$ is the total number of training examples; the sum is overindividual training examples, $x$; $y = y(x)$ is the correspondingdesired output; $L$ denotes the number of layers in the network; and$a^L = a^L(x)$ is the vector of activations output from the networkwhen $x$ is input.</p><p>Okay, so what assumptions do we need to make about our cost function,$C$, in order that backpropagation can be applied? The firstassumption we need is that the cost function can be written as anaverage $C = \frac{1}{n} \sum_x C_x$ over cost functions $C_x$ forindividual training examples, $x$. This is the case for the quadraticcost function, where the cost for a single training example is $C_x =\frac{1}{2} \|y-a^L \|^2$. This assumption will also hold true forall the other cost functions we'll meet in this book.</p><p>The reason we need this assumption is because what backpropagationactually lets us do is compute the partial derivatives $\partial C_x/ \partial w$ and $\partial C_x / \partial b$ for a single trainingexample. We then recover $\partial C / \partial w$ and $\partial C/ \partial b$ by averaging over training examples. In fact, with thisassumption in mind, we'll suppose the training example $x$ has beenfixed, and drop the $x$ subscript, writing the cost $C_x$ as $C$.We'll eventually put the $x$ back in, but for now it's a notationalnuisance that is better left implicit.</p><p>The second assumption we make about the cost is that it can be writtenas a function of the outputs from the neural network:<center><img src="images/tikz18.png"/></center>For example, the quadratic cost function satisfies this requirement,since the quadratic cost for a single training example $x$ may bewritten as<a class="displaced_anchor" name="eqtn27"></a>\begin{eqnarray} C = \frac{1}{2} \|y-a^L\|^2 = \frac{1}{2} \sum_j (y_j-a^L_j)^2,\tag{27}\end{eqnarray}and thus is a function of the output activations. Of course, thiscost function also depends on the desired output $y$, and you maywonder why we're not regarding the cost also as a function of $y$.Remember, though, that the input training example $x$ is fixed, and sothe output $y$ is also a fixed parameter. In particular, it's notsomething we can modify by changing the weights and biases in any way,i.e., it's not something which the neural network learns. And so itmakes sense to regard $C$ as a function of the output activations$a^L$ alone, with $y$ merely a parameter that helps define thatfunction.</p><p></p><p></p><p></p><p><h3><a name="the_hadamard_product_$s_\odot_t$"></a><a href="#the_hadamard_product_$s_\odot_t$">The Hadamard product, $s \odot t$</a></h3></p><p>The backpropagation algorithm is based on common linear algebraicoperations - things like vector addition, multiplying a vector by amatrix, and so on. But one of the operations is a little lesscommonly used. In particular, suppose $s$ and $t$ are two vectors ofthe same dimension. Then we use $s \odot t$ to denote the<em>elementwise</em> product of the two vectors. Thus the components of$s \odot t$ are just $(s \odot t)_j = s_j t_j$. As an example,<a class="displaced_anchor" name="eqtn28"></a>\begin{eqnarray}\left[\begin{array}{c} 1 \\ 2 \end{array}\right] \odot \left[\begin{array}{c} 3 \\ 4\end{array} \right]= \left[ \begin{array}{c} 1 * 3 \\ 2 * 4 \end{array} \right]= \left[ \begin{array}{c} 3 \\ 8 \end{array} \right].\tag{28}\end{eqnarray}This kind of elementwise multiplication is sometimes called the<em>Hadamard product</em> or <em>Schur product</em>. We'll refer to it asthe Hadamard product. Good matrix libraries usually provide fastimplementations of the Hadamard product, and that comes in handy whenimplementing backpropagation.</p><p><h3><a name="the_four_fundamental_equations_behind_backpropagation"></a><a href="#the_four_fundamental_equations_behind_backpropagation">The four fundamental equations behind backpropagation</a></h3></p><p>Backpropagation is about understanding how changing the weights andbiases in a network changes the cost function. Ultimately, this meanscomputing the partial derivatives $\partial C / \partial w^l_{jk}$ and$\partial C / \partial b^l_j$. But to compute those, we firstintroduce an intermediate quantity, $\delta^l_j$, which we call the<em>error</em> in the $j^{\rm th}$ neuron in the $l^{\rm th}$ layer.Backpropagation will give us a procedure to compute the error$\delta^l_j$, and then will relate $\delta^l_j$ to $\partial C/ \partial w^l_{jk}$ and $\partial C / \partial b^l_j$.</p><p>To understand how the error is defined, imagine there is a demon inour neural network:<center><img src="images/tikz19.png"/></center>The demon sits at the $j^{\rm th}$ neuron in layer $l$. As the input to theneuron comes in, the demon messes with the neuron's operation. Itadds a little change $\Delta z^l_j$ to the neuron's weighted input, sothat instead of outputting $\sigma(z^l_j)$, the neuron instead outputs$\sigma(z^l_j+\Delta z^l_j)$. This change propagates through laterlayers in the network, finally causing the overall cost to change byan amount $\frac{\partial C}{\partial z^l_j} \Delta z^l_j$.</p><p>Now, this demon is a good demon, and is trying to help you improve thecost, i.e., they're trying to find a $\Delta z^l_j$ which makes thecost smaller. Suppose $\frac{\partial C}{\partial z^l_j}$ has a largevalue (either positive or negative). Then the demon can lower thecost quite a bit by choosing $\Delta z^l_j$ to have the opposite signto $\frac{\partial C}{\partial z^l_j}$. By contrast, if$\frac{\partial C}{\partial z^l_j}$ is close to zero, then the demoncan't improve the cost much at all by perturbing the weighted input$z^l_j$. So far as the demon can tell, the neuron is already prettynear optimal*<span class="marginnote">*This is only the case for small changes $\Delta z^l_j$, of course. We'll assume that the demon is constrained to make such small changes.</span>. And so there's a heuristic sense inwhich $\frac{\partial C}{\partial z^l_j}$ is a measure of the error inthe neuron.</p><p>Motivated by this story, we define the error $\delta^l_j$ of neuron$j$ in layer $l$ by<a class="displaced_anchor" name="eqtn29"></a>\begin{eqnarray} \delta^l_j \equiv \frac{\partial C}{\partial z^l_j}.\tag{29}\end{eqnarray}As per our usual conventions, we use $\delta^l$ to denote the vectorof errors associated with layer $l$. Backpropagation will give us away of computing $\delta^l$ for every layer, and then relating thoseerrors to the quantities of real interest, $\partial C / \partialw^l_{jk}$ and $\partial C / \partial b^l_j$.</p><p>You might wonder why the demon is changing the weighted input $z^l_j$.Surely it'd be more natural to imagine the demon changing the outputactivation $a^l_j$, with the result that we'd be using $\frac{\partial C}{\partial a^l_j}$ as our measure of error. In fact, if you dothis things work out quite similarly to the discussion below. But itturns out to make the presentation of backpropagation a little morealgebraically complicated. So we'll stick with $\delta^l_j =\frac{\partial C}{\partial z^l_j}$ as our measure of error*<span class="marginnote">*In classification problems like MNIST the term "error" is sometimes used to mean the classification failure rate. E.g., if the neural net correctly classifies 96.0 percent of the digits, then the error is 4.0 percent. Obviously, this has quite a different meaning from our $\delta$ vectors. In practice, you shouldn't have trouble telling which meaning is intended in any given usage.</span>.</p><p><strong>Plan of attack:</strong> Backpropagation is based around fourfundamental equations. Together, those equations give us a way ofcomputing both the error $\delta^l$ and the gradient of the costfunction. I state the four equations below. Be warned, though: youshouldn't expect to instantaneously assimilate the equations. Such anexpectation will lead to disappointment. In fact, the backpropagationequations are so rich that understanding them well requiresconsiderable time and patience as you gradually delve deeper into theequations. The good news is that such patience is repaid many timesover. And so the discussion in this section is merely a beginning,helping you on the way to a thorough understanding of the equations.</p><p>Here's a preview of the ways we'll delve more deeply into theequations later in the chapter: I'll<a href="chap2.html#proof_of_the_four_fundamental_equations_(optional)">give a short proof of the equations</a>, which helps explain why they aretrue; we'll <a href="chap2.html#the_backpropagation_algorithm">restate the equations</a> in algorithmic form as pseudocode, and<a href="chap2.html#the_code_for_backpropagation">see how</a> thepseudocode can be implemented as real, running Python code; and, in<a href="chap2.html#backpropagation_the_big_picture">the final section of the chapter</a>, we'll develop an intuitive picture of whatthe backpropagation equations mean, and how someone might discoverthem from scratch. Along the way we'll return repeatedly to the fourfundamental equations, and as you deepen your understanding thoseequations will come to seem comfortable and, perhaps, even beautifuland natural.</p><p><strong>An equation for the error in the output layer, $\delta^L$:</strong>The components of $\delta^L$ are given by<a class="displaced_anchor" name="eqtnBP1"></a>\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j).\tag{BP1}\end{eqnarray}This is a very natural expression. The first term on the right,$\partial C / \partial a^L_j$, just measures how fast the cost ischanging as a function of the $j^{\rm th}$ output activation. If, forexample, $C$ doesn't depend much on a particular output neuron, $j$,then $\delta^L_j$ will be small, which is what we'd expect. Thesecond term on the right, $\sigma'(z^L_j)$, measures how fast theactivation function $\sigma$ is changing at $z^L_j$.</p><p>Notice that everything in <span id="margin_919611304299_reveal" class="equation_link">(BP1)</span><span id="margin_919611304299" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_919611304299_reveal').click(function() {$('#margin_919611304299').toggle('slow', function() {});});</script> is easily computed. Inparticular, we compute $z^L_j$ while computing the behaviour of thenetwork, and it's only a small additional overhead to compute$\sigma'(z^L_j)$. The exact form of $\partial C / \partial a^L_j$will, of course, depend on the form of the cost function. However,provided the cost function is known there should be little troublecomputing $\partial C / \partial a^L_j$. For example, if we're usingthe quadratic cost function then $C = \frac{1}{2} \sum_j(y_j-a^L_j)^2$, and so $\partial C / \partial a^L_j = (a_j^L-y_j)$,which obviously is easily computable.</p><p>Equation <span id="margin_467685671277_reveal" class="equation_link">(BP1)</span><span id="margin_467685671277" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_467685671277_reveal').click(function() {$('#margin_467685671277').toggle('slow', function() {});});</script> is a componentwise expression for $\delta^L$.It's a perfectly good expression, but not the matrix-based form wewant for backpropagation. However, it's easy to rewrite the equationin a matrix-based form, as<a class="displaced_anchor" name="eqtnBP1a"></a>\begin{eqnarray} \delta^L = \nabla_a C \odot \sigma'(z^L).\tag{BP1a}\end{eqnarray}Here, $\nabla_a C$ is defined to be a vector whose components are thepartial derivatives $\partial C / \partial a^L_j$. You can think of$\nabla_a C$ as expressing the rate of change of $C$ with respect tothe output activations. It's easy to see that Equations <span id="margin_570480201049_reveal" class="equation_link">(BP1a)</span><span id="margin_570480201049" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1a" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L = \nabla_a C \odot \sigma'(z^L) \nonumber\end{eqnarray}</a></span><script>$('#margin_570480201049_reveal').click(function() {$('#margin_570480201049').toggle('slow', function() {});});</script>and <span id="margin_983733590680_reveal" class="equation_link">(BP1)</span><span id="margin_983733590680" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_983733590680_reveal').click(function() {$('#margin_983733590680').toggle('slow', function() {});});</script> are equivalent, and for that reason from now on we'lluse <span id="margin_506574502658_reveal" class="equation_link">(BP1)</span><span id="margin_506574502658" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_506574502658_reveal').click(function() {$('#margin_506574502658').toggle('slow', function() {});});</script> interchangeably to refer to both equations. As anexample, in the case of the quadratic cost we have $\nabla_a C =(a^L-y)$, and so the fully matrix-based form of <span id="margin_365533411234_reveal" class="equation_link">(BP1)</span><span id="margin_365533411234" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_365533411234_reveal').click(function() {$('#margin_365533411234').toggle('slow', function() {});});</script> becomes<a class="displaced_anchor" name="eqtn30"></a>\begin{eqnarray} \delta^L = (a^L-y) \odot \sigma'(z^L).\tag{30}\end{eqnarray}As you can see, everything in this expression has a nice vector form,and is easily computed using a library such as Numpy.</p><p><strong>An equation for the error $\delta^l$ in terms of the error in the next layer, $\delta^{l+1}$:</strong> In particular<a class="displaced_anchor" name="eqtnBP2"></a>\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l),\tag{BP2}\end{eqnarray}where $(w^{l+1})^T$ is the transpose of the weight matrix $w^{l+1}$ forthe $(l+1)^{\rm th}$ layer. This equation appears complicated, buteach element has a nice interpretation. Suppose we know the error$\delta^{l+1}$ at the $l+1^{\rm th}$ layer. When we apply thetranspose weight matrix, $(w^{l+1})^T$, we can think intuitively ofthis as moving the error <em>backward</em> through the network, givingus some sort of measure of the error at the output of the $l^{\rm th}$layer. We then take the Hadamard product $\odot \sigma'(z^l)$. Thismoves the error backward through the activation function in layer $l$,giving us the error $\delta^l$ in the weighted input to layer $l$.</p><p>By combining <span id="margin_520088358076_reveal" class="equation_link">(BP2)</span><span id="margin_520088358076" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_520088358076_reveal').click(function() {$('#margin_520088358076').toggle('slow', function() {});});</script> with <span id="margin_853208206649_reveal" class="equation_link">(BP1)</span><span id="margin_853208206649" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_853208206649_reveal').click(function() {$('#margin_853208206649').toggle('slow', function() {});});</script> we can compute the error$\delta^l$ for any layer in the network. We start byusing <span id="margin_569252953000_reveal" class="equation_link">(BP1)</span><span id="margin_569252953000" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_569252953000_reveal').click(function() {$('#margin_569252953000').toggle('slow', function() {});});</script> to compute $\delta^L$, then applyEquation <span id="margin_646471794768_reveal" class="equation_link">(BP2)</span><span id="margin_646471794768" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_646471794768_reveal').click(function() {$('#margin_646471794768').toggle('slow', function() {});});</script> to compute $\delta^{L-1}$, thenEquation <span id="margin_58635688739_reveal" class="equation_link">(BP2)</span><span id="margin_58635688739" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_58635688739_reveal').click(function() {$('#margin_58635688739').toggle('slow', function() {});});</script> again to compute $\delta^{L-2}$, and so on, allthe way back through the network.</p><p><strong>An equation for the rate of change of the cost with respect to any bias in the network:</strong> In particular:<a class="displaced_anchor" name="eqtnBP3"></a>\begin{eqnarray} \frac{\partial C}{\partial b^l_j} = \delta^l_j.\tag{BP3}\end{eqnarray}That is, the error $\delta^l_j$ is <em>exactly equal</em> to the rate ofchange $\partial C / \partial b^l_j$. This is great news, since<span id="margin_8280399999_reveal" class="equation_link">(BP1)</span><span id="margin_8280399999" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_8280399999_reveal').click(function() {$('#margin_8280399999').toggle('slow', function() {});});</script> and <span id="margin_557075118630_reveal" class="equation_link">(BP2)</span><span id="margin_557075118630" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_557075118630_reveal').click(function() {$('#margin_557075118630').toggle('slow', function() {});});</script> have already told us how to compute$\delta^l_j$. We can rewrite <span id="margin_205000449729_reveal" class="equation_link">(BP3)</span><span id="margin_205000449729" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP3" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial b^l_j} = \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_205000449729_reveal').click(function() {$('#margin_205000449729').toggle('slow', function() {});});</script> in shorthand as<a class="displaced_anchor" name="eqtn31"></a>\begin{eqnarray} \frac{\partial C}{\partial b} = \delta,\tag{31}\end{eqnarray}where it is understood that $\delta$ is being evaluated at the sameneuron as the bias $b$.</p><p><strong>An equation for the rate of change of the cost with respect to any weight in the network:</strong> In particular:<a class="displaced_anchor" name="eqtnBP4"></a>\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j.\tag{BP4}\end{eqnarray}This tells us how to compute the partial derivatives $\partial C/ \partial w^l_{jk}$ in terms of the quantities $\delta^l$ and$a^{l-1}$, which we already know how to compute. The equation can berewritten in a less index-heavy notation as<a class="displaced_anchor" name="eqtn32"></a>\begin{eqnarray} \frac{\partial C}{\partial w} = a_{\rm in} \delta_{\rm out},\tag{32}\end{eqnarray}where it's understood that $a_{\rm in}$ is the activation of theneuron input to the weight $w$, and $\delta_{\rm out}$ is the error ofthe neuron output from the weight $w$. Zooming in to look at just theweight $w$, and the two neurons connected by that weight, we candepict this as:<center><img src="images/tikz20.png"/></center>A nice consequence of Equation <span id="margin_988156473660_reveal" class="equation_link">(32)</span><span id="margin_988156473660" class="marginequation" style="display: none;"><a href="chap2.html#eqtn32" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w} = a_{\rm in} \delta_{\rm out} \nonumber\end{eqnarray}</a></span><script>$('#margin_988156473660_reveal').click(function() {$('#margin_988156473660').toggle('slow', function() {});});</script> isthat when the activation $a_{\rm in}$ is small, $a_{\rm in} \approx0$, the gradient term $\partial C / \partial w$ will also tend to besmall. In this case, we'll say the weight <em>learns slowly</em>,meaning that it's not changing much during gradient descent. In otherwords, one consequence of <span id="margin_30338286679_reveal" class="equation_link">(BP4)</span><span id="margin_30338286679" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP4" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_30338286679_reveal').click(function() {$('#margin_30338286679').toggle('slow', function() {});});</script> is that weights output fromlow-activation neurons learn slowly.</p><p><a name="saturation"></a></p><p>There are other insights along these lines which can be obtainedfrom <span id="margin_250484114753_reveal" class="equation_link">(BP1)</span><span id="margin_250484114753" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_250484114753_reveal').click(function() {$('#margin_250484114753').toggle('slow', function() {});});</script>-<span id="margin_282300343827_reveal" class="equation_link">(BP4)</span><span id="margin_282300343827" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP4" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_282300343827_reveal').click(function() {$('#margin_282300343827').toggle('slow', function() {});});</script>. Let's start by looking at the outputlayer. Consider the term $\sigma'(z^L_j)$ in <span id="margin_16544494861_reveal" class="equation_link">(BP1)</span><span id="margin_16544494861" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_16544494861_reveal').click(function() {$('#margin_16544494861').toggle('slow', function() {});});</script>. Recallfrom the <a href="chap1.html#sigmoid_graph">graph of the sigmoid function in the last chapter</a> that the $\sigma$ function becomesvery flat when $\sigma(z^L_j)$ is approximately $0$ or $1$. When thisoccurs we will have $\sigma'(z^L_j) \approx 0$. And so the lesson isthat a weight in the final layer will learn slowly if the outputneuron is either low activation ($\approx 0$) or high activation($\approx 1$). In this case it's common to say the output neuron has<em>saturated</em> and, as a result, the weight has stopped learning (oris learning slowly). Similar remarks hold also for the biases ofoutput neuron.</p><p>We can obtain similar insights for earlier layers. In particular,note the $\sigma'(z^l)$ term in <span id="margin_998669428161_reveal" class="equation_link">(BP2)</span><span id="margin_998669428161" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_998669428161_reveal').click(function() {$('#margin_998669428161').toggle('slow', function() {});});</script>. This means that$\delta^l_j$ is likely to get small if the neuron is near saturation.And this, in turn, means that any weights input to a saturated neuronwill learn slowly*<span class="marginnote">*This reasoning won't hold if ${w^{l+1}}^T \delta^{l+1}$ has large enough entries to compensate for the smallness of $\sigma'(z^l_j)$. But I'm speaking of the general tendency.</span>.</p><p>Summing up, we've learnt that a weight will learn slowly if either theinput neuron is low-activation, or if the output neuron has saturated,i.e., is either high- or low-activation. </p><p>None of these observations is too greatly surprising. Still, theyhelp improve our mental model of what's going on as a neural networklearns. Furthermore, we can turn this type of reasoning around. Thefour fundamental equations turn out to hold for any activationfunction, not just the standard sigmoid function (that's because, aswe'll see in a moment, the proofs don't use any special properties of$\sigma$). And so we can use these equations to <em>design</em>activation functions which have particular desired learningproperties. As an example to give you the idea, suppose we were tochoose a (non-sigmoid) activation function $\sigma$ so that $\sigma'$is always positive, and never gets close to zero. That would preventthe slow-down of learning that occurs when ordinary sigmoid neuronssaturate. Later in the book we'll see examples where this kind ofmodification is made to the activation function. Keeping the fourequations <span id="margin_678414648914_reveal" class="equation_link">(BP1)</span><span id="margin_678414648914" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_678414648914_reveal').click(function() {$('#margin_678414648914').toggle('slow', function() {});});</script>-<span id="margin_936017178156_reveal" class="equation_link">(BP4)</span><span id="margin_936017178156" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP4" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_936017178156_reveal').click(function() {$('#margin_936017178156').toggle('slow', function() {});});</script> in mind can help explain why suchmodifications are tried, and what impact they can have.</p><p><a name="backpropsummary"></a></p><p><center><img src="images/tikz21.png"/></center></p><p><a id="alternative_backprop"></a></p><p><h4><a name="problem_567109"></a><a href="#problem_567109">Problem</a></h4><ul><li><strong>Alternate presentation of the equations of backpropagation:</strong> I've stated the equations of backpropagation (notably <span id="margin_857011554527_reveal" class="equation_link">(BP1)</span><span id="margin_857011554527" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_857011554527_reveal').click(function() {$('#margin_857011554527').toggle('slow', function() {});});</script> and <span id="margin_446828221170_reveal" class="equation_link">(BP2)</span><span id="margin_446828221170" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_446828221170_reveal').click(function() {$('#margin_446828221170').toggle('slow', function() {});});</script>) using the Hadamard product. This presentation may be disconcerting if you're unused to the Hadamard product. There's an alternative approach, based on conventional matrix multiplication, which some readers may find enlightening. (1) Show that <span id="margin_273825325358_reveal" class="equation_link">(BP1)</span><span id="margin_273825325358" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_273825325358_reveal').click(function() {$('#margin_273825325358').toggle('slow', function() {});});</script> may be rewritten as <a class="displaced_anchor" name="eqtn33"></a>\begin{eqnarray} \delta^L = \Sigma'(z^L) \nabla_a C, \tag{33}\end{eqnarray} where $\Sigma'(z^L)$ is a square matrix whose diagonal entries are the values $\sigma'(z^L_j)$, and whose off-diagonal entries are zero. Note that this matrix acts on $\nabla_a C$ by conventional matrix multiplication. (2) Show that <span id="margin_592545063504_reveal" class="equation_link">(BP2)</span><span id="margin_592545063504" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_592545063504_reveal').click(function() {$('#margin_592545063504').toggle('slow', function() {});});</script> may be rewritten as <a class="displaced_anchor" name="eqtn34"></a>\begin{eqnarray} \delta^l = \Sigma'(z^l) (w^{l+1})^T \delta^{l+1}. \tag{34}\end{eqnarray} (3) By combining observations (1) and (2) show that <a class="displaced_anchor" name="eqtn35"></a>\begin{eqnarray} \delta^l = \Sigma'(z^l) (w^{l+1})^T \ldots \Sigma'(z^{L-1}) (w^L)^T \Sigma'(z^L) \nabla_a C \tag{35}\end{eqnarray} For readers comfortable with matrix multiplication this equation may be easier to understand than <span id="margin_211286309191_reveal" class="equation_link">(BP1)</span><span id="margin_211286309191" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_211286309191_reveal').click(function() {$('#margin_211286309191').toggle('slow', function() {});});</script> and <span id="margin_33317062381_reveal" class="equation_link">(BP2)</span><span id="margin_33317062381" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_33317062381_reveal').click(function() {$('#margin_33317062381').toggle('slow', function() {});});</script>. The reason I've focused on <span id="margin_350024952201_reveal" class="equation_link">(BP1)</span><span id="margin_350024952201" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_350024952201_reveal').click(function() {$('#margin_350024952201').toggle('slow', function() {});});</script> and <span id="margin_968208779254_reveal" class="equation_link">(BP2)</span><span id="margin_968208779254" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_968208779254_reveal').click(function() {$('#margin_968208779254').toggle('slow', function() {});});</script> is because that approach turns out to be faster to implement numerically.</ul></p><p><h3><a name="proof_of_the_four_fundamental_equations_(optional)"></a><a href="#proof_of_the_four_fundamental_equations_(optional)">Proof of the four fundamental equations (optional)</a></h3> </p><p>We'll now prove the four fundamentalequations <span id="margin_322535054922_reveal" class="equation_link">(BP1)</span><span id="margin_322535054922" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_322535054922_reveal').click(function() {$('#margin_322535054922').toggle('slow', function() {});});</script>-<span id="margin_58831286370_reveal" class="equation_link">(BP4)</span><span id="margin_58831286370" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP4" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_58831286370_reveal').click(function() {$('#margin_58831286370').toggle('slow', function() {});});</script>. All four are consequences of thechain rule from multivariable calculus. If you're comfortable withthe chain rule, then I strongly encourage you to attempt thederivation yourself before reading on.</p><p>Let's begin with Equation <span id="margin_669494561666_reveal" class="equation_link">(BP1)</span><span id="margin_669494561666" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_669494561666_reveal').click(function() {$('#margin_669494561666').toggle('slow', function() {});});</script>, which gives an expression forthe output error, $\delta^L$. To prove this equation, recall that bydefinition<a class="displaced_anchor" name="eqtn36"></a>\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial z^L_j}.\tag{36}\end{eqnarray}Applying the chain rule, we can re-express the partial derivativeabove in terms of partial derivatives with respect to the outputactivations,<a class="displaced_anchor" name="eqtn37"></a>\begin{eqnarray} \delta^L_j = \sum_k \frac{\partial C}{\partial a^L_k} \frac{\partial a^L_k}{\partial z^L_j},\tag{37}\end{eqnarray}where the sum is over all neurons $k$ in the output layer. Of course,the output activation $a^L_k$ of the $k^{\rm th}$ neuron depends onlyon the weighted input $z^L_j$ for the $j^{\rm th}$ neuron when $k =j$. And so $\partial a^L_k / \partial z^L_j$ vanishes when $k \neqj$. As a result we can simplify the previous equation to<a class="displaced_anchor" name="eqtn38"></a>\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \frac{\partial a^L_j}{\partial z^L_j}.\tag{38}\end{eqnarray}Recalling that $a^L_j = \sigma(z^L_j)$ the second term on the rightcan be written as $\sigma'(z^L_j)$, and the equation becomes<a class="displaced_anchor" name="eqtn39"></a>\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j),\tag{39}\end{eqnarray}which is just <span id="margin_543019119880_reveal" class="equation_link">(BP1)</span><span id="margin_543019119880" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_543019119880_reveal').click(function() {$('#margin_543019119880').toggle('slow', function() {});});</script>, in component form.</p><p>Next, we'll prove <span id="margin_676464380519_reveal" class="equation_link">(BP2)</span><span id="margin_676464380519" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_676464380519_reveal').click(function() {$('#margin_676464380519').toggle('slow', function() {});});</script>, which gives an equation for the error$\delta^l$ in terms of the error in the next layer, $\delta^{l+1}$.To do this, we want to rewrite $\delta^l_j = \partial C / \partialz^l_j$ in terms of $\delta^{l+1}_k = \partial C / \partial z^{l+1}_k$.We can do this using the chain rule,<a class="displaced_anchor" name="eqtn40"></a><a class="displaced_anchor" name="eqtn41"></a><a class="displaced_anchor" name="eqtn42"></a>\begin{eqnarray} \delta^l_j & = & \frac{\partial C}{\partial z^l_j} \tag{40}\\ & = & \sum_k \frac{\partial C}{\partial z^{l+1}_k} \frac{\partial z^{l+1}_k}{\partial z^l_j} \tag{41}\\ & = & \sum_k \frac{\partial z^{l+1}_k}{\partial z^l_j} \delta^{l+1}_k,\tag{42}\end{eqnarray}where in the last line we have interchanged the two terms on theright-hand side, and substituted the definition of $\delta^{l+1}_k$.To evaluate the first term on the last line, note that<a class="displaced_anchor" name="eqtn43"></a>\begin{eqnarray} z^{l+1}_k = \sum_j w^{l+1}_{kj} a^l_j +b^{l+1}_k = \sum_j w^{l+1}_{kj} \sigma(z^l_j) +b^{l+1}_k.\tag{43}\end{eqnarray}Differentiating, we obtain<a class="displaced_anchor" name="eqtn44"></a>\begin{eqnarray} \frac{\partial z^{l+1}_k}{\partial z^l_j} = w^{l+1}_{kj} \sigma'(z^l_j).\tag{44}\end{eqnarray}Substituting back into <span id="margin_863771588529_reveal" class="equation_link">(42)</span><span id="margin_863771588529" class="marginequation" style="display: none;"><a href="chap2.html#eqtn42" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} & = & \sum_k \frac{\partial z^{l+1}_k}{\partial z^l_j} \delta^{l+1}_k \nonumber\end{eqnarray}</a></span><script>$('#margin_863771588529_reveal').click(function() {$('#margin_863771588529').toggle('slow', function() {});});</script> we obtain<a class="displaced_anchor" name="eqtn45"></a>\begin{eqnarray} \delta^l_j = \sum_k w^{l+1}_{kj} \delta^{l+1}_k \sigma'(z^l_j).\tag{45}\end{eqnarray}This is just <span id="margin_660951274755_reveal" class="equation_link">(BP2)</span><span id="margin_660951274755" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP2" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \nonumber\end{eqnarray}</a></span><script>$('#margin_660951274755_reveal').click(function() {$('#margin_660951274755').toggle('slow', function() {});});</script> written in component form.</p><p>The final two equations we want to prove are <span id="margin_869387373993_reveal" class="equation_link">(BP3)</span><span id="margin_869387373993" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP3" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial b^l_j} = \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_869387373993_reveal').click(function() {$('#margin_869387373993').toggle('slow', function() {});});</script>and <span id="margin_237094968075_reveal" class="equation_link">(BP4)</span><span id="margin_237094968075" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP4" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_237094968075_reveal').click(function() {$('#margin_237094968075').toggle('slow', function() {});});</script>. These also follow from the chain rule, in a mannersimilar to the proofs of the two equations above. I leave them to youas an exercise. </p><p><h4><a name="exercise_835949"></a><a href="#exercise_835949">Exercise</a></h4><ul><li> Prove Equations <span id="margin_501064179739_reveal" class="equation_link">(BP3)</span><span id="margin_501064179739" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP3" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial b^l_j} = \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_501064179739_reveal').click(function() {$('#margin_501064179739').toggle('slow', function() {});});</script> and <span id="margin_840492722163_reveal" class="equation_link">(BP4)</span><span id="margin_840492722163" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP4" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_840492722163_reveal').click(function() {$('#margin_840492722163').toggle('slow', function() {});});</script>.</ul></p><p>That completes the proof of the four fundamental equations ofbackpropagation. The proof may seem complicated. But it's reallyjust the outcome of carefully applying the chain rule. A little lesssuccinctly, we can think of backpropagation as a way of computing thegradient of the cost function by systematically applying the chainrule from multi-variable calculus. That's all there really is tobackpropagation - the rest is details.</p><p><h3><a name="the_backpropagation_algorithm"></a><a href="#the_backpropagation_algorithm">The backpropagation algorithm</a></h3></p><p>The backpropagation equations provide us with a way of computing thegradient of the cost function. Let's explicitly write this out in theform of an algorithm:<ol><li> <strong>Input $x$:</strong> Set the corresponding activation $a^{1}$ for the input layer. </p><p><li> <strong>Feedforward:</strong> For each $l = 2, 3, \ldots, L$ compute $z^{l} = w^l a^{l-1}+b^l$ and $a^{l} = \sigma(z^{l})$.</p><p><li> <strong>Output error $\delta^L$:</strong> Compute the vector $\delta^{L} = \nabla_a C \odot \sigma'(z^L)$.</p><p><li> <strong>Backpropagate the error:</strong> For each $l = L-1, L-2, \ldots, 2$ compute $\delta^{l} = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^{l})$.</p><p><li> <strong>Output:</strong> The gradient of the cost function is given by $\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j$ and $\frac{\partial C}{\partial b^l_j} = \delta^l_j$.</ol></p><p>Examining the algorithm you can see why it's called<em>back</em>propagation. We compute the error vectors $\delta^l$backward, starting from the final layer. It may seem peculiar thatwe're going through the network backward. But if you think about theproof of backpropagation, the backward movement is a consequence ofthe fact that the cost is a function of outputs from the network. Tounderstand how the cost varies with earlier weights and biases we needto repeatedly apply the chain rule, working backward through thelayers to obtain usable expressions.</p><p><h4><a name="exercises_675621"></a><a href="#exercises_675621">Exercises</a></h4><ul><li><strong>Backpropagation with a single modified neuron</strong> Suppose we modify a single neuron in a feedforward network so that the output from the neuron is given by $f(\sum_j w_j x_j + b)$, where $f$ is some function other than the sigmoid. How should we modify the backpropagation algorithm in this case?</p><p><li><strong>Backpropagation with linear neurons</strong> Suppose we replace the usual non-linear $\sigma$ function with $\sigma(z) = z$ throughout the network. Rewrite the backpropagation algorithm for this case.</ul></p><p>As I've described it above, the backpropagation algorithm computes thegradient of the cost function for a single training example, $C =C_x$. In practice, it's common to combine backpropagation with alearning algorithm such as stochastic gradient descent, in which wecompute the gradient for many training examples. In particular, givena mini-batch of $m$ training examples, the following algorithm appliesa gradient descent learning step based on that mini-batch:<ol><li> <strong>Input a set of training examples</strong></p><p><li> <strong>For each training example $x$:</strong> Set the corresponding input activation $a^{x,1}$, and perform the following steps:</p><p><ul><li> <strong>Feedforward:</strong> For each $l = 2, 3, \ldots, L$ compute $z^{x,l} = w^l a^{x,l-1}+b^l$ and $a^{x,l} = \sigma(z^{x,l})$.</p><p><li> <strong>Output error $\delta^{x,L}$:</strong> Compute the vector $\delta^{x,L} = \nabla_a C_x \odot \sigma'(z^{x,L})$.</p><p><li> <strong>Backpropagate the error:</strong> For each $l = L-1, L-2, \ldots, 2$ compute $\delta^{x,l} = ((w^{l+1})^T \delta^{x,l+1}) \odot \sigma'(z^{x,l})$.</ul></p><p><li> <strong>Gradient descent:</strong> For each $l = L, L-1, \ldots, 2$ update the weights according to the rule $w^l \rightarrow w^l-\frac{\eta}{m} \sum_x \delta^{x,l} (a^{x,l-1})^T$, and the biases according to the rule $b^l \rightarrow b^l-\frac{\eta}{m} \sum_x \delta^{x,l}$.</p><p></ol>Of course, to implement stochastic gradient descent in practice youalso need an outer loop generating mini-batches of training examples,and an outer loop stepping through multiple epochs of training. I'veomitted those for simplicity. </p><p></p><p><h3><a name="the_code_for_backpropagation"></a><a href="#the_code_for_backpropagation">The code for backpropagation</a></h3></p><p>Having understood backpropagation in the abstract, we can nowunderstand the code used in the last chapter to implementbackpropagation. Recall from<a href="chap1.html#implementing_our_network_to_classify_digits">that chapter</a> that the code was contained in the <tt>update_mini_batch</tt>and <tt>backprop</tt> methods of the <tt>Network</tt> class. The code forthese methods is a direct translation of the algorithm describedabove. In particular, the <tt>update_mini_batch</tt> method updates the<tt>Network</tt>'s weights and biases by computing the gradient for thecurrent <tt>mini_batch</tt> of training examples:<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Network</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">update_mini_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">mini_batch</span><span class="p">,</span> <span class="n">eta</span><span class="p">):</span>
<span class="sd">"""Update the network's weights and biases by applying</span>
<span class="sd"> gradient descent using backpropagation to a single mini batch.</span>
<span class="sd"> The "mini_batch" is a list of tuples "(x, y)", and "eta"</span>
<span class="sd"> is the learning rate."""</span>
<span class="n">nabla_b</span> <span class="o">=</span> <span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">biases</span><span class="p">]</span>
<span class="n">nabla_w</span> <span class="o">=</span> <span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">w</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">weights</span><span class="p">]</span>
<span class="k">for</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="ow">in</span> <span class="n">mini_batch</span><span class="p">:</span>
<span class="n">delta_nabla_b</span><span class="p">,</span> <span class="n">delta_nabla_w</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">backprop</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="n">nabla_b</span> <span class="o">=</span> <span class="p">[</span><span class="n">nb</span><span class="o">+</span><span class="n">dnb</span> <span class="k">for</span> <span class="n">nb</span><span class="p">,</span> <span class="n">dnb</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">nabla_b</span><span class="p">,</span> <span class="n">delta_nabla_b</span><span class="p">)]</span>
<span class="n">nabla_w</span> <span class="o">=</span> <span class="p">[</span><span class="n">nw</span><span class="o">+</span><span class="n">dnw</span> <span class="k">for</span> <span class="n">nw</span><span class="p">,</span> <span class="n">dnw</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">nabla_w</span><span class="p">,</span> <span class="n">delta_nabla_w</span><span class="p">)]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">weights</span> <span class="o">=</span> <span class="p">[</span><span class="n">w</span><span class="o">-</span><span class="p">(</span><span class="n">eta</span><span class="o">/</span><span class="nb">len</span><span class="p">(</span><span class="n">mini_batch</span><span class="p">))</span><span class="o">*</span><span class="n">nw</span>
<span class="k">for</span> <span class="n">w</span><span class="p">,</span> <span class="n">nw</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">weights</span><span class="p">,</span> <span class="n">nabla_w</span><span class="p">)]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">biases</span> <span class="o">=</span> <span class="p">[</span><span class="n">b</span><span class="o">-</span><span class="p">(</span><span class="n">eta</span><span class="o">/</span><span class="nb">len</span><span class="p">(</span><span class="n">mini_batch</span><span class="p">))</span><span class="o">*</span><span class="n">nb</span>
<span class="k">for</span> <span class="n">b</span><span class="p">,</span> <span class="n">nb</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">biases</span><span class="p">,</span> <span class="n">nabla_b</span><span class="p">)]</span>
</pre></div>
Most of the work is done by the line<tt>delta_nabla_b, delta_nabla_w = self.backprop(x, y)</tt> which usesthe <tt>backprop</tt> method to figure out the partial derivatives$\partial C_x / \partial b^l_j$ and $\partial C_x / \partialw^l_{jk}$. The <tt>backprop</tt> method follows the algorithm in thelast section closely. There is one small change - we use a slightlydifferent approach to indexing the layers. This change is made totake advantage of a feature of Python, namely the use of negative listindices to count backward from the end of a list, so, e.g.,<tt>l[-3]</tt> is the third last entry in a list <tt>l</tt>. The code for<tt>backprop</tt> is below, together with a few helper functions, whichare used to compute the $\sigma$ function, the derivative $\sigma'$,and the derivative of the cost function. With these inclusions youshould be able to understand the code in a self-contained way. Ifsomething's tripping you up, you may find it helpful to consult<a href="chap1.html#implementing_our_network_to_classify_digits">the original description (and complete listing) of the code</a>.<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Network</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">backprop</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="sd">"""Return a tuple "(nabla_b, nabla_w)" representing the</span>
<span class="sd"> gradient for the cost function C_x. "nabla_b" and</span>
<span class="sd"> "nabla_w" are layer-by-layer lists of numpy arrays, similar</span>
<span class="sd"> to "self.biases" and "self.weights"."""</span>
<span class="n">nabla_b</span> <span class="o">=</span> <span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">biases</span><span class="p">]</span>
<span class="n">nabla_w</span> <span class="o">=</span> <span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">w</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">weights</span><span class="p">]</span>
<span class="c1"># feedforward</span>
<span class="n">activation</span> <span class="o">=</span> <span class="n">x</span>
<span class="n">activations</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="p">]</span> <span class="c1"># list to store all the activations, layer by layer</span>
<span class="n">zs</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># list to store all the z vectors, layer by layer</span>
<span class="k">for</span> <span class="n">b</span><span class="p">,</span> <span class="n">w</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">biases</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">weights</span><span class="p">):</span>
<span class="n">z</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="n">activation</span><span class="p">)</span><span class="o">+</span><span class="n">b</span>
<span class="n">zs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">z</span><span class="p">)</span>
<span class="n">activation</span> <span class="o">=</span> <span class="n">sigmoid</span><span class="p">(</span><span class="n">z</span><span class="p">)</span>
<span class="n">activations</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">activation</span><span class="p">)</span>
<span class="c1"># backward pass</span>
<span class="n">delta</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cost_derivative</span><span class="p">(</span><span class="n">activations</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">y</span><span class="p">)</span> <span class="o">*</span> \
<span class="n">sigmoid_prime</span><span class="p">(</span><span class="n">zs</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="n">nabla_b</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">delta</span>
<span class="n">nabla_w</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">delta</span><span class="p">,</span> <span class="n">activations</span><span class="p">[</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span><span class="o">.</span><span class="n">transpose</span><span class="p">())</span>
<span class="c1"># Note that the variable l in the loop below is used a little</span>
<span class="c1"># differently to the notation in Chapter 2 of the book. Here,</span>
<span class="c1"># l = 1 means the last layer of neurons, l = 2 is the</span>
<span class="c1"># second-last layer, and so on. It's a renumbering of the</span>
<span class="c1"># scheme in the book, used here to take advantage of the fact</span>
<span class="c1"># that Python can use negative indices in lists.</span>
<span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_layers</span><span class="p">):</span>
<span class="n">z</span> <span class="o">=</span> <span class="n">zs</span><span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span>
<span class="n">sp</span> <span class="o">=</span> <span class="n">sigmoid_prime</span><span class="p">(</span><span class="n">z</span><span class="p">)</span>
<span class="n">delta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">weights</span><span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">transpose</span><span class="p">(),</span> <span class="n">delta</span><span class="p">)</span> <span class="o">*</span> <span class="n">sp</span>
<span class="n">nabla_b</span><span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="o">=</span> <span class="n">delta</span>
<span class="n">nabla_w</span><span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">delta</span><span class="p">,</span> <span class="n">activations</span><span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">transpose</span><span class="p">())</span>
<span class="k">return</span> <span class="p">(</span><span class="n">nabla_b</span><span class="p">,</span> <span class="n">nabla_w</span><span class="p">)</span>
<span class="o">...</span>
<span class="k">def</span> <span class="nf">cost_derivative</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">output_activations</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
<span class="sd">"""Return the vector of partial derivatives \partial C_x /</span>
<span class="sd"> \partial a for the output activations."""</span>
<span class="k">return</span> <span class="p">(</span><span class="n">output_activations</span><span class="o">-</span><span class="n">y</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">sigmoid</span><span class="p">(</span><span class="n">z</span><span class="p">):</span>
<span class="sd">"""The sigmoid function."""</span>
<span class="k">return</span> <span class="mf">1.0</span><span class="o">/</span><span class="p">(</span><span class="mf">1.0</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">z</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">sigmoid_prime</span><span class="p">(</span><span class="n">z</span><span class="p">):</span>
<span class="sd">"""Derivative of the sigmoid function."""</span>
<span class="k">return</span> <span class="n">sigmoid</span><span class="p">(</span><span class="n">z</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="n">sigmoid</span><span class="p">(</span><span class="n">z</span><span class="p">))</span>
</pre></div>
</p><p><a id="backprop_over_minibatch"></a><h4><a name="problem_269962"></a><a href="#problem_269962">Problem</a></h4><ul><li><strong>Fully matrix-based approach to backpropagation over a mini-batch</strong> Our implementation of stochastic gradient descent loops over training examples in a mini-batch. It's possible to modify the backpropagation algorithm so that it computes the gradients for all training examples in a mini-batch simultaneously. The idea is that instead of beginning with a single input vector, $x$, we can begin with a matrix $X = [x_1 x_2 \ldots x_m]$ whose columns are the vectors in the mini-batch. We forward-propagate by multiplying by the weight matrices, adding a suitable matrix for the bias terms, and applying the sigmoid function everywhere. We backpropagate along similar lines. Explicitly write out pseudocode for this approach to the backpropagation algorithm. Modify <tt>network.py</tt> so that it uses this fully matrix-based approach. The advantage of this approach is that it takes full advantage of modern libraries for linear algebra. As a result it can be quite a bit faster than looping over the mini-batch. (On my laptop, for example, the speedup is about a factor of two when run on MNIST classification problems like those we considered in the last chapter.) In practice, all serious libraries for backpropagation use this fully matrix-based approach or some variant.</ul></p><p><h3><a name="in_what_sense_is_backpropagation_a_fast_algorithm"></a><a href="#in_what_sense_is_backpropagation_a_fast_algorithm">In what sense is backpropagation a fast algorithm?</a></h3></p><p>In what sense is backpropagation a fast algorithm? To answer thisquestion, let's consider another approach to computing the gradient.Imagine it's the early days of neural networks research. Maybe it'sthe 1950s or 1960s, and you're the first person in the world to thinkof using gradient descent to learn! But to make the idea work youneed a way of computing the gradient of the cost function. You thinkback to your knowledge of calculus, and decide to see if you can usethe chain rule to compute the gradient. But after playing around abit, the algebra looks complicated, and you get discouraged. So youtry to find another approach. You decide to regard the cost as afunction of the weights $C = C(w)$ alone (we'll get back to the biasesin a moment). You number the weights $w_1, w_2, \ldots$, and want tocompute $\partial C / \partial w_j$ for some particular weight $w_j$.An obvious way of doing that is to use the approximation<a class="displaced_anchor" name="eqtn46"></a>\begin{eqnarray} \frac{\partial C}{\partial w_{j}} \approx \frac{C(w+\epsilon e_j)-C(w)}{\epsilon},\tag{46}\end{eqnarray}where $\epsilon > 0$ is a small positive number, and $e_j$ is the unitvector in the $j^{\rm th}$ direction. In other words, we can estimate$\partial C / \partial w_j$ by computing the cost $C$ for two slightlydifferent values of $w_j$, and then applyingEquation <span id="margin_674486079288_reveal" class="equation_link">(46)</span><span id="margin_674486079288" class="marginequation" style="display: none;"><a href="chap2.html#eqtn46" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w_{j}} \approx \frac{C(w+\epsilon e_j)-C(w)}{\epsilon} \nonumber\end{eqnarray}</a></span><script>$('#margin_674486079288_reveal').click(function() {$('#margin_674486079288').toggle('slow', function() {});});</script>. The same idea will let uscompute the partial derivatives $\partial C / \partial b$ with respectto the biases.</p><p>This approach looks very promising. It's simple conceptually, andextremely easy to implement, using just a few lines of code.Certainly, it looks much more promising than the idea of using thechain rule to compute the gradient!</p><p>Unfortunately, while this approach appears promising, when youimplement the code it turns out to be extremely slow. To understandwhy, imagine we have a million weights in our network. Then for eachdistinct weight $w_j$ we need to compute $C(w+\epsilon e_j)$ in orderto compute $\partial C / \partial w_j$. That means that to computethe gradient we need to compute the cost function a million differenttimes, requiring a million forward passes through the network (pertraining example). We need to compute $C(w)$ as well, so that's atotal of a million and one passes through the network.</p><p>What's clever about backpropagation is that it enables us tosimultaneously compute <em>all</em> the partial derivatives $\partial C/ \partial w_j$ using just one forward pass through the network,followed by one backward pass through the network. Roughly speaking,the computational cost of the backward pass is about the same as theforward pass*<span class="marginnote">*This should be plausible, but it requires some analysis to make a careful statement. It's plausible because the dominant computational cost in the forward pass is multiplying by the weight matrices, while in the backward pass it's multiplying by the transposes of the weight matrices. These operations obviously have similar computational cost.</span>. And so the total cost ofbackpropagation is roughly the same as making just two forward passesthrough the network. Compare that to the million and one forwardpasses we needed for the approach basedon <span id="margin_884034767764_reveal" class="equation_link">(46)</span><span id="margin_884034767764" class="marginequation" style="display: none;"><a href="chap2.html#eqtn46" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w_{j}} \approx \frac{C(w+\epsilon e_j)-C(w)}{\epsilon} \nonumber\end{eqnarray}</a></span><script>$('#margin_884034767764_reveal').click(function() {$('#margin_884034767764').toggle('slow', function() {});});</script>! And so even though backpropagationappears superficially more complex than the approach basedon <span id="margin_806797723031_reveal" class="equation_link">(46)</span><span id="margin_806797723031" class="marginequation" style="display: none;"><a href="chap2.html#eqtn46" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w_{j}} \approx \frac{C(w+\epsilon e_j)-C(w)}{\epsilon} \nonumber\end{eqnarray}</a></span><script>$('#margin_806797723031_reveal').click(function() {$('#margin_806797723031').toggle('slow', function() {});});</script>, it's actually much, much faster.</p><p>This speedup was first fully appreciated in 1986, and it greatlyexpanded the range of problems that neural networks could solve.That, in turn, caused a rush of people using neural networks. Ofcourse, backpropagation is not a panacea. Even in the late 1980speople ran up against limits, especially when attempting to usebackpropagation to train deep neural networks, i.e., networks withmany hidden layers. Later in the book we'll see how modern computersand some clever new ideas now make it possible to use backpropagationto train such deep neural networks.</p><p><h3><a name="backpropagation_the_big_picture"></a><a href="#backpropagation_the_big_picture">Backpropagation: the big picture</a></h3></p><p>As I've explained it, backpropagation presents two mysteries. First,what's the algorithm really doing? We've developed a picture of theerror being backpropagated from the output. But can we go any deeper,and build up more intuition about what is going on when we do allthese matrix and vector multiplications? The second mystery is howsomeone could ever have discovered backpropagation in the first place?It's one thing to follow the steps in an algorithm, or even to followthe proof that the algorithm works. But that doesn't mean youunderstand the problem so well that you could have discovered thealgorithm in the first place. Is there a plausible line of reasoningthat could have led you to discover the backpropagation algorithm? Inthis section I'll address both these mysteries.</p><p>To improve our intuition about what the algorithm is doing, let'simagine that we've made a small change $\Delta w^l_{jk}$ to someweight in the network, $w^l_{jk}$:<center><img src="images/tikz22.png"/></center>That change in weight will cause a change in the output activationfrom the corresponding neuron:<center><img src="images/tikz23.png"/></center>That, in turn, will cause a change in <em>all</em> the activations inthe next layer:<center><img src="images/tikz24.png"/></center>Those changes will in turn cause changes in the next layer, and thenthe next, and so on all the way through to causing a change in thefinal layer, and then in the cost function:<center><img src="images/tikz25.png"/></center>The change $\Delta C$ in the cost is related to the change $\Deltaw^l_{jk}$ in the weight by the equation<a class="displaced_anchor" name="eqtn47"></a>\begin{eqnarray} \Delta C \approx \frac{\partial C}{\partial w^l_{jk}} \Delta w^l_{jk}.\tag{47}\end{eqnarray}This suggests that a possible approach to computing $\frac{\partial C}{\partial w^l_{jk}}$ is to carefully track how a small change in$w^l_{jk}$ propagates to cause a small change in $C$. If we can dothat, being careful to express everything along the way in terms ofeasily computable quantities, then we should be able to compute$\partial C / \partial w^l_{jk}$.</p><p>Let's try to carry this out. The change $\Delta w^l_{jk}$ causes asmall change $\Delta a^{l}_j$ in the activation of the $j^{\rm th}$ neuron inthe $l^{\rm th}$ layer. This change is given by<a class="displaced_anchor" name="eqtn48"></a>\begin{eqnarray} \Delta a^l_j \approx \frac{\partial a^l_j}{\partial w^l_{jk}} \Delta w^l_{jk}.\tag{48}\end{eqnarray}The change in activation $\Delta a^l_{j}$ will cause changes in<em>all</em> the activations in the next layer, i.e., the $(l+1)^{\rm th}$ layer. We'll concentrate on the way just a single one of thoseactivations is affected, say $a^{l+1}_q$,<center><img src="images/tikz26.png"/></center>In fact, it'll cause the following change:<a class="displaced_anchor" name="eqtn49"></a>\begin{eqnarray} \Delta a^{l+1}_q \approx \frac{\partial a^{l+1}_q}{\partial a^l_j} \Delta a^l_j.\tag{49}\end{eqnarray}Substituting in the expression from Equation <span id="margin_531795813347_reveal" class="equation_link">(48)</span><span id="margin_531795813347" class="marginequation" style="display: none;"><a href="chap2.html#eqtn48" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \Delta a^l_j \approx \frac{\partial a^l_j}{\partial w^l_{jk}} \Delta w^l_{jk} \nonumber\end{eqnarray}</a></span><script>$('#margin_531795813347_reveal').click(function() {$('#margin_531795813347').toggle('slow', function() {});});</script>,we get:<a class="displaced_anchor" name="eqtn50"></a>\begin{eqnarray} \Delta a^{l+1}_q \approx \frac{\partial a^{l+1}_q}{\partial a^l_j} \frac{\partial a^l_j}{\partial w^l_{jk}} \Delta w^l_{jk}.\tag{50}\end{eqnarray}Of course, the change $\Delta a^{l+1}_q$ will, in turn, cause changesin the activations in the next layer. In fact, we can imagine a pathall the way through the network from $w^l_{jk}$ to $C$, with eachchange in activation causing a change in the next activation, and,finally, a change in the cost at the output. If the path goes throughactivations $a^l_j, a^{l+1}_q, \ldots, a^{L-1}_n, a^L_m$ then theresulting expression is<a class="displaced_anchor" name="eqtn51"></a>\begin{eqnarray} \Delta C \approx \frac{\partial C}{\partial a^L_m} \frac{\partial a^L_m}{\partial a^{L-1}_n} \frac{\partial a^{L-1}_n}{\partial a^{L-2}_p} \ldots \frac{\partial a^{l+1}_q}{\partial a^l_j} \frac{\partial a^l_j}{\partial w^l_{jk}} \Delta w^l_{jk},\tag{51}\end{eqnarray}that is, we've picked up a $\partial a / \partial a$ type term foreach additional neuron we've passed through, as well as the $\partialC/\partial a^L_m$ term at the end. This represents the change in $C$due to changes in the activations along this particular path throughthe network. Of course, there's many paths by which a change in$w^l_{jk}$ can propagate to affect the cost, and we've beenconsidering just a single path. To compute the total change in $C$ itis plausible that we should sum over all the possible paths betweenthe weight and the final cost, i.e.,<a class="displaced_anchor" name="eqtn52"></a>\begin{eqnarray} \Delta C \approx \sum_{mnp\ldots q} \frac{\partial C}{\partial a^L_m} \frac{\partial a^L_m}{\partial a^{L-1}_n} \frac{\partial a^{L-1}_n}{\partial a^{L-2}_p} \ldots \frac{\partial a^{l+1}_q}{\partial a^l_j} \frac{\partial a^l_j}{\partial w^l_{jk}} \Delta w^l_{jk},\tag{52}\end{eqnarray}where we've summed over all possible choices for the intermediateneurons along the path. Comparing with <span id="margin_103206561462_reveal" class="equation_link">(47)</span><span id="margin_103206561462" class="marginequation" style="display: none;"><a href="chap2.html#eqtn47" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \Delta C \approx \frac{\partial C}{\partial w^l_{jk}} \Delta w^l_{jk} \nonumber\end{eqnarray}</a></span><script>$('#margin_103206561462_reveal').click(function() {$('#margin_103206561462').toggle('slow', function() {});});</script> wesee that<a class="displaced_anchor" name="eqtn53"></a>\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = \sum_{mnp\ldots q} \frac{\partial C}{\partial a^L_m} \frac{\partial a^L_m}{\partial a^{L-1}_n} \frac{\partial a^{L-1}_n}{\partial a^{L-2}_p} \ldots \frac{\partial a^{l+1}_q}{\partial a^l_j} \frac{\partial a^l_j}{\partial w^l_{jk}}.\tag{53}\end{eqnarray}Now, Equation <span id="margin_963684443937_reveal" class="equation_link">(53)</span><span id="margin_963684443937" class="marginequation" style="display: none;"><a href="chap2.html#eqtn53" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = \sum_{mnp\ldots q} \frac{\partial C}{\partial a^L_m} \frac{\partial a^L_m}{\partial a^{L-1}_n} \frac{\partial a^{L-1}_n}{\partial a^{L-2}_p} \ldots \frac{\partial a^{l+1}_q}{\partial a^l_j} \frac{\partial a^l_j}{\partial w^l_{jk}} \nonumber\end{eqnarray}</a></span><script>$('#margin_963684443937_reveal').click(function() {$('#margin_963684443937').toggle('slow', function() {});});</script> looks complicated. However,it has a nice intuitive interpretation. We're computing the rate ofchange of $C$ with respect to a weight in the network. What theequation tells us is that every edge between two neurons in thenetwork is associated with a rate factor which is just the partialderivative of one neuron's activation with respect to the otherneuron's activation. The edge from the first weight to the firstneuron has a rate factor $\partial a^{l}_j / \partial w^l_{jk}$. Therate factor for a path is just the product of the rate factors alongthe path. And the total rate of change $\partial C / \partialw^l_{jk}$ is just the sum of the rate factors of all paths from theinitial weight to the final cost. This procedure is illustrated here,for a single path:<center><img src="images/tikz27.png"/></center></p><p>What I've been providing up to now is a heuristic argument, a way ofthinking about what's going on when you perturb a weight in a network.Let me sketch out a line of thinking you could use to further developthis argument. First, you could derive explicit expressions for allthe individual partial derivatives inEquation <span id="margin_602209892794_reveal" class="equation_link">(53)</span><span id="margin_602209892794" class="marginequation" style="display: none;"><a href="chap2.html#eqtn53" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = \sum_{mnp\ldots q} \frac{\partial C}{\partial a^L_m} \frac{\partial a^L_m}{\partial a^{L-1}_n} \frac{\partial a^{L-1}_n}{\partial a^{L-2}_p} \ldots \frac{\partial a^{l+1}_q}{\partial a^l_j} \frac{\partial a^l_j}{\partial w^l_{jk}} \nonumber\end{eqnarray}</a></span><script>$('#margin_602209892794_reveal').click(function() {$('#margin_602209892794').toggle('slow', function() {});});</script>. That's easy to do with a bit ofcalculus. Having done that, you could then try to figure out how towrite all the sums over indices as matrix multiplications. This turnsout to be tedious, and requires some persistence, but notextraordinary insight. After doing all this, and then simplifying asmuch as possible, what you discover is that you end up with exactlythe backpropagation algorithm! And so you can think of thebackpropagation algorithm as providing a way of computing the sum overthe rate factor for all these paths. Or, to put it slightlydifferently, the backpropagation algorithm is a clever way of keepingtrack of small perturbations to the weights (and biases) as theypropagate through the network, reach the output, and then affect thecost.</p><p>Now, I'm not going to work through all this here. It's messy andrequires considerable care to work through all the details. If you'reup for a challenge, you may enjoy attempting it. And even if not, Ihope this line of thinking gives you some insight into whatbackpropagation is accomplishing.</p><p>What about the other mystery - how backpropagation could have beendiscovered in the first place? In fact, if you follow the approach Ijust sketched you will discover a proof of backpropagation.Unfortunately, the proof is quite a bit longer and more complicatedthan the one I described earlier in this chapter. So how was thatshort (but more mysterious) proof discovered? What you find when youwrite out all the details of the long proof is that, after the fact,there are several obvious simplifications staring you in the face.You make those simplifications, get a shorter proof, and write thatout. And then several more obvious simplifications jump out atyou. So you repeat again. The result after a few iterations is theproof we saw earlier*<span class="marginnote">*There is one clever step required. In Equation <span id="margin_817509150234_reveal" class="equation_link">(53)</span><span id="margin_817509150234" class="marginequation" style="display: none;"><a href="chap2.html#eqtn53" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = \sum_{mnp\ldots q} \frac{\partial C}{\partial a^L_m} \frac{\partial a^L_m}{\partial a^{L-1}_n} \frac{\partial a^{L-1}_n}{\partial a^{L-2}_p} \ldots \frac{\partial a^{l+1}_q}{\partial a^l_j} \frac{\partial a^l_j}{\partial w^l_{jk}} \nonumber\end{eqnarray}</a></span><script>$('#margin_817509150234_reveal').click(function() {$('#margin_817509150234').toggle('slow', function() {});});</script> the intermediate variables are activations like $a_q^{l+1}$. The clever idea is to switch to using weighted inputs, like $z^{l+1}_q$, as the intermediate variables. If you don't have this idea, and instead continue using the activations $a^{l+1}_q$, the proof you obtain turns out to be slightly more complex than the proof given earlier in the chapter.</span> - short, butsomewhat obscure, because all the signposts to its construction havebeen removed! I am, of course, asking you to trust me on this, butthere really is no great mystery to the origin of the earlier proof.It's just a lot of hard work simplifying the proof I've sketched inthis section.</p><p><br/><br/><br/></p><p></div><div class="footer"> <span class="left_footer"> In academic work,
please cite this book as: Michael A. Nielsen, "Neural Networks and
Deep Learning", Determination Press, 2015
<br/>
<br/>
This work is licensed under a <a rel="license"
href="http://creativecommons.org/licenses/by-nc/3.0/deed.en_GB"
style="color: #eee;">Creative Commons Attribution-NonCommercial 3.0
Unported License</a>. This means you're free to copy, share, and
build on this book, but not to sell it. If you're interested in
commercial use, please <a
href="mailto:mn@michaelnielsen.org">contact me</a>.
</span>
<span class="right_footer">
Last update: Thu Jan 19 06:09:48 2017
<br/>
<br/>
<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc/3.0/deed.en_GB"><img alt="Creative Commons Licence" style="border-width:0" src="http://i.creativecommons.org/l/by-nc/3.0/88x31.png" /></a>
</span>
</div>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-44208967-1', 'neuralnetworksanddeeplearning.com');
ga('send', 'pageview');
</script>
</body>
</html>