forked from fastai/fastai
-
Notifications
You must be signed in to change notification settings - Fork 0
/
callbacks.rnn.html
156 lines (109 loc) · 8.95 KB
/
callbacks.rnn.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
---
title: callbacks.rnn
keywords: fastai
sidebar: home_sidebar
summary: "Implementation of a callback for RNN training"
---
<!--
#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: docs_src/callbacks.rnn.ipynb
# instructions: https://docs.fast.ai/gen_doc_main.html
-->
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Training-tweaks-for-an-RNN">Training tweaks for an RNN<a class="anchor-link" href="#Training-tweaks-for-an-RNN">¶</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
</div>
<div class="cell border-box-sizing code_cell rendered">
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This callback regroups a few tweaks to properly train RNNs. They all come from <a href="https://arxiv.org/abs/1708.02182">this article</a> by Stephen Merity et al.</p>
<p><strong>Activation Regularization:</strong> on top of weight decay, we apply another form of regularization that is pretty similar and consists in adding to the loss a scaled factor of the sum of all the squares of the outputs (with dropout applied) of the various layers of the RNN. Intuitively, weight decay tries to get the network to learn small weights, this is to get the model to learn to produce smaller activations.</p>
<p><strong>Temporal Activation Regularization:</strong> lastly, we add to the loss a scaled factor of the sum of the squares of the <code>h_(t+1) - h_t</code>, where <code>h_i</code> is the output (before dropout is applied) of one layer of the RNN at the time step i (word i of the sentence). This will encourage the model to produce activations that don’t vary too fast between two consecutive words of the sentence.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_markdown rendered_html output_subarea ">
<h2 id="RNNTrainer" class="doc_header"><code>class</code> <code>RNNTrainer</code><a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/rnn.py#L8" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#RNNTrainer-pytest" style="float:right; padding-right:10px">[test]</a></h2><blockquote><p><code>RNNTrainer</code>(<strong><code>learn</code></strong>:<a href="/basic_train.html#Learner"><code>Learner</code></a>, <strong><code>alpha</code></strong>:<code>float</code>=<strong><em><code>0.0</code></em></strong>, <strong><code>beta</code></strong>:<code>float</code>=<strong><em><code>0.0</code></em></strong>) :: <a href="/basic_train.html#LearnerCallback"><code>LearnerCallback</code></a></p>
</blockquote>
<div class="collapse" id="RNNTrainer-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#RNNTrainer-pytest" class="close" aria-label="Close"><span aria-hidden="true">×</span></a><p>No tests found for <code>RNNTrainer</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div><p><a href="/callback.html#Callback"><code>Callback</code></a> that regroups lr adjustment to seq_len, AR and TAR.</p>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Create a <a href="/callback.html#Callback"><code>Callback</code></a> that adds to learner the RNN tweaks for training on data with <code>bptt</code>. <code>alpha</code> is the scale for AR, <code>beta</code> is the scale for TAR.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Callback-methods">Callback methods<a class="anchor-link" href="#Callback-methods">¶</a></h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>You don't call these yourself - they're called by fastai's <a href="/callback.html#Callback"><code>Callback</code></a> system automatically to enable the class's functionality.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_markdown rendered_html output_subarea ">
<h4 id="RNNTrainer.on_epoch_begin" class="doc_header"><code>on_epoch_begin</code><a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/rnn.py#L15" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#RNNTrainer-on_epoch_begin-pytest" style="float:right; padding-right:10px">[test]</a></h4><blockquote><p><code>on_epoch_begin</code>(<strong>**<code>kwargs</code></strong>)</p>
</blockquote>
<div class="collapse" id="RNNTrainer-on_epoch_begin-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#RNNTrainer-on_epoch_begin-pytest" class="close" aria-label="Close"><span aria-hidden="true">×</span></a><p>No tests found for <code>on_epoch_begin</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div><p>Reset the hidden state of the model.</p>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_markdown rendered_html output_subarea ">
<h4 id="RNNTrainer.on_loss_begin" class="doc_header"><code>on_loss_begin</code><a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/rnn.py#L19" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#RNNTrainer-on_loss_begin-pytest" style="float:right; padding-right:10px">[test]</a></h4><blockquote><p><code>on_loss_begin</code>(<strong><code>last_output</code></strong>:<code>Tuple</code>[<code>Tensor</code>, <code>Tensor</code>, <code>Tensor</code>], <strong>**<code>kwargs</code></strong>)</p>
</blockquote>
<div class="collapse" id="RNNTrainer-on_loss_begin-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#RNNTrainer-on_loss_begin-pytest" class="close" aria-label="Close"><span aria-hidden="true">×</span></a><p>No tests found for <code>on_loss_begin</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div><p>Save the extra outputs for later and only returns the true output.</p>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The fastai RNNs return <code>last_output</code> that are tuples of three elements, the true output (that is returned) and the hidden states before and after dropout (which are saved internally for the next function).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_markdown rendered_html output_subarea ">
<h4 id="RNNTrainer.on_backward_begin" class="doc_header"><code>on_backward_begin</code><a href="https://github.com/fastai/fastai/blob/master/fastai/callbacks/rnn.py#L24" class="source_link" style="float:right">[source]</a><a class="source_link" data-toggle="collapse" data-target="#RNNTrainer-on_backward_begin-pytest" style="float:right; padding-right:10px">[test]</a></h4><blockquote><p><code>on_backward_begin</code>(<strong><code>last_loss</code></strong>:<code>Rank0Tensor</code>, <strong><code>last_input</code></strong>:<code>Tensor</code>, <strong>**<code>kwargs</code></strong>)</p>
</blockquote>
<div class="collapse" id="RNNTrainer-on_backward_begin-pytest"><div class="card card-body pytest_card"><a type="button" data-toggle="collapse" data-target="#RNNTrainer-on_backward_begin-pytest" class="close" aria-label="Close"><span aria-hidden="true">×</span></a><p>No tests found for <code>on_backward_begin</code>. To contribute a test please refer to <a href="/dev/test.html">this guide</a> and <a href="https://forums.fast.ai/t/improving-expanding-functional-tests/32929">this discussion</a>.</p></div></div><p>Apply AR and TAR to <code>last_loss</code>.</p>
</div>
</div>
</div>
</div>
</div>
</div>