Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add identity mappings in resnets paper
- Loading branch information
Showing
6 changed files
with
28 additions
and
0 deletions.
There are no files selected for viewing
28 changes: 28 additions & 0 deletions
28
neural-nets/Identity_Mappings_in_Deep_Residual_Networks.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
# Paper | ||
|
||
* **Title**: Identity Mappings in Deep Residual Networks | ||
* **Authors**: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun | ||
* **Link**: http://arxiv.org/abs/1603.05027v2 | ||
* **Tags**: Neural Network, residual | ||
* **Year**: 2016 | ||
|
||
# Summary | ||
|
||
* What | ||
* The authors reevaluate the original residual design of neural networks. | ||
* They compare various architectures of residual units and actually find one that works quite a bit better. | ||
|
||
* How | ||
* The new variation starts the transformation branch of each residual unit with BN and a ReLU. | ||
* It removes BN and ReLU after the last convolution. | ||
* As a result, the information from previous layers can flow completely unaltered through the shortcut branch of each residual unit. | ||
* The image below shows some variations (of the position of BN and ReLU) that they tested. The new and better design is on the right: | ||
![BN and ReLU positions](images/Identity_Mappings_in_Deep_Residual_Networks__activations.png?raw=true "BN and ReLU positions") | ||
* They also tried various alternative designs for the shortcut connections. However, all of these designs performed worse than the original one. Only one (d) came close under certain conditions. Therefore, the recommendation is to stick with the old/original design. | ||
![Shortcut designs](images/Identity_Mappings_in_Deep_Residual_Networks__shortcuts.png?raw=true "Shortcut designs") | ||
|
||
* Results | ||
* Significantly faster training for very deep residual networks (1001 layers). | ||
* Better regularization due to the placement of BN. | ||
* CIFAR-10 and CIFAR-100 results, old vs. new design: | ||
![Old vs new results](images/Identity_Mappings_in_Deep_Residual_Networks__old_vs_new.png?raw=true "Old vs new results") |
Binary file added
BIN
+48.7 KB
neural-nets/images/Identity_Mappings_in_Deep_Residual_Networks__activations.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+175 KB
neural-nets/images/Identity_Mappings_in_Deep_Residual_Networks__activations.xcf
Binary file not shown.
Binary file added
BIN
+23.5 KB
neural-nets/images/Identity_Mappings_in_Deep_Residual_Networks__old_vs_new.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+49.1 KB
neural-nets/images/Identity_Mappings_in_Deep_Residual_Networks__shortcuts.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+128 KB
neural-nets/images/Identity_Mappings_in_Deep_Residual_Networks__shortcuts.xcf
Binary file not shown.