Added BN+Sigmoid and TanH networks

ducha-aiki · Mar 5, 2016 · 429cd60 · 429cd60
1 parent 2d7739f
commit 429cd60
Show file tree

Hide file tree

Showing 15 changed files with 104,705 additions and 22 deletions.
diff --git a/Activations.md b/Activations.md
@@ -15,6 +15,7 @@ Because LRN layers add nothing to accuracy, they were removed for speed reasons
 | -------|---------:| -------:|:-----------|
 | [ReLU](http://machinelearning.wustl.edu/mlpapers/paper_files/icml2010_NairH10.pdf) |0.470| 2.36 | With LRN layers|
 | ReLU |0.470| 2.36 | No LRN, as in rest |
+| TanH |0.401| 2.78 |  |
 | [VLReLU](https://web.stanford.edu/~awni/papers/relu_hybrid_icml2013_final.pdf) |0.469| 2.40|y=max(x,x/3)|
 | [RReLU](http://arxiv.org/abs/1505.00853) |0.478| 2.32| |
 | [Maxout](http://arxiv.org/abs/1302.4389) |0.482| 2.30| sqrt(2) narrower layers, 2 pieces|
@@ -32,6 +33,8 @@ Because LRN layers add nothing to accuracy, they were removed for speed reasons
 | PReLU |**0.503**| **2.19** | |
 | ELU |0.498| 2.23 | |
 | Maxout |0.487| 2.28| |
+| Sigmoid |0.475| 2.35| |
+
 
 ![CaffeNet128 test accuracy](/logs/activations/img/0.png)
 

diff --git a/README.md b/README.md
@@ -29,6 +29,7 @@ On-going evaluations with graphs:
 | -------|---------:| -------:|:-----------|
 | [ReLU](http://machinelearning.wustl.edu/mlpapers/paper_files/icml2010_NairH10.pdf) |0.470| 2.36 | With LRN layers|
 | ReLU |0.470| 2.36 | No LRN, as in rest |
+| TanH |0.401| 2.78 |  |
 | [VLReLU](https://web.stanford.edu/~awni/papers/relu_hybrid_icml2013_final.pdf) |0.469| 2.40|y=max(x,x/3)|
 | [RReLU](http://arxiv.org/abs/1505.00853) |0.478| 2.32| |
 | [Maxout](http://arxiv.org/abs/1302.4389) |0.482| 2.30| sqrt(2) narrower layers, 2 pieces|
@@ -258,6 +259,7 @@ So in all next experiments, BN is put after non-linearity
 | PReLU |**0.503**| **2.19** | |
 | ELU |0.498| 2.23 | |
 | Maxout |0.487| 2.28| |
+| Sigmoid |0.475| 2.35| |
 
 ### BN and dropout
 

diff --git a/ResNetGenerator.ipynb b/ResNetGenerator.ipynb
diff --git a/batchnorm.md b/batchnorm.md
@@ -37,6 +37,7 @@ So in all next experiments, BN is put after non-linearity
 | PReLU |**0.503**| **2.19** | |
 | ELU |0.498| 2.23 | |
 | Maxout |0.487| 2.28| |
+| Sigmoid |0.475| 2.35| |
 
 ### BN and dropout
 

diff --git a/logs/activations/caffenet128_lsuv_ELU*.log → logs/activations/caffenet128_lsuv_ELU^.log b/logs/activations/caffenet128_lsuv_ELU*.log → logs/activations/caffenet128_lsuv_ELU^.log