Skip to content

Code for reproducibility of the E-swish paper experiments

Notifications You must be signed in to change notification settings

hypnopump/E-swish

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

E-swish: Adjusting Activations to Different Network Depths

This repository will contain the code for reproducibility of the experiments of this paper

Uploading soon.

Formula

Abstract

Activation functions have a notorious impact on neural networks on both training and testing the models against the desired problem. Currently, the most used activation function is the Rectified Linear Unit (ReLU). This paper introduces a new and novel activation function, closely related with the new activation Swish=x∗sigmoid(x) (Ramachandran et al., 2017) which generalizes it. We call the new activation E−swish=βx∗sigmoid(x) . We show that E-swish outperforms many other well-known activations including both ReLU and Swish. For example, using E-swish provided 1.5% and 4.6% accuracy improvements on Cifar10 and Cifar100 respectively for the WRN 10-2 when compared to ReLU and 0.35% and 0.6% respectively when compared to Swish. The code to reproduce all our experiments can be found at https://github.com/EricAlcaide/E-swish

Results

CIFAR10

  • WRN 10-2
Activations % Accuracy (median of 3 runs)
Relu 89.98
Swish 91.52
E-swish (beta=1.375) 91.89
  • SimpleNet
Activations % Accuracy (single model performance)
Relu 95.33
Swish 95.76
E-swish (beta=1.125) 96.02
E-swish (beta=1.25) 95.73

CIFAR100

  • WRN 10-2
Activations % Accuracy (median of 3 runs)
Relu 64.27
Swish 68.32
E-swish (beta=1.375) 68.91
  • WRN 16-4
Activations % Accuracy (single model performance)
Relu 75.92
Swish 76.88
E-swish (beta=1.25) 77.35
E-swish (beta=1.5) 77.25
E-swish (beta=1.75) 77.29
E-swish (beta=2) 77.14

Contact

About

Code for reproducibility of the E-swish paper experiments

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published