Small changes for faster convergence

[NikolasMarkou]

Removed redundant relus
Replaced relus with prelus with very small constant multiplier.
Haven't tested on imagenet but on all my classification tests it converged faster with slightly higher accuracy usually ~ >0.5%

[forresti]

Interesting! Nice work.

In order to upstream this, I would like:

• include an ImageNet trained model
• name this SqueezeNet v1.11

[NikolasMarkou]

That is ImageNet 2011, right ?

[forresti]

2012

…

On Tue, Feb 14, 2017 at 12:22 AM Nikolas Markou ***@***.***> wrote: That is ImageNet 2011, right ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB7SqkxZF8CPlDExK4oNScjZX7IHEkg7ks5rcWRagaJpZM4L_xPM> .

[NikolasMarkou]

Great, I'll get on to it, in the meantime it just occured to me after moving the relu/prelus that it can go a step further , since pooling is of type max everywhere there is no problem moving the relus/prelus after the pools and thus saving flops on that too and remaining equivelant to the original squeezenet v1.1

this =>
layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" convolution_param { num_output: 64 kernel_size: 3 stride: 2 weight_filler { type: "xavier" } } } layer { name: "relu_conv1" type: "ReLU" bottom: "conv1" top: "conv1" } layer { name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" pooling_param { pool: MAX kernel_size: 3 stride: 2 } }

transforming to this
layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" convolution_param { num_output: 64 kernel_size: 3 stride: 2 weight_filler { type: "xavier" } } } layer { name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" pooling_param { pool: MAX kernel_size: 3 stride: 2 } } layer { name: "relu_conv1" type: "ReLU" bottom: "conv1" top: "conv1" }

[TechnikEmpire]

Has imagenet been trained with this yet?