Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemetation of Concatenated ReLU Activation #1843

Merged
merged 20 commits into from May 3, 2019

Conversation

Projects
None yet
3 participants
@jeffin143
Copy link
Contributor

commented Apr 6, 2019

I tried implementing CRELU Activation, following the research paper by Wenling Shang, Kihyuk Sohn, Diogo Almeida, Honglak Lee, in 2016

This activation scheme preserves both positive and negative phase information while enforcing non-saturated non-linearity.The unique nature of CReLU allows a mathematical characterization of convolution layers in terms of reconstruction property.

CReLU has shown to perform very well in Convolutional Neural Networks.CReLU naturally reduces redundancy of learning separate filters that differ in phase only.

I encountered lot of doubts during implementation and have done some tweaking during implementation.So it may be not be fully sound on implementation part. So i won't mind even if takes long for you to review this PR, But i would like to get your input on the code.

jeffin143 added some commits Apr 4, 2019

Merge pull request #7 from mlpack/master
Syncing branch with master
@jeffin143

This comment has been minimized.

Copy link
Contributor Author

commented Apr 6, 2019

I made a comparison between LeakyRelu and CRelu, using Mlpack/models repo examples .
I took DigitRecognizer program and the dataset and Used the following model

Model for Leaky Relu

  FFN<NegativeLogLikelihood<>, RandomInitialization> model;
  model.Add<Linear<> >(trainX.n_rows,100);
  model.Add<LeakyReLU<> >();
  model.Add<Linear<> >( 100,100);
  model.Add<LeakyReLU<> >();
  model.Add<Linear<> >(100, 10);
  model.Add<LogSoftMax<> >();

Screenshot from 2019-04-06 09-01-59

Model For Crelu:

  FFN<NegativeLogLikelihood<>, RandomInitialization> model;
  model.Add<Linear<> >(trainX.n_rows,100);
  model.Add<CReLU<> >();
  model.Add<Linear<> >( 200,100);
  model.Add<CReLU<> >();
  model.Add<Linear<> >(200, 10);
  model.Add<LogSoftMax<> >();

Screenshot from 2019-04-06 09-01-55

Both gave almost same amount of performance,but may be some amount of tweaking in the implementation part may increase the accuracy.
CRELU ran for 234 sec, and LeakyReLU ran for 199 sec , may be optimizing CRELU could be possible by avoiding copying of data.

Though the following results show that CRELU and LEakyReLU are performing almost same, but the research paper had some good results when it comes to CNN part over the time.

Would like the community to comment on it.

jeffin143 added some commits Apr 8, 2019

@zoq
Copy link
Member

left a comment

Would be great to have a test for the new activation function.

Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu.hpp Outdated
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu.hpp Outdated
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu_impl.hpp Outdated
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu_impl.hpp Outdated

jeffin143 added some commits Apr 16, 2019

Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu.hpp
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu.hpp Outdated
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu.hpp Outdated
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu.hpp Outdated
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu.hpp
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu.hpp Outdated
Show resolved Hide resolved src/mlpack/tests/activation_functions_test.cpp Outdated

jeffin143 added some commits Apr 21, 2019

@jeffin143 jeffin143 force-pushed the jeffin143:crelu branch from 6af83ce to 3b8b9d2 Apr 26, 2019

@zoq
Copy link
Member

left a comment

If you can fix the two style issue, I think this is ready to be merged.

Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu_impl.hpp Outdated
Show resolved Hide resolved src/mlpack/methods/ann/layer/c_relu_impl.hpp Outdated
@zoq

zoq approved these changes Apr 28, 2019

Copy link
Member

left a comment

Looks good to me, thanks!

@mlpack-bot

mlpack-bot bot approved these changes Apr 29, 2019

Copy link

left a comment

Second approval provided automatically after 24 hours. 👍

@zoq zoq merged commit 6e19a40 into mlpack:master May 3, 2019

6 checks passed

LaTeX Documentation Checks Build finished.
Details
Memory Checks Build finished.
Details
Static Code Analysis Checks Build finished.
Details
Style Checks Build finished.
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@zoq

This comment has been minimized.

Copy link
Member

commented May 3, 2019

Thanks for the contribution.

@jeffin143 jeffin143 deleted the jeffin143:crelu branch May 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.