New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alpha dropout #1349

Merged
merged 16 commits into from Apr 7, 2018

Conversation

Projects
None yet
4 participants
@sourabhvarshney111
Contributor

sourabhvarshney111 commented Apr 2, 2018

This is towards completion of PR #1278.

dakshitagrawal97 and others added some commits Feb 26, 2018

@sourabhvarshney111

This comment has been minimized.

Contributor

sourabhvarshney111 commented Apr 3, 2018

@rcurtin @zoq
I have written some comments in #1278. Please tell me about them. Then, I think it is ready for some review.

sourabhvarshney111 added some commits Apr 3, 2018

* @param ratio The probability of setting a value to alphaDash.
* @param alphaDash The initial value for alphaDash.
*/
AlphaDropout(const double ratio = 0.5, double alphaDash = -alpha * lambda);

This comment has been minimized.

@zoq

zoq Apr 4, 2018

Member

What about using const alphaDash and let's use a more descriptive comment for the parameter something like: dropout scaling parameter.

This comment has been minimized.

@sourabhvarshney111

sourabhvarshney111 Apr 4, 2018

Contributor

Done : )

@zoq

zoq approved these changes Apr 4, 2018

If you could handle the issue I pointed out, I think this is ready and I'll go ahead and merge this in 3 days to leave time for any other comments.

{
ar & BOOST_SERIALIZATION_NVP(ratio);
ar & BOOST_SERIALIZATION_NVP(a);
ar & BOOST_SERIALIZATION_NVP(b);

This comment has been minimized.

@zoq

zoq Apr 4, 2018

Member

We should add alphaDash as well.

This comment has been minimized.

@sourabhvarshney111

sourabhvarshney111 Apr 4, 2018

Contributor

Done : )

@sourabhvarshney111

This comment has been minimized.

Contributor

sourabhvarshney111 commented Apr 4, 2018

@zoq
Please close PR #1278 after merging this.

@rcurtin

rcurtin approved these changes Apr 6, 2018

Looks good to me, thanks for finishing this one. 👍

*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat

This comment has been minimized.

@rcurtin

rcurtin Apr 6, 2018

Member

This is a really minor issue, but if we want to put the typenames on separate lines here, they should be indented only four spaces, not eight. :)

/**
* The alpha - dropout layer is a regularizer that randomly with probability 'ratio'
* sets input values to alphaDash.

This comment has been minimized.

@rcurtin

rcurtin Apr 6, 2018

Member

I think that alpha-dropout is meant to be used with the SELU activation function; should we add a note of that in the documentation? Correct me if I am wrong.

This comment has been minimized.

@sourabhvarshney111

sourabhvarshney111 Apr 6, 2018

Contributor

Agreed. Clearly explained in this https://arxiv.org/pdf/1706.02515.pdf paper, alpha dropout is mainly used for Selu activation function for cases where successive layers don't have same mean and variance to avoid vanishing and exploding gradients. So, I would add a note that it is meant to be used mostly for Selu activation function.

@zoq zoq merged commit bb503e2 into mlpack:master Apr 7, 2018

5 checks passed

Memory Checks
Details
Static Code Analysis Checks Build finished.
Details
Style Checks Build finished.
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@zoq

This comment has been minimized.

Member

zoq commented Apr 7, 2018

Thanks @sourabhvarshney111 @dakshitagrawal97 for the contribution.

@zoq zoq referenced this pull request Apr 7, 2018

Merged

Add Alpha Dropout Layer #1278

@dakshitagrawal97

This comment has been minimized.

Contributor

dakshitagrawal97 commented Apr 7, 2018

Thank you @zoq @sourabhvarshney111 for seeing this to completion!

Happy to contribute, and will do so in free time. :)

@sourabhvarshney111

This comment has been minimized.

Contributor

sourabhvarshney111 commented Apr 8, 2018

We will be happy to see you contribute in open source : ) @dakshitagrawal97

@sourabhvarshney111 sourabhvarshney111 deleted the sourabhvarshney111:alphaDropout branch Apr 8, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment