Vanishing gradient notebook

[karanchahal]

Poornima and I have compared an LSTM and RNN and visualized the gradients with respect to the input. We see that the gradients for the RNN are much smaller compared to the LSTM.

We are able to train MNIST for a large input sequence with an LSTM and failed to do so with an RNN.

Hope this is useful. If we need to make any chances, please let us know !

[review-notebook-app]

Check out this pull request on 

You'll be able to see Jupyter notebook diff and discuss changes. Powered by ReviewNB.

[Atcold]

Hmm... Let's discuss on the Slack channel before submitting new notebooks.

Anyhow, can you please get this to play with the existing notebooks on RNN and LSTM?
I'm referring to https://github.com/Atcold/pytorch-Deep-Learning/blob/master/08-seq_classification.ipynb, which I will be teaching tomorrow.
It would be helpful not having to go through another notebook in class, since it adds overhead and people will get confused quickly.

[Atcold]

I can try to help out merging this tomorrow.
Perhaps could you drop by my office?

[karanchahal]

Sure, I'll drop by your office today.

[Atcold]

Any update on this, @karanchahal?

[karanchahal]

@Atcold no, I have not made any progress on this as yet.

[Atcold]

Can you give it a try, when you get a chance?
Thaaaanks!

[Atcold]

Closing for the moment.