Implement the following

[disooqi]

https://www.almaany.com/ar/dict/ar-ar/%D9%86%D9%90%D8%AD%D9%92%D8%B1%D9%90%D9%8A%D8%B1/
https://www.almaany.com/ar/dict/ar-ar/%D9%86%D8%B7%D8%A7%D8%B3%D9%8A/
https://www.almaany.com/ar/thes/ar-ar/%D9%86%D8%B7%D8%A7%D8%B3%D9%8A/
https://www.maajim.com/dictionary/%D9%86%D8%B7%D8%A7%D8%B3%D9%8A
https://ar.wikipedia.org/wiki/%D8%A7%D8%B3%D8%AA%D9%82%D8%B1%D8%A7%D8%A1_(%D9%85%D9%86%D8%B7%D9%82)

• https://www.analyticsvidhya.com/blog/2018/03/comprehensive-collection-deep-learning-datasets/

backpropagation

• https://blogs.msdn.microsoft.com/uk_faculty_connection/2017/07/04/how-to-implement-the-backpropagation-using-python-and-numpy/
• https://sydney.edu.au/stuserv/documents/maths_learning_centre/compositefunctionrule.pdf
• https://eli.thegreenplace.net/2016/the-chain-rule-of-calculus/
• http://colah.github.io/posts/2015-08-Backprop/

Softmax

• https://www.ics.uci.edu/~pjsadows/notes.pdf

• https://ai.stackexchange.com/questions/6343/how-do-i-implement-softmax-forward-propagation-and-backpropagation-to-replace-si?newreg=955c85b8c8704de1be03d7b566f51405

• https://stats.stackexchange.com/questions/235528/backpropagation-with-softmax-cross-entropy

• https://algorithmsdatascience.quora.com/BackPropagation-a-collection-of-notes-tutorials-demo-and-codes

• https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/

• https://stackoverflow.com/questions/33541930/how-to-implement-the-softmax-derivative-independently-from-any-loss-function

• https://stackoverflow.com/questions/40575841/numpy-calculate-the-derivative-of-the-softmax-function

• https://en.wikipedia.org/wiki/Softmax_function#Artificial_neural_networks

• (?) http://www.wildml.com/2015/09/implementing-a-neural-network-from-scratch/

• https://medium.com/@14prakash/back-propagation-is-very-simple-who-made-it-complicated-97b794c97e5c

• https://medium.com/@aerinykim/how-to-implement-the-softmax-derivative-independently-from-any-loss-function-ae6d44363a9d

• http://www.cs.toronto.edu/~tijmen/csc321/documents/softmax.pdf

• https://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function
  https://peterroelants.github.io/posts/cross-entropy-softmax/

• https://stats.stackexchange.com/questions/235528/backpropagation-with-softmax-cross-entropy

• https://stats.stackexchange.com/questions/79454/softmax-layer-in-a-neural-network

• https://deepnotes.io/softmax-crossentropy

• https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/

CNN

• http://deeplearning.net/tutorial/lenet.html
• https://www.kdnuggets.com/2018/04/derivation-convolutional-neural-network-fully-connected-step-by-step.html#.WtijFNOWxlI.facebook
• https://www.youtube.com/watch?v=BvrWiL2fd0M
• https://pdfs.semanticscholar.org/5d79/11c93ddcb34cac088d99bd0cae9124e5dcd1.pdf
• https://becominghuman.ai/back-propagation-in-convolutional-neural-networks-intuition-and-code-714ef1c38199
• https://stackoverflow.com/questions/43373521/how-to-do-convolution-matrix-operation-in-numpy
• http://machinelearninguru.com/computer_vision/basics/convolution/image_convolution_1.html
• https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/fc_layer.html
• http://www.cs.toronto.edu/~kriz/cifar.html
• http://cs231n.github.io/convolutional-networks/

LSTM

• https://arxiv.org/abs/1503.04069
• http://colah.github.io/posts/2015-08-Understanding-LSTMs/
• http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-a-grulstm-rnn-with-python-and-theano/
• https://deeplearning4j.org/lstm.html
• https://r2rt.com/written-memories-understanding-deriving-and-extending-the-lstm.html
• https://www.quora.com/What-is-the-clearest-presentation-of-backpropagation-through-time-for-LSTMs
• https://stackoverflow.com/questions/41555576/lstm-rnn-backpropagation
• https://towardsdatascience.com/back-to-basics-deriving-back-propagation-on-simple-rnn-lstm-feat-aidan-gomez-c7f286ba973d
• https://medium.com/themlblog/time-series-analysis-using-recurrent-neural-networks-in-tensorflow-2a0478b00be7
• https://skymind.ai/wiki/lstm

RNN

• https://arxiv.org/pdf/1610.02583.pdf
• http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/
• http://peterroelants.github.io/posts/rnn_implementation_part01/
• http://willwolf.io/2016/10/18/recurrent-neural-network-gradients-and-lessons-learned-therein/
• https://www.analyticsvidhya.com/blog/2017/12/introduction-to-recurrent-neural-networks/
• http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/
• https://machinelearningmastery.com/gentle-introduction-backpropagation-time/
• https://github.com/pangolulu/rnn-from-scratch
• https://freecontent.manning.com/recurrent-neural-networks/
• https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/recurrent_neural_networks.html
• https://medium.com/themlblog/time-series-analysis-using-recurrent-neural-networks-in-tensorflow-2a0478b00be7
• https://arxiv.org/pdf/1610.02583v3.pdf

Embeddings

https://towardsdatascience.com/neural-network-embeddings-explained-4d028e6f0526

Attension Mechanism

• https://arxiv.org/pdf/1409.0473.pdf (1st paper)
• https://medium.com/syncedreview/a-brief-overview-of-attention-mechanism-13c578ba9129

Tensorflow

• https://www.tensorflow.org/versions/master/get_started/

Numpy

http://ajcr.net/Basic-guide-to-einsum/
https://stackoverflow.com/questions/26089893/understanding-numpys-einsum
https://machinelearningmastery.com/broadcasting-with-numpy-arrays/
https://docs.scipy.org/doc/numpy-1.15.0/user/basics.broadcasting.html

Ensemble learning

https://en.wikipedia.org/wiki/Ensemble_learning
https://towardsdatascience.com/ensemble-methods-in-machine-learning-what-are-they-and-why-use-them-68ec3f9fef5f

Automatic Differentiation

https://www.youtube.com/watch?v=sq2gPzlrM0g
https://arxiv.org/abs/1502.05767
http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/slides/lec8a.pdf
http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/slides/lec8b.pdf

heatmapping

http://www.heatmapping.org/
http://www.heatmapping.org/tutorial/

Transformer

• https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf (first paper)

• https://www.youtube.com/watch?v=iDulhoQ2pro (first paper)

• https://mchromiak.github.io/articles/2017/Sep/12/Transformer-Attention-is-all-you-need/

• https://jalammar.github.io/illustrated-transformer/

• https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270

• https://arxiv.org/abs/1810.04805

• https://medium.com/@kolloldas/building-the-mighty-transformer-for-sequence-tagging-in-pytorch-part-i-a1815655cd8

• https://arxiv.org/abs/1807.03819

• https://joshvarty.com/2018/02/19/ltfn-6-weight-initialization/

• http://adventuresinmachinelearning.com/weight-initialization-tutorial-tensorflow/

• https://eli.thegreenplace.net/2015/memory-layout-of-multi-dimensional-arrays

• https://eli.thegreenplace.net/2018/elegant-python-code-for-a-markov-chain-text-generator/

• https://hackernoon.com/automated-text-generator-using-markov-chain-de999a41e047

• http://www.emergentmind.com/neural-network

• http://www.cs.toronto.edu/~tijmen/csc321/

https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html?fbclid=IwAR3vrT1pUt5xgzGcZdHPwkOqh-NjsgIKUEEa0JdmmBL66I2mKhA7DrHbGxc