"Neural Networks: Zero to Hero" by A. Karpathy

[fxlrnrpt]

About: Following the legendary series "Neural Networks: Zero to Hero" by A. Karpathy
Schedule: Async

What this thread is for:

• Announcing homework tasks and theoretical content;
• Discussing questions related to the course;
• Posting links to complete homeworks for review.

Please, refrain from spamming! It forces others to turn off notifications and, subsequently, leads to a slow and painful death of the community!

[fxlrnrpt]

HW1

Deadline: July 11
Tasks:

• Watch https://youtu.be/VMj-3S1tku0?si=KWRNO_pkYyKe3P-X
• Complete this Google colab

Add 👍 if you are going to join!

[fxlrnrpt]

Due to a popular demand, meetings have been move to Friday. See you on July 11!

[fxlrnrpt]

HW1 #11
Review and comments are welcome!

[fxlrnrpt]

HW2

Deadline: July 18
Lecture:

• Watch https://youtu.be/PaCmpygFfXo?si=heMXDWRA3YUz3Qsi

HW:
E01: train a trigram language model, i.e. take two characters as an input to predict the 3rd one. Feel free to use either counting or a neural net. Evaluate the loss; Did it improve over a bigram model?
E02: split up the dataset randomly into 80% train set, 10% dev set, 10% test set. Train the bigram and trigram models only on the training set. Evaluate them on dev and test splits. What can you see?
E03: use the dev set to tune the strength of smoothing (or regularization) for the trigram model - i.e. try many possibilities and see which one works best based on the dev set loss. What patterns can you see in the train and dev set loss as you tune this strength? Take the best setting of the smoothing and evaluate on the test set once and at the end. How good of a loss do you achieve?
E04: we saw that our 1-hot vectors merely select a row of W, so producing these vectors explicitly feels wasteful. Can you delete our use of F.one_hot in favor of simply indexing into rows of W?
E05: look up and use F.cross_entropy instead. You should achieve the same result. Can you think of why we'd prefer to use F.cross_entropy instead?

Add 👍 if you are going to join!

[fxlrnrpt]

HW2 #12
Review and comments are welcome!

[fxlrnrpt]

HW3

Deadline: July 25
Lecture:

• Watch https://youtu.be/TCH_1BHY58I?si=BXWYh-E2viFdfDjA

HW:
E01: Tune the hyperparameters of the training to beat my best validation loss of 2.2
E02: I was not careful with the intialization of the network in this video. (1) What is the loss you'd get if the predicted probabilities at initialization were perfectly uniform? What loss do we achieve? (2) Can you tune the initialization to get a starting loss that is much more similar to (1)?
E03: Read the Bengio et al 2003 paper, implement and try any idea from the paper. Did it work?

Add 👍 if you are going to join!

[fxlrnrpt]

Meeting moved to Friday at 17:30

[fxlrnrpt]

HW3 #14
Review and comments are welcome!

[fxlrnrpt]

Moving this meeting to async scheduling without a regular call due to low popularity. Still going to post my HW solutions as I go through them. Questions are welcome!

[fxlrnrpt]

Added JAX implmentation for MLP

• https://colab.research.google.com/drive/1MN9gMsZQkrdtD_6EogzWuFYeQTRbh9uq?usp=sharing
• https://github.com/aigoncharov/ml_misfits/blob/main/courses/nn_zero_to_hero/hw3/MLP_jax.ipynb