Ilya's Reading List

This is a compilation of my notes & code on the Ilya Sutskever's top 30 reading list.

The mini-project is structured as follows:

• resources/ contains the original papers and articles for reference.
• NOTES.md contains my detailed notes on each paper or resource in the reading list.
• notebooks/ contains Jupyter notebooks with my notes and code snippets for each paper or resource in the reading list.
• rag-chatbot/ contains my implementation of a Retrieval-Augmented Generation (RAG) chatbot using the concepts from the reading list.

The list should be from 2020 and it was mentioned by John Carmack in a 2023 interview:

Exclusive Q&A: John Carmack's 'Different Path' to Artificial General Intelligence

The List

#	Source	Type	Topics	Other Related Resources	My Notes	My Code
1	The Annotated Transformer - Sasha Rush et al., 2018
2	The First Law of Complexodynamics - Scott Aaronson
3	The Unreasonable Effectiveness of Recurrent Neural Networks - Andrej Karpathy, 2015
4	Understanding LSTM Networks - Christopher Olah, 2015
5	Recurrent Neural Network Regularization - Wojciech Zaremba et al., 2014
6	Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - Geoffrey E. Hinton & Drew van Camp
7	Pointer Networks - Oriol Vinyals et al., 2015
8	ImageNet Classification with Deep Convolutional Neural Networks - Alex Krizhevsky et al., 2012
9	Order Matters: Sequence to Sequence for Sets - Oriol Vinyals et al., 2015
10	GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism - Yanping Huang et al., 2018
11	Deep Residual Learning for Image Recognition - Kaiming He et al., 2015
12	Multi-Scale Context Aggregation by Dilated Convolutions - Fisher Yu & Vladlen Koltun, 2015
13	Neural Message Passing for Quantum Chemistry - Justin Gilmer et al., 2017
14	Attention Is All You Need - Ashish Vaswani et al., 2017
15	Neural Machine Translation by Jointly Learning to Align and Translate - Dzmitry Bahdanau et al., 2014
16	Identity Mappings in Deep Residual Networks - Kaiming He et al., 2016
17	A Simple Neural Network Module for Relational Reasoning - Adam Santoro et al., 2017
18	Variational Lossy Autoencoder - Xi Chen et al., 2016
19	Relational Recurrent Neural Networks - Adam Santoro et al., 2018
20	Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - Scott Aaronson et al., 2017
21	Neural Turing Machines - Alex Graves et al., 2014
22	Deep Speech 2: End-to-End Speech Recognition in English and Mandarin - Dario Amodei et al., 2015
23	Scaling Laws for Neural Language Models - Jared Kaplan et al., 2020
24	A Tutorial Introduction to the Minimum Description Length Principle - Peter Grünwald, 2018
25	Machine Super Intelligence - Shane Legg
26	Kolmogorov Complexity and Algorithmic Randomness - A. Shen, V. A. Uspensky, N. Vereshchagin
27	CS231n: Convolutional Neural Networks for Visual Recognition

Setup

In order to run the code provided in this repository, you will need to set up a Python environment with the required dependencies.

In the following, I provide a recipe to set up a conda environment with the necessary packages.

# Create the necessary Python environment
# NOTE: specific folders might require their own environment
# and have their own requirements.txt
conda env create -f conda.yaml
conda activate ilya

# If you have CUDA, install CUDA support with the propper CUDA version, e.g. v12.1 (doesn't need to match 13.0)
pip install torch torchvision torchaudio torchtext --index-url https://download.pytorch.org/whl/cu121
# OTHERWISE, install CPU version -- BUT many examples won't work!
pip install torch torchvision torchaudio torchtext
# Compile rest of dependencies and install them
pip-compile requirements.in
pip install -r requirements.txt

# If we need a new dependency,
# add it to requirements.in 
# (WATCH OUT: try to follow alphabetical order)
# And then:
pip-compile requirements.in
pip install -r requirements.txt

Authorship

Mikel Sagardia, 2026.
Feel free to use it with propper attribution.
No guarantees.