Deep Learning Study Notes

This repository documents my progress in learning deep learning through first-principles implementation and controlled experiments. The main thread is the hands-on study track in study_free/, which develops each topic from a NumPy implementation into reproducible PyTorch experiments.

MIT 6.S191 provided the starting point and remains an important lecture reference, while CS231n and other materials are used where they support the implementation roadmap. The goal is to understand not only how a model runs, but why it works, how to verify it, and how experimental choices affect the result.

This is an active study repository. Incomplete code and intermediate experiments may remain as part of the learning record.

Current Progress

Stage	Topic	Status
Reference	MIT 6.S191 course materials	Collected
1	Two-layer NumPy MLP and manual backpropagation	Completed
2	Numerical gradient checking	Completed
3	Learning-rate and initialization experiments	Completed
4	PyTorch fundamentals and autograd comparison	Completed
5	Reproducible experiment structure	Completed
6	MIT 6.S191 overview and question log	In progress
7	NumPy convolution and pooling forward/backward	Planned
8	Autograd internals and PyTorch comparison	Planned
9	CNN training experiment	Planned
10	Tiny PINN	Planned

CS231n topics such as optimization, convolution, normalization, and regularization will be studied alongside the implementation roadmap.

Study Notes

Core Implementation Track

study_free/ is the central learning track in this repository. Its notebooks connect theory, implementation, verification, controlled experiments, and written interpretation in one cumulative sequence.

Notebook	Focus
day1_propagation.ipynb	Two-layer NumPy MLP for XOR, manual forward/backward passes, SGD updates, and tensor-shape checks
day2_gradient.ipynb	Numerical gradient checking, central differences, relative error, and diagnosis of ReLU kinks, reduction mismatches, and poor step sizes
day3_SgdLr.ipynb	Controlled experiments on learning rates, weight initialization, train/validation splits, reproducibility, and multi-seed stability
day4_introPyTorch.ipynb	PyTorch tensor operations, nn.Module, an SGD training loop, evaluation mode, and comparison of manual NumPy gradients with autograd
day5_reproducible_experiment.ipynb	Reusable seed control, centralized experiment configuration, isolated run directories, and reproducibility checks

Generated tables and plots from the experiments are stored in study_free/notes/results/.

Course References

MIT-6.S191/ contains lecture-based notes and examples on:

• Perceptrons and multi-output networks
• Activation functions and loss functions
• Gradient descent and backpropagation
• Introductory PyTorch and TensorFlow syntax

Key Results

Manual Backpropagation

A two-layer NumPy MLP learned XOR without autograd:

X (4, 2) -> Linear (2, 8) -> ReLU -> Linear (8, 1) -> Sigmoid

The BCE loss decreased from 0.695377 to 0.001799, and the final predicted classes matched [0, 1, 1, 0].

Gradient Checking

The analytical gradients were compared with central-difference numerical gradients. All parameters passed the target relative error of 1e-5.

Parameter	Maximum relative error
W1	4.827e-07
b1	7.555e-07
W2	1.575e-08
b2	1.820e-09

The experiment also showed that a failed check does not always imply an incorrect backward pass: non-differentiable ReLU points and poorly chosen finite-difference step sizes can produce misleading errors.

Optimization Experiments

Learning-rate and initialization experiments were run under controlled conditions with fixed data, architecture, training duration, and random seeds.

• A learning rate of 0.1 converged reliably in the tested setup, while 1e-5 learned too slowly and 10.0 was unstable.
• Zero initialization failed because hidden units remained symmetric.
• Xavier and He initialization each succeeded on 9/10 tested seeds.
• A fast result from one seed was not sufficient evidence of a robust setup; validation performance and multi-seed success rates were also needed.

PyTorch and Autograd

The NumPy binary classifier was rebuilt with nn.Linear, nn.ReLU, BCEWithLogitsLoss, and torch.optim.SGD.

• After 1,000 epochs, the training loss reached 0.008252 and the validation loss reached 0.006746.
• Training and validation accuracy both reached 100% on the fixed split.
• Manual NumPy gradients matched PyTorch autograd for W1, b1, W2, and b2 with an absolute tolerance of 1e-6.
• The largest reported gradient difference was 6.985e-10 for W1.

Reproducible Experiment Structure

The reusable experiment skeleton now centralizes the seed and hyperparameters, creates an isolated directory for every run, and stores configuration and metrics together.

• Python, NumPy, and PyTorch randomness are reset through one set_seed function.
• Learning rate, hidden dimension, epochs, and batch size are controlled by a single configuration dictionary.
• Two clean executions with the same configuration produced the same final loss (0.0268004) and accuracy (99.5%).
• Timestamped run directories prevent a new experiment from overwriting an earlier result. Generated run directories are intentionally not committed.

Repository Structure

DeepLearning-StudyNote/
├── README.md
├── MIT-6.S191/
│   ├── README.md
│   ├── lecture1/
│   ├── lecture2/
│   └── lecture_slide/
└── study_free/
    ├── README.md
    └── notes/
        ├── day1_propagation.ipynb
        ├── day2_gradient.ipynb
        ├── day3_SgdLr.ipynb
        ├── day4_introPyTorch.ipynb
        ├── day5_reproducible_experiment.ipynb
        └── results/

Study Principles

• Implement core operations and gradients from scratch before relying on higher-level abstractions.
• Change one major experimental variable at a time.
• Record seeds, hyperparameters, metrics, failure cases, and interpretations.
• Keep code executable and document enough context to reproduce key results.

Next Steps

1. Work through MIT 6.S191 as a fast overview and record unresolved concepts.
2. Implement convolution and max-pooling forward/backward passes in NumPy and verify them with numerical gradient checks.
3. Study computation graphs, topological backpropagation, and PyTorch autograd after completing the manual CNN operations.
4. Reuse the experiment structure for a CNN training comparison, then build a small 1D heat-equation PINN.

References

• MIT 6.S191: Introduction to Deep Learning
• Stanford CS231n: Deep Learning for Computer Vision
• CS231n Course Notes

Notes

• Copyright for lecture materials belongs to their respective authors and institutions.
• External code is cited in the relevant file when referenced or modified.