Re-thinking Generalization: CIFAR-10 Experiments

This project contains a PyTorch reproduction of several experiments inspired by the paper:

Understanding Deep Learning Requires Rethinking Generalization

The notebooks explore how deep networks can fit random labels, noisy inputs, and structured perturbations while still achieving surprisingly strong or weak generalization under different training conditions. The goal is to reproduce the key observations that motivate the paper: training performance can be near-perfect even when the learned mapping is not truly generalizing in the usual sense.

Project structure

• Expt1.ipynb — ResNet-style experiment with random labels on CIFAR-10.
• Expt2_3.ipynb — Label corruption experiments across multiple corruption rates.
• Expt4.ipynb — Fixed-pixel-shuffle dataset variant to study structured input corruption.
• Expt5.ipynb — Gaussian-noise perturbed CIFAR-10 variant.
• data/ — CIFAR-10 dataset directory used by the notebooks.
• 1611.03530v2.pdf — Original paper PDF.
• Understanding Deep Learning Requires Rethinking Generalization Reproduction Summary.pdf — Summary notes for the project.

Dataset

The experiments use the CIFAR-10 dataset from torchvision.datasets.CIFAR10.

The dataset is expected to live under:

./data

The first time a notebook runs, it will download the dataset automatically if it is missing.

Requirements

This project is built with Python and PyTorch.

Recommended setup:

python -m venv .venv
source .venv/bin/activate    # Linux/macOS
# or .venv\Scripts\activate  # Windows

pip install torch torchvision numpy matplotlib tqdm jupyter

Running the notebooks

Open the notebooks in Jupyter or VS Code Notebook mode:

jupyter notebook

Then run the cells in order for the experiment you want to reproduce.

Experiment overview

Expt1: random labels

This notebook trains a ResNet-18-like architecture on CIFAR-10 while replacing the labels with random class assignments. It is designed to show that deep networks can memorize even when the training labels contain almost no real signal.

Expt2_3: controlled label corruption

This notebook repeatedly trains a CNN on CIFAR-10 with varying label corruption rates such as 0%, 20%, 40%, 60%, 80%, and 100%. The results illustrate how training accuracy can remain very high even when a significant fraction of labels are corrupted, while test accuracy degrades as the corruption increases.

Expt4: fixed-pixel-shuffle

This experiment shuffles pixel positions using a fixed permutation to create a structured but semantically broken input space. The notebook tests whether a network can still learn or generalize when the input structure has been deliberately altered.

Expt5: Gaussian noise dataset

This notebook creates a CIFAR-10 variant where images are replaced by Gaussian-noise reconstructions based on the dataset statistics. This is used to probe how models behave when the input distribution itself is distorted.

Typical training setup

Most notebooks use a simple CNN or modified ResNet-like architecture with:

• CIFAR-10 data loader
• cross-entropy loss
• SGD or Adam optimizer
• learning-rate scheduling
• evaluation on the held-out test set

Typical configuration values include:

• batch size: 128 or 256
• epochs: 50 to 150 depending on the experiment
• device: CUDA if available, otherwise CPU

Notes

• This repository is intended for research and educational reproduction rather than production code.
• Results can vary slightly by machine, random seed, and library version.
• The notebooks are exploratory and are best read in sequence for the corresponding study.

References

• Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2017). Understanding Deep Learning Requires Rethinking Generalization.
• Original paper PDF included in this repository as 1611.03530v2.pdf.

License

This repository is for academic and research use. Please check the paper and repository context before reusing the code in a publication or larger project.