EvolutionaryMLP

Neural architecture search for MLPs using a genetic algorithm. The GA evolves the width, depth and per-layer activation function of a plain MLP, scoring each candidate by validation accuracy after a short training run on MNIST or EMNIST.

The idea is small and self-contained — no Optuna, no Ray, no checkpoints. The goal is to see evolution work: random shallow networks at generation 0, specialized architectures by generation ~10.

Concept

Each individual is a Seed — a list of hidden-layer sizes (powers of two, between 2 and 2048) and a matching list of activation functions chosen from {ReLU, GELU, Tanh, Sigmoid}. A seed builds an MLP by stacking Linear → activation blocks and adding a final linear classifier.

Each generation:

1. Evaluate — every seed is trained for a budget-controlled number of epochs and scored on the validation set (optionally penalized by parameter count).
2. Select — the top elite_size seeds survive untouched; the rest of the next generation is sampled via softmax-weighted selection over fitness.
3. Mutate — each child can have a layer width doubled/halved, an activation swapped, a layer inserted, or a layer removed.

A budget schedule gives early generations cheap evaluation (1 epoch, large population) and later generations more accurate evaluation (10 epochs, small population) — see default_budget in main.py.

Results

On MNIST, with the default settings, the GA reliably converges to wide 1–2 hidden-layer MLPs with GELU as the first activation. Two reproducible runs:

Seed	Generations	Best architecture	Params	Test acc
42	12	256(GELU)-64(TANH)	218k	98.04%
7	15	512(GELU)-128(TANH)	469k	97.40%

Longer runs reach >98% with wider single-hidden-layer architectures (1024(RELU) / 1024(GELU)). With the --penalty-mode divide and --shrink-bias flags the GA instead searches the small end of the Pareto front, finding e.g. 19k-param architectures at 96.4% test acc. See results.md for the full per-generation history, the Pareto front across all runs, and results/ for the raw JSONL logs.

Quickstart

# 1. Install (Python 3.10+, CUDA optional)
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. Run the GA on MNIST (downloads ~10MB on first run)
python main.py --generations 15

# 3. Or on EMNIST balanced split
python main.py --dataset emnist --emnist-split balanced --generations 15

# 4. Train one fixed architecture (no GA) as a baseline
python baseline.py

The first run downloads MNIST/EMNIST into ./data/ automatically. Per-generation logs and the final summary are written to ./results/.

Useful flags

Flag	Default	Meaning
--generations	200	Number of GA generations
--pop-size	20	Initial population size (then shrinks per budget)
--elite-size	2	Top-k seeds carried over untouched
--temperature	0.1	Softmax-selection temperature (lower = greedier)
--p-size, --p-act, --p-add, --p-remove	0.5/0.3/0.1/0.1	Per-mutation probabilities
--penalty-mode	none	none / subtract / divide / exp — penalize fitness by param count
--lam	1e-7	Strength of the parameter penalty
--shrink-bias	off	Bias mutate_layer_size toward halving (structural pressure toward small)
--seed	None	RNG seed for reproducibility

File structure

seed.py          # Seed = (layer sizes, activations) + mutation operators
mlp.py           # MLP built from a Seed
data.py          # MNIST / EMNIST DataLoaders
training.py      # train_one_epoch + evaluate_accuracy
ga.py            # fitness, selection (softmax + elitism), next_generation
main.py          # CLI entry point — runs the full GA + final test-set eval
baseline.py      # Trains one fixed architecture, for sanity-checking
plot_results.py  # Per-run plots: fitness, activation share, width
pareto.py        # Cross-run plot: params vs validation accuracy

Algorithm details

Fitness. Train a fresh MLP from the seed for epochs epochs with Adam, then return validation accuracy. With --penalty-mode divide, fitness becomes val_acc / (1 + λ · params), which pushes the GA toward smaller models.

Selection. Top-elite_size carried over. The rest is sampled with softmax(scores / temperature) — at T = 0.1 selection is sharp; raising T flattens it toward uniform.

Mutations. Per child, each mutation is rolled independently:

• mutate_layer_size — pick a hidden layer, double or halve its width (clamped to [min_units, max_units], kept as a power of two).
• mutate_activation — pick a layer, swap its activation for a different one.
• add_layer — insert a new layer at a random position. The new width is chosen from {0.5×, 1×, 2×} of a neighbor (snapped to the nearest power of two).
• remove_layer — drop a random layer (only if more than min_layers remain).

Budget schedule. Early generations evaluate many seeds cheaply; later generations evaluate fewer seeds more accurately. The default in default_budget is:

Gens	Epochs / seed	Population
0–2	1	20
3–5	2	14
6–8	3	10
9–14	5	6
15+	10	5

License

MIT — see LICENSE.