Noise: Speedrun for pretraining without backpropagation

Noise is a pretraining speedrun for learning algorithms that never run a backward pass. We think replacing backprop with a much better credit assignment algorithm is inevitable. Deep learning's architectures, optimizers and hardware have co-evolved with backprop, but as compute grows, the bitter lesson suggests that more general, brute-force learning algorithms based on search will win. Dust is a big step forward: it rivals backprop under SGD and is 1,000 to 10,000x more efficient than EGGROLL, the state-of-the-art weight-space ES, but still trails backprop under Adam. These methods have also had a small fraction of the research that went into backprop. We made Noise to close the gap: better estimators, and optimizers co-designed for them instead of for backprop gradients.

The model, the data and the score are fixed: the 37.7M-parameter transformer of Dust, its 10M-token FineWeb training set, and the cross-entropy on its validation set. What changes is the learning algorithm, which may only run the model forward. The architecture is fixed for now, but we will relax this constraint later.

1. Validation loss. The lowest validation loss after one epoch on one 8xH100 node, with no time limit.
2. Speedrun. The shortest training time on one 8xH100 node to reach validation loss 4.2.

• No backpropagation. No autograd (reverse or forward mode) and no hand-written backward pass. Learning signal comes from evaluating the model forward: every error signal is measured against a loss by running the model forward, and may then take at most one step through a nonlinear part of the model. Steps through linear maps don't count, and a signal that has taken a step can't take another, so credit can't be chained back through the network block by block. Entries are judged by the spirit of this rule: a method should still apply to networks with non-differentiable parts.
• Fixed model. Final weights must load into utils/model.py with a strict state-dict match. How training computes the same function, and how it initializes it, is free.
• Fixed data. Train only on data/train.pt (4,880 sequences of 2,048 predictions), for at most one epoch: each training sequence feeds at most one update. Batch size and order are free.
• Score. utils/eval.py on the final weights: mean next-token cross-entropy over the 1,114,112 predictions of data/val.pt. No checkpoint selection on validation loss.
• Timing (speedrun track; reported on the other). From the start of the first update to the end of the last, excluding any validation. An untimed warmup on random tokens may compile kernels, as long as training then starts from a fresh state (the baselines rebuild everything from the seed). Runs use seed 0.

Requires Linux, an NVIDIA GPU with bfloat16 support and uv.

uv sync
uv run utils/prepare_data.py

utils/prepare_data.py streams FineWeb sample-10BT, tokenizes it with the fixed BPE-4096 tokenizer in tokenizer/, and writes data/train.pt and data/val.pt, the train_10M.pt and val.pt of the Dust repo. Both are checked against pinned hashes.

uv run torchrun --standalone --nproc_per_node=8 train/val/run.py         # track 1
uv run torchrun --standalone --nproc_per_node=8 train/speedrun/run.py    # track 2
uv run torchrun --standalone --nproc_per_node=8 train/val/eggroll.py

Each run writes config.json, metrics.jsonl, final.pt and result.json (validation loss and training time) to logs/. uv run utils/eval.py logs/val/final.pt rescores a checkpoint.

The same model, data and epoch trained with backprop, scored the same way:

The goal is to beat backprop by a large margin.

Open a PR that changes its track's training script in place, train/val/run.py or train/speedrun/run.py, with the run's log (config.json, metrics.jsonl and result.json from logs/) in the log/ folder next to it. A record must beat the current one in its track under the rules above; the training script is the record.

@misc{noise_2026,
  title  = {Noise: Speedrun for pretraining without backpropagation},
  author = {G{\"u}lbahar, Serdar and Mandal, Bishwas and Vegesna, Akshay and Dahal, Samip},
  year   = {2026},
  url    = {https://github.com/qlabs-eng/noise}
}

Core contributors: Serdar Gülbahar and Bishwas Mandal.

Released under the MIT License.