Ava 0.2: a stronger brain that can answer questions

Ava is a brain-like language model in pure Rust: spiking neurons, fast synapses that rewrite themselves as she reads, and sparse codes. She has no attention, and her whole memory of a text is a fixed 408 KB, however long the text is. She is a tiny research model, trained on one home GPU.

What's new

Ava, a much stronger brain. 19.4M weights, trained on 387M tokens in 44 minutes on one RTX 4070.

• 1.487 bits per byte on held-out text (v0.1: 1.872; lower is better)
• 76.1% on BLiMP grammar pairs (chance 50%)

Ava-QA, taught to answer questions. 95 seconds of teaching on top of Ava. While she learned to answer, she also re-read her ordinary text, as the brain replays old memories while it learns.

• SciQ science questions: 41.1% → 56.4% right (66.2% with the passage shown; chance 25%)
• She may say "I don't know"; then 64% of the answers she gives are right
• Replay cut her forgetting of plain reading by about three quarters

A faster trainer. Hand-written training steps and fused CUDA kernels, bf16 matrix multiplies, and a chunk form of her memory that trains long texts 1.8× faster.

Three pre-registered tests against a transformer (v34–v36). Her design learns more per weight at small sizes, but a transformer gains more from size: at 17.8M weights she is 0.038 BPB behind its fitted line. Making her curve steeper is the main open problem.

Downloads

File	Model	Save as
ava.safetensors	Ava	release/ava/model.safetensors
ava-qa.safetensors	Ava-QA	release/ava-qa/model.safetensors

Check them with sha256sum -c SHA256SUMS in each folder. The model cards (Ava, Ava-QA) have the exact commands, seeds, checksums and unselected samples. The README's Quick start shows how to build, run, ask and train.

Every number comes from a test written down before it ran, and every earlier look at a test set is disclosed.

Licence: PolyForm Noncommercial 1.0.0. Free for research, study, teaching and other non-commercial use; commercial use needs a separate licence.