Ava 0.3: numbers, long books and sleep

Ava is a brain-like language model in pure Rust: spiking neurons, fast synapses that rewrite themselves as she reads, and sparse codes. Her working memory of a text is a fixed 408 KB, however long the text is. She is a tiny research model, trained on one home GPU.

What's new

Ava 0.3: one model that reads, counts, reads whole books, answers questions and sleeps. 20.0M weights, grown from v39's number-reading Ava on 108M more tokens (books, number practice, answers, recall practice), then put to sleep and taught by reward. Each test was pre-registered and run once on fresh data:

• Sleep: 23.8% → 96.2%. She reads facts about 150 new people, sleeps on them for 25 seconds, and answers questions about them (chance 25%).
• Synaptic consolidation. Before she sleeps, each weight's importance to her reading is measured, and important weights are held in place. Sleep then cost her reading +0.016 bits per byte instead of +0.046.
• Numbers: +18 points (60.5% vs 42.4%) on a fresh number test. She sees how long a number is before reading its digits; comparing numbers of different lengths went from 46.5% to 90.5%.
• Long books: her memory helps across 16,384 tokens of unseen books (0.033 bits per byte better than a memory cut every 256 tokens).
• Questions: 40.7% vs 33.0% on 1,200 held-out science questions (SciQ 56.5%).

What failed, reported as such:

• An attention read added so she could look things up went unused.
• Reward learning added nothing measurable.
• Her reading is 1.511 bits per byte, just past the pre-registered limit of 1.497 and a little below v0.2's Ava (1.487).

Downloads

File	Model	Save as
ava-0.3.safetensors	Ava 0.3	release/ava-0.3/model.safetensors

Check it with sha256sum -c SHA256SUMS in release/ava-0.3. Run her with --glance (see the model card, which also shows how to teach her your own facts in her sleep). The v0.2 models are still on the v0.2.0 release page.

Every number comes from a test written down before it ran, and every earlier look at a test set is disclosed.

Licence: PolyForm Noncommercial 1.0.0. Free for research, study, teaching and other non-commercial use; commercial use needs a separate licence.