Repository navigation
Ava 0.4
Ava is a small language model built like a brain. She has spiking neurons, a fast memory that she writes and reads
as she goes, and a library she looks things up in. She is written in pure Rust and runs on an ordinary Windows PC.
The brain in this release is Ava 0.4.
What's new since 0.3
A chat app you can download. Ava now comes with a free Windows app. Double-click it and a chat page opens in
your browser. Behind the chat, a 3D brain lights up from her real activity, word by word. Everything runs on your
own computer. Nothing you type leaves it, and it works without internet or a graphics card.
She looks things up before she answers. She has a library of about 297,000 short passages, mostly from the
Simple English Wikipedia. Before each answer she finds the two best passages and reads them. The app shows you
what she looked up.
She can say "I don't know." She was taught to answer from what she reads, to greet you back, and to say
"I don't know" when she is not sure. Part of this she learned from her own mistakes. No rule in the code pastes in
that answer.
She keeps learning while you chat. A small part that works like the cerebellum adjusts her next-word guesses
from what she reads and says. Each chat keeps its own changes, and they slowly fade. Her saved brain never changes.
Delete a chat and she forgets what she learned in it.
A new brain inside. Ava 0.4's blocks have 14 new options, several of them brain-like. Nine of them won small
tests, alone or in pairs. The other five were too slow to test alone, so all 14 were tested together, and together
they beat the old block. For example, her neurons send small signed spikes (from -2 to +2) and their firing
thresholds adapt, and a quarter of the units in her wide layer may be active at once (one in eight before).
She read about half a billion word pieces. 387 million came from a new mix: about half of it is educational web
pages, and the rest is science sentences, the Simple English Wikipedia, speech and dialogue, books and reasoning
lessons. Then she read 108 million more from Ava 0.3's mix, with whole books and number practice.
She is as wide and as deep as Ava 0.3, but has more weights. Her size: d = 384, 8 layers, 6 memory heads of 64; 26,840,893 weights. Ava 0.3's: d = 384, 8 layers,
12 memory heads of 32; 19,962,048 weights.
New in v52. On top of Ava 0.4's 14 block options, she has the brain parts v52 kept (each tested alone first, and kept by a rule written down before the runs): short-term plasticity: each spiking neuron's output tires and recovers with use (depression and facilitation); neuromodulators: a small nucleus reads the first block and sends three signals to the later blocks, like noradrenaline (input gain), acetylcholine (memory write strength) and dopamine (output scale); dendrites: each unit of the wide layer is scaled by a gain from its block's memory read, like the top dendrites of cortical neurons; letters: every word piece also carries a small vector made from its own letters and their places, so she can see spelling; dopamine: from the second block on, the wide layer's output is scaled by how surprised she was by the last word; mood: four slow units follow how surprised she has been lately and scale the input of every later block's neurons; rhythms: every block's memory writes and reads are scaled, per head, by slow and fast rhythms, like theta and gamma waves; a thalamus: a small relay of 16 units reads every block's output and sets graded gates on what each block passes on; a cerebellum: 12,288 granule units make a sparse code of her state and the current word, and a learned read-out corrects her next-word guess; it is the part that keeps learning in the chat app; the homeostasis nudge: the rule that moves each neuron's threshold toward its target firing rate stays on while she grows and is taught. Her training also changed: the spike reset now passes its gradient back (--lif-reset-grad), the fix for the training spikes found in v52.
Fast on a normal processor. She talks at about 425 word pieces a second on a CPU. She uses 8-bit weights
for this. The first build of Ava 0.4 managed 60 to 80.
Steadier training. We found what made the first Ava 0.4's training blow up: a rule that kept pushing some
neurons' firing thresholds higher. With that rule switched off while Ava 0.4b grew and was taught, the blow-ups
stopped.
How she scores
| Ava 0.3 | Ava 0.4 | |
|---|---|---|
| Reading, bits per byte (lower is better) | 1.511 | 1.513 |
| Grammar, BLiMP | 74.1% | 75.9% |
| Science multiple choice (dev set) | 46.8% | 41.7% |
| Simple questions: right / "I don't know" / wrong, of 100 | – | 54 / 31 / 15 |
| Fresh sealed test, run once | – | not run |
| Talking speed in the app (CPU) | – | 425 tokens/s |
Good to know
- She is small. Her answers are often wrong or odd, so do not rely on them.
- On science multiple-choice questions she scores 41.7%, against 46.8% for Ava 0.3.
- She speaks English only. The app is for Windows 10 and 11 (64-bit).
Download and start
- Download Ava-0.4-windows.zip. It holds the app and her brain.
- Unzip the folder.
- Double-click Ava.exe. Your browser opens the chat page.
Windows may warn that the publisher is unknown, because the app is not signed. Choose "More info", then "Run anyway".
She needs about 1 GB of free memory. No graphics card and no internet are needed.
Only want the weights? ava-0.4.safetensors is her brain on its own. It is the same file as model.safetensors in
the zip. SHA256SUMS.txt lists the checksums of every file.
What comes next
Ava's development is coming to an end for now. She needs more GPUs than I have: my single RTX 4070 cannot handle
the scale she needs in order to grow. If you want to use Ava, or would like me to keep working on her and you have
GPU time available, I would be happy to continue her development. Open an issue on the repository to get in touch.
Licence and credits
- Ava (the app and her brain): PolyForm Noncommercial License 1.0.0, non-commercial use only.
- The 3D brain in the app: BodyParts3D, © The Database Center for Life Science (DBCLS), CC BY 4.0.
- Her library: text from the Simple English Wikipedia by Wikipedia's contributors, CC BY-SA 4.0, changed (cut into
short passages). library.tsv is shared under CC BY-SA 4.0 as well.