Skip to content

Releases

ngram-learn RC3

ngram-learn RC3 Pre-release
Pre-release

Choose a tag to compare

@rhys101 rhys101 released this 11 Oct 07:47

ngram-learn RC3

RC2 plus: learning that requests do not notice, and facts about the user that stay the user's. Same branch
(axonzeta/TensorFold ngram-learn, tag ngram-learn-rc3); start at docs/recipe.md.

What changed since RC2

  • Learning is invisible to requests. A grad op's start is now a step per lesson (each lesson's capture prefill),
    and serve.sh serves two streams with LEARN=1, so a request runs beside learnd's background chats instead of
    behind them. Under chat traffic every ~10 s while learning: time to first token p50 82 ms, p99 0.25 s, no request
    past 0.5 s, the same as without learning (RC2: p99 1.24 s, 5% of requests past 0.5 s).
  • Facts about the user stay the user's. They train only rows whose n-gram holds "my"/"I" (so "your dog's name"
    is untouched), from 4–6 key phrases the model lists (the plain question forms included), with no training question
    that holds the answer. They are committed when every training question comes back with nothing damaged; held-out
    questions written freely may hold none of the keys and are reported. Through the server, from "A few things about
    me: my dog is called Biscuit, I live in Llandudno, and my favourite colour is teal.": all three facts learn; "Where
    do I live?" → "You live in Llandudno."; five of nine new phrasings answer; "What is my cat called?" and the other
    controls are unchanged.

Results (unchanged from RC2 unless noted)

Recall Held-out Neighbours unchanged s / fact
20 chat facts 0.87 0.78 0.95 167 (RC1 build)
17 named facts 0.98 0.91 0.96 —
2 facts, current build 1.00 1.00 1.00 71–77
3 facts about the user, through the server (new) 1.00 0.67 1.00 ~190

1,000 sequential edits keep earlier facts at 0.995; MMLU 86.0% with and without the 1,000-edit overlay
(experiments/07-scale).

Gates: zig build test; tools/batch_check.py (BATCH-OK); tools/rc_gate.py (GATE-OK); the recipe from a clean
clone.

Known limits

  • A fact is found only by questions holding one of its keys ("Which colour do I like most?" misses a fact keyed on
    "my favourite colour").
  • Two named facts on one key compete.
  • Under traffic heavier than a request every ~10 s, learning slows (it runs in the gaps).
  • One rank; overlays live in the server's memory and learnd pushes them again after a restart.

ngram-learn RC2

ngram-learn RC2 Pre-release
Pre-release

Choose a tag to compare

@rhys101 rhys101 released this 11 Oct 05:27

ngram-learn RC2

RC1 plus: facts about the user, twice the speed, learning while serving without delaying requests, and the scale
evaluation. Same branch (axonzeta/TensorFold ngram-learn, tag ngram-learn-rc2); start at
docs/recipe.md.

What changed since RC1

  • Facts about the user learn ("My dog is called Biscuit.", "I live in Llandudno.", "My favourite colour is
    teal."). In RC1 none did. learnd now gives a fact in the first person its own lesson: the model lists the phrases
    people ask about it by ("my dog", "my dog's name", "my puppy"), the training questions use them, every phrase is a
    key, and the rows train without the frequency penalty (a personal tag's overlay reaches only that tag's
    requests). Through the server: "What did I name my puppy?" → "You named your puppy Biscuit."
    (experiments/06-user-facts).
  • A fact in ~75 s (RC1: 124–187 s). The delta rule's backward no longer sums across columns with atomics at every
    row (70% of a training pass); the pass is 1.6 s instead of 4.1 s. Greedy checks run only once the loss is low
    (experiments/08-speed).
  • Learning while serving. The training pass is resumable layer by layer: the server advances it one layer an
    idle moment, so a request waits for a layer (~35 ms), not a pass. learnd's own chats go at background priority.
    Under chat traffic every ~10 s while learning: time to first token p50 88 ms (79 ms without learning), p90 0.28 s
    (RC1-style back-to-back learning: 0.70 s and 1.63 s).
  • Scale (experiments/07-scale): 1,000 sequential CounterFact edits in the engine keep earlier facts at 0.995
    with the control facts unchanged (51,005 rows); MMLU 86.0% with and without the 1,000-edit overlay (14 of 570
    answers change, 5 to right and 5 to wrong).
  • Fixes: the probe applied an overlay loaded at start to no scoring op (TF_NGRAM_OVERLAY with score and
    worker); learnd chose keys from held-out questions and could pick the answer as a key; metadata reaches the
    engine (RC1).
  • Tried and dropped, with the evidence: stored targets with a joint solve (EngramEdit-style; experiments/05-v2:
    v1 keeps earlier facts better), packed-int4 expert products (08-speed: no faster once served capture removed the
    training forward).

Results

Recall Held-out Neighbours unchanged s / fact
20 chat facts, RC1 0.87 0.78 0.95 167
17 named facts, RC1 0.98 0.91 0.96 —
3 user facts, multi-key (chat_edit) 0.89 0.83 1.00 —
2 facts, RC2 speed 1.00 1.00 1.00 71–77

Gates: zig build test; tools/batch_check.py (BATCH-OK); tools/rc_gate.py against the server (GATE-OK: tags
change only tagged replies, drafted = plain with and without a tag); train_op_check.sh (every training kernel
against fp64 PyTorch); the recipe from a clean clone.

Known limits

  • A fact about the user is found only by questions that contain one of its key phrases ("Where do I live?" misses
    a fact keyed on "my town" and "where I live").
  • Two facts on one key compete (the walkthrough's second Kestrel fact).
  • About 5% of requests still wait up to ~1.3 s while learning runs (the start of a grad op, a greedy check, or a
    background chat on the one stream).
  • One rank; overlays live in the server's memory and learnd pushes them again after a restart.

ngram-learn RC1

ngram-learn RC1 Pre-release
Pre-release

Choose a tag to compare

@rhys101 rhys101 released this 10 Oct 20:11

ngram-learn RC1

Facts learned into Qwen3.8 Flash Next's n-gram memory while it serves, on one DGX Spark: the model is taught a fact
in about three minutes, with no second copy of its weights and no dense weight change, and requests read it back
when they ask for its tag.

Branch: axonzeta/TensorFold ngram-learn (tag ngram-learn-rc1), on TensorFold zig-flashnext db28187
with MiaAI-Lab's Flash Next CUDA patches. Start here: docs/recipe.md, then
docs/walkthrough.md.

What's in it

  • An overlay of n-gram table rows in the engine's n-gram gather, per sequence: a fact's change is ~25–55 rows of
    160 values, removable exactly.
  • A training pass inside the CUDA engine on the served prefill's own activations and MoE routes, back through
    every layer to the n-gram rows (fp32 gradients; every kernel checked against fp64 PyTorch); several questions in
    one pass.
  • Learning in the server (LEARN=1): learn ops queued on POST /v1/ngram/op and run between requests;
    overlays by tag, which a request picks with "metadata": {"tag": "..."}.
  • learnd, the learner beside the server: facts from text, model-written lessons, training, checks through the
    server's chat (recall, held-out questions, twins and keep prompts unchanged), rounds, commit, per-fact delete;
    /v1/slide/learn as Living Weights streams it.

Results (experiments/04-chat-facts)

20 fictional facts taught as chat turns, all on one Spark, the RC build:

Recall Held-out phrasing Neighbours unchanged s / fact
all 20 facts 0.87 0.78 0.95 167
the 17 about named things (projects, people, organisations, products, places, identifiers) 0.98 0.91 0.96 —

Learned one after another into one overlay, all 20 still score 0.85 / 0.70 / 0.95 together at the end (named
things 0.96 / 0.82 / 0.96).

Through the server, tagged requests change and untagged ones do not; drafted replies equal undrafted ones, with and
without a tag (tools/rc_gate.py). zig build test passes.

Known limits

  • Facts about the user keyed on common words ("my dog", "I live", "my favourite colour") do not generalise:
    their rows are shared with every sentence about any dog. Named things work.
  • One-token keys ("Kestrel", "Zorn") leak into other contexts with that token, and two facts on the same key
    compete (the walkthrough's second Kestrel fact).
  • One rank, and a learn op holds the engine for one op at a time (~5 s): a request arriving mid-op waits for it.
  • Overlays live in the server's memory; learnd keeps them on disk and pushes them again at its start or on
    POST /v1/slide/sync.

Fixed on the way

  • Random GPU faults (Xid 31) while training: training uploads raced the non-blocking training stream (a pageable
    cuMemcpyHtoD returns before its DMA lands). Uploads are now stream-ordered, and upstream's 2f8284e is
    backported. ~1,500 passes on three Sparks after the fix without a fault.
  • metadata reached the engine only after the chat route's field whitelist learned it.