Skip to content

v3.7.0

Choose a tag to compare

@github-actions github-actions released this 30 Oct 13:29
· 8 commits to main since this release
feat(infer): a decoding engine on top of any windowed language model

TextEngine wraps a Module with the parts every chat loop rewrites: keeping the
window, turning logits into a token, batching, and persisting weights.

What it adds over writing the loop by hand:

- scoresBatch runs N contexts through one forward, so decoding N conversations
  costs roughly one conversation's worth of native crossings per step instead of
  N. generateBatch is the same idea applied to greedy decoding.
- sample does temperature softmax over the top-k candidates with a seeded RNG,
  which is what keeps a small model from emitting a rare character every step.
  topK=1 degenerates to greedy, and the test asserts exactly that.
- save/load store weights by name next to the vocabulary and window, so loading
  into a differently shaped module fails with the offending name instead of
  silently reshaping.

Where the work happens is deliberate: every multiply and activation is executed
by the Rust backend, and the loop stays in Java because the module defines which
ops run. Recording a replayable native program would mean reimplementing the
architecture in Rust and keeping the two in sync; the batching is the part that
actually removes boundary crossings, so that is what got built.

Tests (7): batched scoring equals single-row scoring, greedy is deterministic and
truncates to the tail, batched generation matches sequential, sampling replays
under a seed and honours top-k, weights round-trip bit-exactly, bad shapes are
rejected, and a model trained on five characters decodes them back exactly.

Version 3.7.0. Docs: 19 pages, 243 public types, 1563 of 1563 members documented,
0 placeholders, 0 broken links. verify.sh green.