You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(infer): a decoding engine on top of any windowed language model
TextEngine wraps a Module with the parts every chat loop rewrites: keeping the
window, turning logits into a token, batching, and persisting weights.
What it adds over writing the loop by hand:
- scoresBatch runs N contexts through one forward, so decoding N conversations
costs roughly one conversation's worth of native crossings per step instead of
N. generateBatch is the same idea applied to greedy decoding.
- sample does temperature softmax over the top-k candidates with a seeded RNG,
which is what keeps a small model from emitting a rare character every step.
topK=1 degenerates to greedy, and the test asserts exactly that.
- save/load store weights by name next to the vocabulary and window, so loading
into a differently shaped module fails with the offending name instead of
silently reshaping.
Where the work happens is deliberate: every multiply and activation is executed
by the Rust backend, and the loop stays in Java because the module defines which
ops run. Recording a replayable native program would mean reimplementing the
architecture in Rust and keeping the two in sync; the batching is the part that
actually removes boundary crossings, so that is what got built.
Tests (7): batched scoring equals single-row scoring, greedy is deterministic and
truncates to the tail, batched generation matches sequential, sampling replays
under a seed and honours top-k, weights round-trip bit-exactly, bad shapes are
rejected, and a model trained on five characters decodes them back exactly.
Version 3.7.0. Docs: 19 pages, 243 public types, 1563 of 1563 members documented,
0 placeholders, 0 broken links. verify.sh green.