Skip to content

v3.8.0

Choose a tag to compare

@github-actions github-actions released this 30 Oct 13:42
· 7 commits to main since this release
feat(infer): stream tokens as they are produced

TextEngine could only hand back a finished array, so a caller that wanted to show
progress had to fake it by chunking the result. stream and streamSampled call a
sink per token instead.

The decoding rules are untouched: stream produces exactly what generate would
have produced, streamSampled exactly what sample would have produced under the
same seed. The test asserts both, token by token, because "streaming" that
quietly picks different tokens is a common way to ship a subtle bug.

Measured on the demo model (1.04 M parameters, window 16, one CPU core):
3380 tokens/second without display delay.

Version 3.8.0. Docs: 1565 of 1565 members documented, 0 placeholders. verify.sh
green with 9 suite groups.