You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(infer): stream tokens as they are produced
TextEngine could only hand back a finished array, so a caller that wanted to show
progress had to fake it by chunking the result. stream and streamSampled call a
sink per token instead.
The decoding rules are untouched: stream produces exactly what generate would
have produced, streamSampled exactly what sample would have produced under the
same seed. The test asserts both, token by token, because "streaming" that
quietly picks different tokens is a common way to ship a subtle bug.
Measured on the demo model (1.04 M parameters, window 16, one CPU core):
3380 tokens/second without display delay.
Version 3.8.0. Docs: 1565 of 1565 members documented, 0 placeholders. verify.sh
green with 9 suite groups.