Skip to content

Project Maya v1.0.8

Choose a tag to compare

@mw00 mw00 released this 08 Oct 08:06
· 497 commits to main since this release

--bench now warms up before it measures prefill (reading the prompt), so speed reports compare fairly.

Update: git pull, then ./setup.sh (Windows: START-MAYA.bat). This release changes only the installer script, so nothing recompiles.

What's new

  • --bench: one unmeasured prompt is read before the 2k and 8k prefill measurements. The 2k figure used to be the first prompt after start-up, with the caches still cold, and could read lower than the 8k one (152 vs 238 tokens/s on a Titan RTX pair). Reported by @dummerjindabin (#8).
  • --bench and --report: a request that only read a prompt shows just its prefill time in the engine lines, not a meaningless decode (writing the answer) speed (it read "96,000 tokens/s").