Skip to content

v3.8.4

Choose a tag to compare

@github-actions github-actions released this 06 Oct 13:57
· 97 commits to main since this release
v3.8.4
4735de3

The prompt cache, on the band and in the pane.

Added

  • The band and the pane show the prompt cache. On a mods host the module
    reads the token usage the host reports for every request, the main loop's and
    each agent's, and keeps a hit rate per loop: the share of what it sent that
    the cache served. A cache break is a request that read back less than the same
    loop's previous request cached, on the same model with no compaction, /clear
    or rewind in between, and a cache that expired between requests counts. The
    band counts the session's breaks after the status once there is one. The pane
    gets a CACHE section after SPEND with the main loop's rate, its last request's
    rate, its breaks and the tokens the last one lost, and the breaks in agents,
    and each running agent's row ends in its own rate and breaks. The figures are
    this session's, held in memory and gone on a module reload, and none of them
    reach the trace yet (GH-309).