Skip to content

strixite v0.7.0

Choose a tag to compare

@shawnshekari shawnshekari released this 05 Nov 18:32
· 3 commits to main since this release

strixite v0.7.0

Clients that rewrite what they sent, and control over which tokens the model may pick.

Editing old messages costs only what comes after the edit

  • Some clients rewrite messages they sent long ago: an agent that trims old tool output to keep its context small, a
    client that drops old screenshots to stay under an image limit, a chat that lets you edit an earlier message. The
    model's linear-attention layers keep a running state instead of a history, so until now everything after the
    system prompt had to be read again.
  • The server now keeps small resume points at the start of every reply and every user message (about 0.11 GB
    each, however deep the conversation), and resumes from the last one before the edit, over the K/V the previous
    version left in its slot or in the prompt cache. With several slots, the edited request goes to the slot that holds
    the previous version. prompt-cache-points (16) sets how many are kept, in RAM only.
    How it works
  • An edit inside the system prompt or the tool list still means reading everything again: keep those stable within a
    conversation.

logit_bias

  • On /v1/chat/completions and /v1/completions, in OpenAI's shape: {"<token id>": bias}, -100 to 100. The bias
    is added on every token the model samples, thinking and answer alike - -100 all but bans a token, +100 all but
    forces it. A response_format schema still wins over a bias.
  • It biases tokens, not words: ban the "Paris" tokens and the model answers 巴黎. To keep a script out of the replies
    (stray ideographs in non-English sessions, the case in #26), bias every token that holds one of its characters -
    about 66,000 for the CJK scripts, under 1% slower decode.
    Details

Also

  • The developer role (OpenAI's newer name for system) is accepted as a leading system message (#28).
  • A checkpoint with a different number of experts loads: the server takes the count from the weights instead of
    assuming 512, and a model without n-gram rows no longer trips the stats (#35).

Speed (this build, end to end through the server, one request at a time, two runs per depth; v0.6.0 numbers as
published for it)

context prefill t/s decode t/s
32k 1,845 (v0.6: 1,843) 59.5 (63.8)
120k 1,783 (1,808) 59.9 (58.3)
250k 1,692 (1,731) 57.7 (58.8)
350k 1,672 (1,665) 58.6 (56.5)
481k 1,618 (1,632) 55.9 (56.1)

The same speed as v0.6 - prefill within 2.3%, decode within run-to-run spread (v0.6's 32k decode was a high run:
v0.5 measured 60.3 there).

Container

podman pull ghcr.io/shawnshekari/strixite:v0.7.0

Upgrading from v0.6

  • Container: pull :v0.7.0 (or :latest) and restart.
  • From source: git pull, then build as before. The weights are unchanged.
  • Your own copy of strix-server.conf: one new setting, prompt-cache-points (16), which works at its default
    without a line in the file. git diff v0.6.0 v0.7.0 -- deploy/strix-server.conf shows the change.
  • Prompt-cache entries from v0.6 stay valid - the saved-state format is unchanged.

Contributors: @shawnshekari, @Cheblan (#28), @ajaxdude (#35), @mikenavi (suggested in #32 and #33), @maxm86545
(suggested in #26)