Repository navigation
strixite v0.7.0
strixite v0.7.0
Clients that rewrite what they sent, and control over which tokens the model may pick.
Editing old messages costs only what comes after the edit
- Some clients rewrite messages they sent long ago: an agent that trims old tool output to keep its context small, a
client that drops old screenshots to stay under an image limit, a chat that lets you edit an earlier message. The
model's linear-attention layers keep a running state instead of a history, so until now everything after the
system prompt had to be read again. - The server now keeps small resume points at the start of every reply and every user message (about 0.11 GB
each, however deep the conversation), and resumes from the last one before the edit, over the K/V the previous
version left in its slot or in the prompt cache. With several slots, the edited request goes to the slot that holds
the previous version.prompt-cache-points(16) sets how many are kept, in RAM only.
How it works - An edit inside the system prompt or the tool list still means reading everything again: keep those stable within a
conversation.
logit_bias
- On
/v1/chat/completionsand/v1/completions, in OpenAI's shape:{"<token id>": bias}, -100 to 100. The bias
is added on every token the model samples, thinking and answer alike - -100 all but bans a token, +100 all but
forces it. Aresponse_formatschema still wins over a bias. - It biases tokens, not words: ban the "Paris" tokens and the model answers 巴黎. To keep a script out of the replies
(stray ideographs in non-English sessions, the case in #26), bias every token that holds one of its characters -
about 66,000 for the CJK scripts, under 1% slower decode.
Details
Also
- The
developerrole (OpenAI's newer name forsystem) is accepted as a leading system message (#28). - A checkpoint with a different number of experts loads: the server takes the count from the weights instead of
assuming 512, and a model without n-gram rows no longer trips the stats (#35).
Speed (this build, end to end through the server, one request at a time, two runs per depth; v0.6.0 numbers as
published for it)
| context | prefill t/s | decode t/s |
|---|---|---|
| 32k | 1,845 (v0.6: 1,843) | 59.5 (63.8) |
| 120k | 1,783 (1,808) | 59.9 (58.3) |
| 250k | 1,692 (1,731) | 57.7 (58.8) |
| 350k | 1,672 (1,665) | 58.6 (56.5) |
| 481k | 1,618 (1,632) | 55.9 (56.1) |
The same speed as v0.6 - prefill within 2.3%, decode within run-to-run spread (v0.6's 32k decode was a high run:
v0.5 measured 60.3 there).
Container
podman pull ghcr.io/shawnshekari/strixite:v0.7.0
Upgrading from v0.6
- Container: pull
:v0.7.0(or:latest) and restart. - From source:
git pull, then build as before. The weights are unchanged. - Your own copy of
strix-server.conf: one new setting,prompt-cache-points(16), which works at its default
without a line in the file.git diff v0.6.0 v0.7.0 -- deploy/strix-server.confshows the change. - Prompt-cache entries from v0.6 stay valid - the saved-state format is unchanged.
Contributors: @shawnshekari, @Cheblan (#28), @ajaxdude (#35), @mikenavi (suggested in #32 and #33), @maxm86545
(suggested in #26)