Repository navigation
v0.3.251-alpha
A computer that already guesses ahead with a small model now fetches a better one when the
swarm has it. When a model is split across computers, a small model of the same family guesses
the next words, so each trip between the computers keeps several of them instead of one. Since
0.3.249 a computer with a graphics card may guess with a model up to half the size of the one it
splits — the 3B for a 7B, which ran 18.1 tokens a second against the 1.5B's 16.0 on a real
long-distance link. But a computer that already held a smaller guessing model, fetched under the
older rule or kept for any other reason, went on using it. It now fetches the larger one from the
swarm (at most 2 GB, as before) beside the one it has, keeps guessing with the old one while the
new one arrives, and switches once the new one is complete.
Another computer can no longer make this one use unbounded memory, or crash part of it, by
sending more than it was asked for. Four places took a size from the computer on the other end
on trust. The reply another computer streams back for a split request was buffered with no limit.
The compressed results computers exchange when they share one layer's work were unpacked with no
cap, so a few kilobytes could expand to gigabytes. A cached-prompt snapshot from another computer
was cut up by lengths it chose, and a length that overflowed crashed the check. And the first
computer's share of a shared layer was not size-checked like the others'. Each is now held to the
size this computer asked for or expects. A computer that sends more is set aside for that request,
and the error names what it did. A HuggingFace repository name another computer passes on is now
checked before this computer asks HuggingFace about it, as every other HuggingFace request already
was.
A reply at temperature 0 picks the same word with and without guessing ahead. When two words
scored exactly the same, the normal path took the last of them, the paths that check guesses took
the first, and a graphics card checking guesses took whichever its own work order reached first, so
in that rare case one prompt could get different replies. All of them now take the first, as
llama.cpp does.
The Performance page says it could not load instead of showing a computer that served
nothing. When the node refused the request (while starting up, or with an expired key), the
page drew the refusal as a row of zeros.
Full commit list: v0.3.250-alpha...v0.3.251-alpha