Skip to content

v0.4.17

  • v0.4.17
  • c7d541f
  • Verified

    This tag was signed with the committer’s verified signature.
    tamnd Tam Nguyen Duc
  • Choose a tag to compare

  • v0.4.17
  • c7d541f
  • Choose a tag to compare

  • Verified

    This tag was signed with the committer’s verified signature.
    tamnd Tam Nguyen Duc
@tamnd tamnd tagged this 05 Sep 22:41
The repack cache is 1.9 per cent larger than the model file where it was ten
per cent, so an 8B holds 4781 MiB of weights against 5151 and a 4090 holds
5540 MiB of them at a context of 2048 against 5910. That is 1.066 times what
llama.cpp holds on the card, and it costs 2.5 per cent of a decode.

There is no longer a lossy step anywhere in the repack. All eight quantized
types round trip bit for bit and the logit corpus agrees with llama.cpp on
all thirteen device cases on both backends.
Assets 2
Loading