Skip to content

v1.10.6

Choose a tag to compare

@mohitsoni48 mohitsoni48 released this 07 Aug 17:18
· 1300 commits to main since this release
bf7a4de

Fixed

  • Auto-tune no longer picks a configuration that secretly runs from system RAM. When a model
    doesn't fit on your GPU, the driver doesn't fail — it quietly keeps the overflow in system memory,
    and the reported VRAM usage stops changing once the card is full. Auto-tune was reading only that
    number, so those configurations looked like a clean fit, and the search kept pushing more onto the
    GPU until it landed on one that crawled, timed out, or crashed. It now measures how much memory a
    configuration actually pushes into system RAM and rejects the ones that spill. On a 40 GB model
    and a 16 GB card, the offload settings that looked identical by VRAM turned out to differ by
    8.8 GB of hidden spill.
  • Auto-tune no longer gives up when its chosen configuration fails. If the benchmark run
    crashed or timed out, the whole sweep ended with no result at all. It now steps back to the next
    safer setting and keeps going, so a run returns a working configuration instead of nothing.

Changed

  • Auto-tune now picks the winner by how quickly you get a complete answer, combining
    prompt-processing and generation speed, rather than by generation speed alone. Generation speed
    keeps improving as more of a model is forced onto the GPU — including past the point where it
    starts spilling — so optimising for it alone favoured settings that were slower in real use. On a
    32 GB model the newly-chosen setting finishes a turn 24% sooner than the old one.
  • Auto-tune runs are also shorter, since detecting a bad configuration no longer requires waiting
    out a full benchmark.

Verified live across three models and both search paths (dense and MoE), spanning 9.4 GB to
39.7 GB on a 16 GB card, with each chosen configuration's spill independently re-measured
afterward. Full details in PR #158.