Repository navigation
v1.10.6
Fixed
- Auto-tune no longer picks a configuration that secretly runs from system RAM. When a model
doesn't fit on your GPU, the driver doesn't fail — it quietly keeps the overflow in system memory,
and the reported VRAM usage stops changing once the card is full. Auto-tune was reading only that
number, so those configurations looked like a clean fit, and the search kept pushing more onto the
GPU until it landed on one that crawled, timed out, or crashed. It now measures how much memory a
configuration actually pushes into system RAM and rejects the ones that spill. On a 40 GB model
and a 16 GB card, the offload settings that looked identical by VRAM turned out to differ by
8.8 GB of hidden spill. - Auto-tune no longer gives up when its chosen configuration fails. If the benchmark run
crashed or timed out, the whole sweep ended with no result at all. It now steps back to the next
safer setting and keeps going, so a run returns a working configuration instead of nothing.
Changed
- Auto-tune now picks the winner by how quickly you get a complete answer, combining
prompt-processing and generation speed, rather than by generation speed alone. Generation speed
keeps improving as more of a model is forced onto the GPU — including past the point where it
starts spilling — so optimising for it alone favoured settings that were slower in real use. On a
32 GB model the newly-chosen setting finishes a turn 24% sooner than the old one. - Auto-tune runs are also shorter, since detecting a bad configuration no longer requires waiting
out a full benchmark.
Verified live across three models and both search paths (dense and MoE), spanning 9.4 GB to
39.7 GB on a 16 GB card, with each chosen configuration's spill independently re-measured
afterward. Full details in PR #158.