Skip to content

Releases: KoopahTManiac/speed.llama.cpp

Release list

b10373

b10373 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Aug 14:19
b10373
53fbd13

speculative: never-worse scheduler - verify-all cold start, robust fits

Field regression on a 5090 (coding 600 -> 350-400 t/s) exposed three
defects in the self-calibrating scheduler:

  • cold sessions ran admission on the head's raw pessimistic
    confidences; worse, the resulting truncation censored the very
    full-block rounds calibration needs, so the first fit was ~6000
    rounds away. Uncalibrated mode now admits everything (same
    convention as sglang's flat-SPS verify-all default): full fixed-depth
    speed from round one, and every round feeds calibration, putting the
    first fit ~1 long prompt away.
  • early small-window fits produced extreme temperatures (T=0.087 seen,
    crushing deep-position survival). Fits now require >=32 samples of
    each outcome class per position (identity otherwise) and shrink
    geometrically toward identity by window size.
  • the admission scan now demands a meaningful predicted gain before
    truncating (4% margin on the causal early stop, and full admission
    is preferred whenever it scores within the margin of an interior
    argmax): near-flat theta curves - easy content on fast GPUs - admit
    everything, encoding the measured lesson that trading accepted
    tokens for noise-level predicted savings always loses.

Also: profiled SPS curve forced monotone (Windows timing outliers put
size 1 above size 2), and the calibration cache version bumped to v2
so existing caches from the noisy-fit era are discarded on load.

Isolation on RTX PRO 6000 (GSM k8, fresh caches, 4 rounds): OFF 588
t/s mean vs sched+adapt 587 (parity, accept len 7.18 vs 7.08); cold
first round 600.6 (was 350-400 on the affected 5090 setup).

Website:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Assets 26

b10372

b10372

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Aug 12:06
b10372
e8c50dc

speculative: self-calibrating scheduler with persistent cache

--spec-sts required a manual offline fitting pipeline; calibration is
now automatic. The server collects (confidence vector, realized
accepted length) pairs from full-block rounds into a ring buffer -
with an exploration round every 16th drafted round verifying the full
block, so scheduled truncation never censors deep positions out of the
data - and refits the paper-3.2.1 sequential cumprod-ECE grid search
in-process every 256 rounds (~3ms on the most recent 1024 rounds).

STS temperatures and the online cost model (t_fix, c_tok) persist to a
per-draft-model cache file (--spec-sched-cache, default 'auto' in the
llama cache dir), so later boots start warm. --spec-sts still accepts
fixed temperatures and disables auto-refit.

Smoke-tested: cold boot refits at 384/640 rounds and writes the cache;
warm boot loads it (sts=8, t_fix=6.2ms).

Website:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Loading

b10365

b10365

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Aug 01:31
b10365
13980d9

ci: point release download links at the publishing repository

The release-notes template hardcoded the upstream repo, so every link
in a fork's release body pointed at ggml-org instead of the release's
own assets.

Website:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Loading

b10364

b10364

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Aug 00:39
b10364
34343a9

ci: drop the s390x release build

The s390x partner runner routinely queues for hours and the release job
needs the whole ubuntu-cpu matrix, so it stalled every release.

Website:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Loading