Skip to content

b10715

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 31 Aug 10:11
662a0b0

spec : fuse the DFlash encoder into the KV cache injection (#27310)

  • dflash : fuse the encoder into the KV injection decode

The encoder is a single fc + norm, but running it as a separate
llama_encode forced a device-to-host round trip of its output before the
injection decode could re-upload it, plus a second graph build per
round. Fold the encoder into the decoder's embd branch and feed the
target features directly to one llama_decode.

Assisted-by: Claude Fable

  • nit

  • Apply batched suggestions from code review

Co-authored-by: Ruixiang Wang wangruixiang07@outlook.com

  • Fix missing references from renaming

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co
Co-authored-by: Ruixiang Wang wangruixiang07@outlook.com

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI: