Skip to content

b12293

  • b12293
  • 7f24aac
  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
  • Choose a tag to compare

  • b12293
  • 7f24aac
  • Choose a tag to compare

  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
@ServeurpersoCom ServeurpersoCom tagged this 11 Nov 10:55
* cpu: accumulate flash attention V in F32 in the one-chunk kernel

The one-chunk kernel accumulated an F16 V cache in an F16 running sum
that is only normalized at the end, which overflows past 65504 on long
contexts: inf, then NaN on every following token.

Every V type now accumulates in F32, like the tiled kernel. F16 rows go
through ggml_vec_mad_f16_f32, which converts and multiplies in the same
register (F16C/AVX512, NEON, RVV) with no round trip through a temporary
F32 row, so decode keeps master speed on x86 and loses about 4% at 4k
context on ARM, where master used native F16 FMA.

* cpu: widen the F16 V rows like ggml_vec_dot_f16_f32

ggml_vec_mad_f16_f32 required F16C and FMA, so AVX builds without FMA
fell back to a scalar loop and ran 3x slower than master. It now
follows the guards of ggml_vec_dot_f16_f32: AVX512, then AVX with F16C
through GGML_F32x8_FMA, then NEON and RVV, then the scalar loop.
Assets 2
Loading