Skip to content

b11730

  • b11730
  • 4c1004c

  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode

  • Choose a tag to compare

    View all tags
b11730: cuda: cap MMQ Stream-K blocks at available work (#30553)
  • b11730
  • 4c1004c

  • Choose a tag to compare

    View all tags

  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
@joelagnel joelagnel tagged this 19 Oct 07:41 
* cuda: cap Stream-K blocks at the number of k-blocks

Stream-K normally launches one CUDA block per SM. For small matrices,
this can exceed the total number of output-tile/k-block pairs, leaving
blocks with no work and adding launch and fixup overhead. Cap the grid
at that count to remove these excess blocks without changing the useful
rounded K ranges or numerical results.

Assisted-by: Codex

* cuda: cap Stream-K blocks at the number of k iterations

Cap the grid at the number of K_vram iterations across all output tiles, rounding up partial iterations. This removes extra CUDA blocks left by the k-block cap.

Partial iterations can change FP32 summation grouping. MUL_MAT and MUL_MAT_ID backend tests pass on RTX 5090.

* Update ggml/src/ggml-cuda/mmq.cuh

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Assets 2
Loading