Skip to content

b11902

  • b11902
  • 02269b4
  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
  • Choose a tag to compare

  • b11902
  • 02269b4
  • Choose a tag to compare

  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
@Captain-Tripps Captain-Tripps tagged this 26 Oct 14:12
* sycl: add set_tensor_2d for faster tensor-split loading

Pack host rows into chunks of up to 64 MiB when destination rows are
contiguous, avoiding a separate upload call for each row. Keep the
existing per-row copy path for non-contiguous destinations.

On two Arc Pro B70s with gpt-oss-120b UD-Q4_K_XL, total llama-perplexity
runtime drops from 263-268 s to 114-115 s. A full-load readback check
verified 26,784,256 rows with no mismatches.

* sycl: batch 2D tensor uploads through shared staging slots

Implement set_tensor_2d to speed up tensor-split model loading by
packing rows into the existing pinned staging slots and uploading
each batch with a single device copy.

Share staging initialization with set_tensor and wait only when
reusing a slot, allowing CPU packing and device copies to overlap.

Fall back to per-row uploads for noncontiguous device rows,
oversized rows, or unavailable staging.

---------

Co-authored-by: Jon Staples <250234834+Captain-Tripps@users.noreply.github.com>
Assets 2
Loading