Skip to content

Exp 2026 06 13 Task Batching

Jump to bottom
Michael Fischer edited this page Jun 13, 2026 · 1 revision

Experiment — Task-batching per RKNPU_SUBMIT

Date: 2026-06-13 · Status: ❌ RULED OUT · Path: ork-driver

Hypothesis

A transformer fires ~168–197 small matmuls/token, each a separate RKNPU_SUBMIT ioctl paying the ~134 µs round-trip floor. Hypothesis: batching multiple matmuls into one submit (task_number > 1, multiple task_obj_addr entries) would amortize that floor — and this might be how the closed runtime is faster.

Implementation

  • Commit: 95f2481 (oRKLLM/ork-driver) — tools/batch_probe.c + ork_npu_probe_batch.
  • Method: time ntask identical small int8 matmuls as ntask separate ioctls vs one ioctl with task_number=ntask.
  • Build: make batch_probe && sudo ./batch_probe [ntask] [K] [N].

The core of ork_npu_probe_batch — same task[] array, then (a) ntask separate ioctls vs (b) one ioctl with task_number=ntask:

struct rknpu_task *t = c->task.cpu;            /* ntask tasks, same regcmd */
for (int i=0;i<ntask;i++){ memset(&t[i],0,sizeof t[i]); t[i].enable_mask=0xd;
    t[i].int_mask=0x300; t[i].int_clear=0x1ffff; t[i].regcfg_amount=108; t[i].regcmd_addr=c->regcmd.dma; }
// ...
double t0=ork_now_us();                        /* (a) ntask separate ioctls */
for (int i=0;i<ntask;i++){ sub.task_number=1; sub.subcore_task[0]=(struct rknpu_subcore_task){0,1};
    if(ioctl(fd,DRM_IOCTL_RKNPU_SUBMIT,&sub)) return -1; bsync(fd,&O,RKNPU_MEM_SYNC_FROM_DEVICE); }
*us_unbatched = ork_now_us()-t0;

sub.task_number=ntask; sub.subcore_task[0]=(struct rknpu_subcore_task){0,(uint32_t)ntask};
t0=ork_now_us();                               /* (b) one ioctl, ntask tasks */
if(ioctl(fd,DRM_IOCTL_RKNPU_SUBMIT,&sub)){ perror("batched SUBMIT"); return -1; }   /* <-- TIMES OUT */
bsync(fd,&O,RKNPU_MEM_SYNC_FROM_DEVICE);
*us_batched = ork_now_us()-t0;

Results

  • The batched ioctl (task_number>1) TIMES OUT: task counter: 0x0, the NPU dispatches nothing; the kernel soft-resets and recovers. A naive task[] / subcore config does not drive multi-task execution.
  • Premise was also wrong: the closed runtime's captured 12-task submit is one matmul's program (4 sub-tasks × 3 subcores), not multiple matmuls batched. So librkllmrt also does ~1 submit/matmul and pays the same floor (~11 tok/s on 1.7B, which ork-driver matches).

Benchmark (standardized — 2026-06-13)

This is a kernel-level submit probe, not model inference, so the tok/s standard doesn't apply (per the methodology, probe experiments report the probe result). Re-confirmed on RK3588:

$ sudo ./batch_probe 2
batched SUBMIT: Connection timed out
submit-batching probe (single core, int8 896x896):
  ntask=2  rc=-1 (wedge/dims)

ntask=2 batched into one RKNPU_SUBMIT → the ioctl times out (-1), the NPU dispatches nothing. The kernel rejects naive multi-task submits; result is stable across runs.

Verdict

❌ Ruled out twice over: the kernel rejects naive cross-matmul batching, and the reference runtime doesn't use it. The per-matmul submit floor (~134 µs ioctl/IRQ) is inherent hardware, not a lever. Decode multi-core already overlaps the 3 cores (207 µs vs 350 µs single).

Clone this wiki locally