RFC: Metal cooperative cancellation for abort callbacks

[QuintinShaw]

I've been looking at what it would take to support abort_callback on Metal.

This is a design discussion, not a patch. I have a local prototype, but I don't want to turn it into a PR before the expected semantics are clear.

Related: #10509 discussed cancellation during llama_decode, and #12525 discussed exposing abort status. I couldn't find an existing discussion specifically covering cooperative cancellation in Metal.

This RFC focuses on Metal first. If the approach makes sense, I can investigate equivalent support for HIP/Vulkan/CUDA separately.

All measurements below are from an Apple M1 on llama.cpp cc231cb0d, with Metal n_cb=1 (the default).

Current behavior

llama.h currently documents abort_callback as working only with CPU execution.

I found two separate gaps on Metal.

First, llama_set_abort_callback() does not reach the Metal backend. It looks up:

ggml_backend_reg_get_proc_address(reg, "ggml_backend_set_abort_callback")

but Metal does not register that proc, so the lookup returns NULL.

Second, even when I bypass that and call ggml_backend_metal_set_abort_callback() directly, the callback is not consulted on the normal non-capture path.

With a ~105 ms Metal matmul graph:

• llama_set_abort_callback(): callback calls = 0
• direct Metal setter: callback calls = 0
• abort requested at 5 ms: graph still runs to completion
• returned status: GGML_STATUS_SUCCESS

ggml_backend_graph_compute_async() returns SUCCESS shortly after submission, well before GPU completion. The GPU work continues after submission, and the caller later waits for it in ggml_backend_synchronize(), which currently returns void.

As a control, CPU calls the callback from the compute thread and can return GGML_STATUS_ABORTED.

The only existing Metal abort check I found is on the capture wait path, between command buffers.

Local prototype

I made a small local prototype to see whether cooperative cancellation inside an already committed command buffer is practical.

It uses:

• a host-visible abort flag
• a 1-thread Metal gate before each dispatch
• indirect dispatch, with the gate writing a zero grid after abort
• no gate when no callback is installed
• concurrent encoder disabled while abort is enabled

I did not change the signature of ggml_backend_synchronize() or add a new public ABI.

To make GGML_STATUS_ABORTED observable by the caller, the current prototype waits for GPU completion inside Metal graph compute when an abort callback is installed. So the abort-enabled path becomes synchronous.

The synthetic graphs below repeat the same 2048x2048 mul_mm dispatch. The longer case is many normal dispatches, not one giant kernel.

graph	abort requested	completion	status
~105 ms (8 dispatches)	5 ms	median ~15 ms	ABORTED
several seconds (80 dispatches)	5 ms	median ~17 ms (13–23 ms)	ABORTED

The longer graph had a median baseline around 7.7 s in these runs; its exact throughput is not the point of this test.

The 5 ms mark is when the callback requests abort, not when the GPU has stopped. In the long graph, completion was around 17 ms from graph start, or about 12 ms after the request.

The in-flight kernel is not interrupted. The important observation is that cancellation time stayed roughly the same when the amount of remaining graph work increased substantially. It tracks kernel/dispatch granularity rather than the total work left in the graph.

A single multi-second kernel would not behave this way.

I also tried two polling strategies:

• a 1 ms background timer: works, but the callback runs on a helper thread
• polling from the graph-compute caller after submission: also works, while keeping the callback on the caller thread

So a helper thread is not required if abort-enabled Metal compute is allowed to block.

Abort-enabled path overhead

I also measured a real workload.

This is not just gate overhead. The current abort-enabled path includes per-dispatch gating, concurrent encoder disabled, and callback polling.

llama-bench, Qwen3-4B Q4_K_M, -ngl 99, warmup, 10 runs:

test	baseline median (min–max)	abort enabled, callback always false	delta
pp512	212.29 (210.99–213.84) t/s	202.32 (199.91–204.40) t/s	-4.7%
tg128	18.77 (18.43–20.01) t/s	18.48 (17.96–18.76) t/s	-1.5%

The pp512 regression was consistent across these runs; the observed ranges did not overlap. The tg128 difference is smaller and closer to run-to-run variation.

With no callback installed, the existing Metal path is unchanged and no gate is used.

Questions

Before taking the prototype any further, there are three things I'd like to get direction on.

1. Should Metal participate in the generic abort mechanism?

That would mean registering ggml_backend_set_abort_callback on Metal so llama_set_abort_callback() actually reaches the backend.

If Metal abort is not something we want to support, there is no reason to take the rest further.

2. If Metal supports abort, what execution semantics should it have?

I see three consistent options:

A. Keep Metal fully async.

The callback has to be polled from another execution context while the GPU is running. This preserves async submission, but changes where the callback may run.

B. Keep the callback on the graph-compute caller and return GGML_STATUS_ABORTED without changing the generic backend API.

This is what the same-thread prototype does. The trade-off is that Metal graph compute becomes synchronous while an abort callback is installed.

C. Preserve both fully async submission and caller-thread callback semantics.

That seems to require a generic way for an async backend to surface execution-time completion status. Today graph_compute_async() returns submit-time status and synchronize() returns void, so an abort that happens after submission otherwise has nowhere to propagate back to the caller.

3. If cooperative gating is the right mechanism, what granularity/cost is acceptable?

Gating every dispatch gives good cancellation latency, but the current abort-enabled path costs about 4.7% on pp512 and 1.5% on tg128 in this M1 setup.

It may make more sense to gate less frequently or only around sufficiently expensive work. There is also the question of how abort should interact with concurrent encoders and the Metal capture path.

If this is something we want to support upstream, I'm happy to turn the preferred semantics into a focused PR.

@ggerganov would be interested in your take on which direction fits the Metal backend best.