Concurrent MLX GPU + Apple Neural Engine serving for Laya — looking for cross-SoC benchmarks

[tc3oliver]

laya-apple is a correctness-validated heterogeneous Laya runtime for Apple silicon. It serves requests concurrently across the MLX GPU and the Apple Neural Engine.

Same 102 requests, same arrival times — GPU-only vs concurrent MLX GPU + Apple Neural Engine serving.

1080p MP4 · where every number on screen comes from. The workload is laya-typed-decisions with open-loop bursty arrivals, and playback is 3.4× slower than real time.

What the video shows:

• GPU-only: short requests wait in the queue behind long and medium requests on the one GPU.
• GPU + ANE: validated short requests run on the ANE, while long and multi-question work stays on the MLX GPU.
• One device per request: requests are never split across the two engines.
• Where the improvement comes from: concurrency and queue isolation, not a faster engine. A single short request takes about the same time on either device: 9.9 ms on the ANE against 12.2 ms on MLX.
• Result for this workload: short-request P99, measured from arrival, drops from 1592.9 ms to 79.5 ms, with 0 answer mismatches.

Key benchmark

Mixed-workload throughput, GPU-only against concurrent GPU + ANE serving:

Model	GPU-only	GPU + ANE	Gain
laya	41.9 req/s	122.5 req/s	2.92×
laya-multilingual	55.7 req/s	241.8 req/s	4.34×
laya-typed-decisions	24.0 req/s	109.6 req/s	4.57×

• Setup: Apple M4 Max, macOS 26.6.2.
• Workload: a mixed short + long workload, the same one defined in the repository benchmark. One short and one long closed-loop request stream run through one Laya(execution="workers") instance.
• Data: the raw data and reproduction steps are public in benchmarks/v1.0.md.

The gain comes from using both engines concurrently, not from the ANE being 4.57× faster per request. These are results from one machine.

Correctness finding

In the tested configuration, the ordinary Core ML export ran on the ANE without errors. On some workloads, though, it changed decisions against upstream PyTorch: up to 85 hard mismatches on the validation rows. The same export on CPU_AND_GPU matched. This finding covers these models on this setup, not Core ML or the ANE in general.

laya-apple therefore uses only ANE artifacts that pass two checks: a parity gate and a placement validation. On the shipped validation rows, the laya-apple ANE path has 0 hard decision mismatches (docs/correctness.md).

Why heterogeneous routing

Request	Goes to
Short, validated, single-question	ANE
Long or multi-question	MLX GPU
Independent requests under load	Both engines serve concurrently, and the router also compares queue backlogs
Any request on unvalidated hardware	MLX, until local calibration and validation pass

Help test other Apple silicon

I only have an M4 Max, so I don't want to assume these routing thresholds or gains generalize across Apple silicon. If you have another Mac, a result from it is the most useful contribution right now:

• M1 / M2 → Add an M1/M2 MLX-only benchmark result tc3oliver/laya-apple#1
• M3 Max → Add an M3 Max benchmark result tc3oliver/laya-apple#2
• M4 Pro → Add an M4 Pro benchmark result tc3oliver/laya-apple#3
• Other Apple silicon → https://github.com/tc3oliver/laya-apple/blob/main/docs/community-benchmarks.md

No code changes required. After setup, run:

uv run python scripts/hardware_report.py --quick

The script writes a hardware-results/ bundle that you can submit as-is in a pull request. Each issue has the full steps. Results that differ from mine, including failures, are just as useful.

---

Repository: https://github.com/tc3oliver/laya-apple
Community benchmark matrix: https://github.com/tc3oliver/laya-apple/blob/main/docs/community-benchmarks.md
Reproducible benchmark: https://github.com/tc3oliver/laya-apple/blob/main/benchmarks/v1.0.md
Listed in Laya's upstream Community Tools: https://github.com/NandhaKishorM/laya#community-tools

[tc3oliver]

Update: the first external cross-SoC result has landed.

An M4 Pro (48 GB, macOS 27.0) passed both MLX FP16 and ANE parity with 0 hard mismatches. After local calibration, short single-question requests routed to the ANE, and the heterogeneous GPU+ANE check also passed.

This is one community --quick run on laya-typed-decisions, so I'm keeping it separate from the M4 Max release benchmark and not generalizing it to all M4 Pro systems.

Result:
tc3oliver/laya-apple#32

Still looking for M1/M2/M3/M5 and other Apple Silicon results:
https://github.com/tc3oliver/laya-apple/blob/main/docs/community-benchmarks.md