Repository navigation
Patch release. ONNX Runtime keeps its worker threads spinning on the CPU after each run, waiting for the next one. On a 4-core Intel host one short query left three cores busy for about 1.4 s after the response, which is when a caller on the same host does its own work. 0.9.1 turns that off by default. Vectors are unchanged.
What changed
- Every ONNX provider's session is built with
session.intra_op.allow_spinning = 0. ONNX_ALLOW_SPINNING=1restores the runtime's default. Only0and1are accepted; anything else stops the server at start.
Measured
On a 4-core Intel N150 (Debian 12, onnxruntime 1.30.0, jina-v5-nano fp32, CPU):
| spinning (before) | not spinning (0.9.1) | |
|---|---|---|
CPU used by the server in the 1.0 s after one /embed |
3.00 / 3.01 / 3.01 CPU-s | 0.00 / 0.02 / 0.00 |
| CPersona recall over 100,000 memories, query embedded through this server, median (registered run) | 1.94 s | 1.58 s |
| the same, CPersona with its recall arms run concurrently | 1.64 s | 1.25 s |
| 25 embed calls back to back through the server, median | 80.8 / 81.3 ms | 79.4 / 78.6 ms |
A bare session took 53-55 ms per short run and 12.3 s per 8 x 512-token batch with or without spinning. On an Apple laptop (onnxruntime 1.27.0) the spinning stopped within 20 ms of a run, so the cost shows mainly on Intel hosts.
Upgrading
Nothing to do. To keep the previous behaviour, set ONNX_ALLOW_SPINNING=1.