Skip to content

v0.9.1 — no spinning after each run

Latest

Choose a tag to compare

@Cloto-dev Cloto-dev released this 06 Oct 11:50

Patch release. ONNX Runtime keeps its worker threads spinning on the CPU after each run, waiting for the next one. On a 4-core Intel host one short query left three cores busy for about 1.4 s after the response, which is when a caller on the same host does its own work. 0.9.1 turns that off by default. Vectors are unchanged.

What changed

  • Every ONNX provider's session is built with session.intra_op.allow_spinning = 0.
  • ONNX_ALLOW_SPINNING=1 restores the runtime's default. Only 0 and 1 are accepted; anything else stops the server at start.

Measured

On a 4-core Intel N150 (Debian 12, onnxruntime 1.30.0, jina-v5-nano fp32, CPU):

spinning (before) not spinning (0.9.1)
CPU used by the server in the 1.0 s after one /embed 3.00 / 3.01 / 3.01 CPU-s 0.00 / 0.02 / 0.00
CPersona recall over 100,000 memories, query embedded through this server, median (registered run) 1.94 s 1.58 s
the same, CPersona with its recall arms run concurrently 1.64 s 1.25 s
25 embed calls back to back through the server, median 80.8 / 81.3 ms 79.4 / 78.6 ms

A bare session took 53-55 ms per short run and 12.3 s per 8 x 512-token batch with or without spinning. On an Apple laptop (onnxruntime 1.27.0) the spinning stopped within 20 ms of a run, so the cost shows mainly on Intel hosts.

Upgrading

Nothing to do. To keep the previous behaviour, set ONNX_ALLOW_SPINNING=1.