QEC decoding server: cut HOST_CALL latency ~10x (direct dispatch + bounded spin-then-block) - #750
Merged
Conversation
…obes A served RPC crosses three thread boundaries (dispatcher->recv inbox, recv->session worker queue, worker->dispatcher response promise), each a condvar/promise futex wakeup costing ~3-4 us on Grace-class hosts plus a ~90-115 us p99 tail from deep idle-state exits -- the dominant share of the HOST_CALL handler latency observed on the opnic rig. Fixes, measured per-hop over the udp two-process rig (d3r12 surface code, multi_error_lut, 2000 shots) and confirmed over the real HSB FPGA wire: 1. Direct dispatch: CqrTransceiver::inject() hands each translated frame straight to RpcDispatcher::dispatch on its calling CUDAQ dispatcher thread (ITransceiver::install_dispatch_sink, installed by DecodingServer at construction) instead of queueing it for a recv() thread -- one full cross-thread handoff removed from every RPC. The inbox/recv path remains for transports without a sink; RpcDispatcher::dispatch gains a catch-all so no handler exception can unwind into the transport; the registry/dispatch table are read-only post-construction, so concurrent per-ring dispatcher threads dispatch safely. 2. Bounded spin-then-block waits (SpinPolicy.h): the session worker spins on an atomic queue sequence and the blocking-RPC waiter on a stack-owned completion flag (stored by every completer immediately before promise::set_value) before falling back to their untouched blocking primitives. Default budget 200 us; QEC_DECODING_SERVER_SPIN_US overrides (0 = always block, -1 = never block). Idle threads park after one budget window. 3. queue_cv.notify_one() hoisted out of queue_mutex in try_enqueue, and an env-gated /dev/cpu_dma_latency hold in the decoding_server tool (QEC_DECODING_SERVER_CPU_DMA_LATENCY_US) that removes the deep-idle wakeup tail. HopStats.h adds per-request timestamp probes across the hops (plus stage durations and notify cost), correlated through a request_id-keyed slot array and reported at shutdown (QEC_DECODING_SERVER_HOP_STATS[_CSV]). Off by default: one predicted branch per probe; handler-total distributions with probes on vs entry/exit-only stamps match within 0.1 us at p50. QEC_PIN_DISPATCHER/RECV/WORKER pin the server threads for controlled measurements, and the threads are named (cqr-dispatch, qec-recv, qec-worker) so scheduler traces read cleanly. Stock configuration, no env vars -- HOST_CALL round trip, median (p99): enqueue_syndromes 11.0 -> 0.8 us (108 -> 5.3) get_corrections 18.1 -> 1.6 us (124 -> 7.2) reset_decoder 18.0 -> 4.8 us (120 -> 8.5) Per-hop p50s 4.5/3.0/4.8 -> 0.03/0.14/0.51 us; FPGA-wire get_corrections median 1.70 us. New unit tests cover inline dispatch (thread identity, ACK-before-handler ordering, the no-mutex-across-dispatch invariant), shutdown release of a blocked waiter, and spin-window + post-budget arrivals. Full realtime suite, surface_code-1/4/5 cqr ctests, and the realtime_decoding_demo matrix (udp + cpu_roce loopback + FPGA, all available decoders) pass unchanged. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
cketcham2333
marked this pull request as ready for review
July 27, 2026 19:28
…ency Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
bmhowe23
approved these changes
Jul 28, 2026
melody-ren
added a commit
to melody-ren/cudaqx
that referenced
this pull request
Aug 5, 2026
Conflict resolutions, all in favor of upstream's post-NVIDIA#682/NVIDIA#750 realtime structure with the inproc_rpc path kept out: - qec_realtime_session.cpp: deleted (upstream's touch was to removed code). - GpuRoceTransceiver.cpp: deleted; NVIDIA#754 renamed it to DeviceGraphTransceiver.cpp. The stale qec_realtime_session comment reference the branch scrubbed moved to DeviceGraphRingConsumer.cpp and is scrubbed there instead. - realtime_decoding.cpp: reset_decoder takes NVIDIA#698's pin_decode_device(), dropping the rpc_producer dispatch branch. - decoding_server.cpp: upstream's per-ring rewrite; its comments no longer reference qec_realtime_session, so the branch's scrub is subsumed. - app_examples/CMakeLists.txt: upstream's add_surface_code_4_yaml_test signature (onnx_path folded into ARGN) with an empty test environment. - surface_code-4-yaml-test.sh: keep the num_logical echo, drop the CUDAQ_QEC_REALTIME_MODE echo. - realtime_relay_bp.rst: single Building subsection renamed Hololink -> HSB, target list on the NVIDIA#754 names, CI-unit-test section dropped (the test is removed) and the Surface Code Test section kept. Upstream added no new inproc_rpc surface: DeviceGraphTransceiver, DeviceGraphRingConsumer, surface_code-5-per-decoder-rings and realtime_decoding_demo are all inproc-free. Signed-off-by: Melody Ren <melodyr@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A HOST_CALL RPC served by the decoding server crossed three thread
boundaries — CUDAQ dispatcher → receiver-thread inbox, receiver → session
worker queue, worker → dispatcher response promise. Each handoff is a
condvar/promise futex sleep/wake costing ~3–4 µs on Grace-class hosts, plus
a ~90–115 µs p99 tail when the sleeping core sits in a deep idle state.
Those handoffs, not the decoders, dominated the handler latency observed during testing.
This PR removes or short-circuits all three, on top of the #682
architecture:
CqrTransceiver::inject()hands each translatedframe straight to
RpcDispatcher::dispatch()on the calling CUDAQdispatcher thread via a sink installed by
DecodingServeratconstruction (
ITransceiver::install_dispatch_sink). No receiver threadis started for CQR transports; the inbox/recv path remains for
transports that decline the sink (and stays unit-tested).
RpcDispatcher::dispatchgains a catch-all so no handler exception canunwind into the transport.
SpinPolicy.h) — the per-decoder sessionworker spins on an atomic queue sequence, and the blocking-RPC waiter on
a stack-owned completion flag, before falling back to their untouched
condvar/promise waits. Default budget 200 µs;
QEC_DECODING_SERVER_SPIN_USoverrides (0= always block,-1=never block). Idle threads park after one budget window.
notify_one()hoisted out ofqueue_mutexintry_enqueue, and an opt-in/dev/cpu_dma_latencyhold in thestandalone tool (
QEC_DECODING_SERVER_CPU_DMA_LATENCY_US) that removesthe deep-idle p99 tail.
HopStats.hadds env-gated per-request latency probes across the hops(
QEC_DECODING_SERVER_HOP_STATS[=1|total|_CSV]), off by default (onepredicted branch per probe; p50 unchanged within 0.1 µs), plus
QEC_PIN_DISPATCHER/WORKERthread pinning for controlled measurements.Runtime / performance impact
Stock configuration (no env vars), HOST_CALL round trip, median (p99), udp
two-process rig, d3/r12 surface code, 2000 shots: