Fix GB200 gpu_roce decoding-server validation issues by vedika-saravanan · Pull Request #683 · NVIDIA/cudaqx

vedika-saravanan · 2026-07-13T03:41:38Z

Description

Follow-up fixes on top of the HSB/FPGA gpu_roce decoding-server path, found during end-to-end GB200 validation of RelayBP / nv-qldpc-decoder with the FPGA emulator.

Validated after rebase with an 85-shot / 425-frame gpu_roce RelayBP run against the FPGA emulator, using 512 playback windows. The emulator processed all windows successfully with 425 responses, 0 send errors, and 0 timeouts.

Changes:

Pin decoding-server graph capture to the decoder cuda_device_id, and reconcile it with the transitional HOLOLINK_GPU_ID setting.
Validate the server-side gpu_roce DOCA ring allocation against host page-size alignment, and make the HSB test harness choose an aligned server ring depth while keeping playback window capacity explicit.
Link optional private decoder_rpc_dispatch.cu support into decoding_server when present; public builds remain unaffected.
Avoid duplicate public/proprietary cudevice whole-archive linkage in decoding_server.
Make the gpu_roce link check verify symbol/link availability without constructing a transceiver or selecting GPU 0.
Fix HSB playback verification to wait for the expected ILA sample count instead of stabilization, and fail clearly if the capture times out before enough samples are collected.
Add separate HSB harness controls for server gpu_roce ring pages and FPGA/emulator playback window pages.

Note: I left this local intentionally because the HOLOLINK_GPU_ID path is transitional and PR692 removes it. Sharing the env parser now would add churn and increase conflicts for code that is about to disappear.

Runtime / performance impact

na

Self-review checklist

Please confirm each item before requesting review. Check [x] or strike
through and explain.

Before requesting review

I reviewed my own full diff in GitHub or my editor.
PR is in Draft if it is not yet ready for review.
Temporary / debugging changes have been removed.
Local test logs reviewed; no unexplained warnings or errors.
CI logs reviewed; no unexplained warnings or errors.
Full CI has been run.

Scope and size

PR is under ~1000 lines, or an exception is justified in the description.
Refactoring-only changes are isolated in their own PR(s).
No existing tests were disabled or modified just to make this PR pass
(if so, an issue has been raised).

Tests

New functionality has new tests.
Tests fail if the new functionality is broken (including crashes), not
just when it is missing.
Negative tests added where exceptions are expected.
Truth data added where simple EXPECT_* / assert checks are
insufficient for algorithmic correctness.
CI runtime impact considered; team notified if significant.

Documentation

Public-facing APIs have Doxygen docs.
User-visible behavior changes have public docs, or a follow-up is
tracked.

Code style

Naming follows the existing convention (snake_case vs camelCase) for
the area being modified.

Dependencies

No new third-party dependencies, or the team has been notified and
OSRB tickets filed.

copy-pr-bot · 2026-07-14T13:54:24Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…ge alignment, private dispatch symbols Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>

Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>

bmhowe23

LGTM once Chuck's threads are marked as resolved.

…ridge branch Main landed the squashed PR 670 plus NVIDIA#656/NVIDIA#673/NVIDIA#674/NVIDIA#678/NVIDIA#680/NVIDIA#683/NVIDIA#688/ NVIDIA#690/NVIDIA#691/NVIDIA#692. Resolution keeps this branch's design and ports main's content into it: - Naming: main's GpuRoce{Transceiver,Factory,LinkCheck} map 1:1 onto this branch's DeviceGraph* files (the de-transport-ing follow-up requested in the PR 670 review). Main's post-review fixes are ported into the renamed files: ring-size overflow + host-page-size alignment validation, gpu_id resolved from the decoder's cuda_device_id instead of an env var, the factory taking pinned_cuda_device, and the linkcheck exercising the new factory signature. - Schema: per-decoder 'dispatch: host|device_graph' plus the top-level transport section stay; main's per-decoder 'transport:' key does not return. cuda_device_id is adopted as a per-decoder placement knob (config struct + YAML trait), and the hsb script now injects 'dispatch: device_graph' + cuda_device_id and drives the server with QEC_DEVICE_GRAPH_* env and the device_graph READY sentinel. - Features adopted from main: decoder CUDA-device pinning (worker-thread pin via promise, graph-capture pin, hardware_guards.h), per-session decode counters + print_session_stats + QEC_DECODING_SERVER_STATS server-side stats printing, DecodingServer ctor exception safety, virtualized realtime decoder API (NVIDIA#674), CMP0126 fix, CUDA::cudart on the core lib, and the cudevice-archive dedup in the server link. - DecodingSession combines main's set_graph_capture_device with this branch's reserved-SMs decode-graph capture (QEC_DEVICE_GRAPH_RESERVED_SMS). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>

)" This reverts commit 2cdc9df. Signed-off-by: Angela Burton <angelab@nvidia.com>

…an FPGA or a QPU kernel A shipped example that compiles against the installed SDK and exercises the delivered decoding_server from two syndrome sources, plus its docs/CI registration and an FPGA RX-ring sizing fix in the unittests driver. The example (docs/sphinx/examples/qec/realtime_decoding_demo/): - surface_code_realtime_decoding.cpp (a slimmed surface_code-4-yaml: the Ising-bundle and syndrome-replay machinery removed) is built two ways by a standalone CMakeLists.txt using installed headers/libs only -- a generator (writes the decoder config, and for the FPGA source the playback syndrome file) and the lowered -cqr kernel (-frealtime-lowering). - run_realtime_decoding.sh drives either source with pymatching / trt_decoder / nv-qldpc-decoder: --source qpu-kernel the lowered kernel streams syndromes to the server over UDP (no hardware); --source fpga the delivered hololink_fpga_syndrome_playback streams syndromes from a real FPGA over RoCE (cpu_roce for CPU decoders, gpu_roce for the nv-qldpc device path). - Pass/fail matches the existing surface-code tests. The qpu-kernel path applies the same checks as the surface_code-4-yaml-test.sh ctest driver: hard decoder-failure greps, the "Number of corrections decoder found:" completion proof, a residual logical-error ceiling of num_shots/50 (min 1), the server-owned-decoders report, and a server dispatch-count floor of num_shots * (num_rounds + 3). The fpga path passes on the playback tool's per-shot correction verification, same as hsb_fpga_decoding_server_test.sh. - The gpu_roce RX ring runs at the HSB QP's 64-entry receive WQE depth (the receiver pre-posts one WQE per ring slot; deeper rings drop frames under load), with a preflight of DOCA's host-page alignment that fails fast with the stride constraint on 16K/64K-page hosts. The server gets --timeout=300 (total-runtime failsafe, matching the in-tree external-server tests). Docs + CI: - New narrative page examples_rst/qec/realtime_decoding_demo.rst (literalincludes the source), registered in the QEC examples toctree; the FPGA predecoder + Relay BP pages point at the example for end-to-end runs. - conf.py.in excludes examples/**/README.md from sphinx (example READMEs are for the shipped tree / GitHub; the '.md' source suffix would otherwise ingest them as toctree-orphan pages). - lib_qec.yaml: build-and-test (CPU) builds the two example binaries against the installed SDK (enforcing the installed-headers-only rule) and runs the qpu-kernel pymatching decode over UDP loopback -- a real decode, no hardware -- archiving the SDK install + example build for the GPU job. gpu-test runs the example's trt_decoder qpu-kernel decode over UDP reusing those artifacts (+ pip install onnx). This is the only CI coverage of trt_decoder THROUGH the decoding server: the TRTDecoder unit tests exercise the decoder standalone, and the app_examples happy-path trt test is configured out because its onnx gate is evaluated before onnx is pip-installed. Gated on TensorRT availability (skips arm64-cu12). - unittests/utils/CMakeLists.txt installs hololink_fpga_syndrome_playback alongside decoding_server (already hololink-gated; a no-op in the vanilla release) so the example can resolve it from the install prefix. hsb_fpga_decoding_server_test.sh -- fix HSB FPGA RX-ring sizing (two bugs, both rooted in the HSB QP's fixed 64 receive WQEs; the receiver pre-posts one WQE per ring slot, so a ring with MORE slots than WQEs drops frames): 1. cpu_roce modulus desync (regression from NVIDIA#683 / 2cdc9df): NVIDIA#683 added PLAYBACK_NUM_PAGES=512 as the playback --num-pages to size the gpu_roce ring, but run_playback is SHARED with cpu_roce, whose server RX ring is 64 slots. The FPGA writes frame rid to slot (rid % num-pages), so frames rid>=64 landed outside the server's registered MR and the test starved after ~10 shots (DISPATCHED=64). Fix: program the playback ring modulus from the server ring depth per transport, and clamp both to the WQE depth so the invariant is structural. 2. gpu_roce over-sized ring: the ring was 512 slots but the DOCA receiver only pre-posts 64 WQEs, dropping frames under load. Cap it at the WQE depth (clamp+warn on an over-sized --gpu-roce-num-pages), and preflight DOCA's host-page alignment (64 x 384 B covers 4K/8K-page kernels; 16K/64K-page hosts fail fast with the --page-size constraint spelled out). The emulator does not parse --num-pages (its RDMA-write modulus is the SIF target the playback programs), so the dead arg, the PLAYBACK_NUM_PAGES variable, and the --playback-num-pages option are removed. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>

…698) ## Background - #690 introduced the cuda_device_id placement knob for GPU decoders. - #678 made worker threads and the inproc graph capture honor the pinned device, and pinned decoder construction. - #683 closed the remaining gap: the gpu_roce decoding-server capture path now pins too. Each fix added a device pin to one more path via its own ad-hoc helper. The "which device" decision ended up duplicated across dispatch, capture, the worker, construction, and the gpu_roce transport. And that is the motivation for this PR. ## What this PR does Basically, every path should be following a single source of truth. Consolidate all of it to: the decoder's `cuda_device_id`, resolved one way (`decode_device_for`) and applied through two sanctioned wrappers — `pin_decode_device` (dispatch/decode) and `capture_graph_pinned` (graph capture) — in `hardware_guards.h.` Every path now derives its device from that one field; none selects a device by any other means. Unpinned (< 0) keeps existing semantics: dispatch no-ops, capture defaults to device 0. Signed-off-by: Melody Ren <melodyr@nvidia.com>

vedika-saravanan changed the title ~~Relaybp gpuroce gb200 fixes~~ GB200 fixes for HSB/FPGA RelayBP gpu_roce decoding-server path Jul 13, 2026

vedika-saravanan force-pushed the relaybp-gpuroce-gb200-fixes branch from 017229c to 8e5e9f7 Compare July 14, 2026 13:54

fix(qec/realtime): GB200 gpu_roce integration fixes: GPU affinity, pa…

f2eb103

…ge alignment, private dispatch symbols Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>

vedika-saravanan force-pushed the relaybp-gpuroce-gb200-fixes branch from 8e5e9f7 to 75780c3 Compare July 14, 2026 13:56

make gpuroce link check avoid transceiver construction

3de6452

Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>

vedika-saravanan force-pushed the relaybp-gpuroce-gb200-fixes branch from 75780c3 to 3de6452 Compare July 14, 2026 14:04

vedika-saravanan changed the title ~~GB200 fixes for HSB/FPGA RelayBP gpu_roce decoding-server path~~ Fix GB200 gpu_roce decoding-server validation issues Jul 14, 2026

vedika-saravanan force-pushed the relaybp-gpuroce-gb200-fixes branch 2 times, most recently from a92b03c to fc0ceaf Compare July 14, 2026 16:15

vedika-saravanan marked this pull request as ready for review July 14, 2026 16:29

fix(qec/realtime): avoid duplicate proprietary cudevice link

df7d51e

Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>

vedika-saravanan force-pushed the relaybp-gpuroce-gb200-fixes branch from fc0ceaf to df7d51e Compare July 14, 2026 16:39

vedika-saravanan requested review from bmhowe23, cketcham2333 and melody-ren July 14, 2026 16:43

bmhowe23 reviewed Jul 14, 2026

View reviewed changes

Comment thread libs/qec/lib/realtime/decoding-server-cqr/DecodingSession.cpp Outdated

bmhowe23 reviewed Jul 14, 2026

View reviewed changes

Comment thread libs/qec/lib/realtime/CMakeLists.txt Outdated

avoid installing private dispatch archive

b7c0963

Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>

cketcham2333 reviewed Jul 14, 2026

View reviewed changes

Comment thread libs/qec/lib/realtime/decoding-server-cqr/DecodingSession.cpp Outdated

cketcham2333 reviewed Jul 14, 2026

View reviewed changes

Comment thread libs/qec/tools/decoding-server/CMakeLists.txt

vedika-saravanan added 2 commits July 14, 2026 15:46

address pr comment

272b84d

Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>

Merge branch 'main' into relaybp-gpuroce-gb200-fixes

8ca1e87

bmhowe23 approved these changes Jul 14, 2026

View reviewed changes

vedika-saravanan merged commit 2cdc9df into NVIDIA:main Jul 14, 2026
24 checks passed

bmhowe23 mentioned this pull request Jul 15, 2026

DRAFT: QEC decoding server: transport-agnostic bridge providers, one ring per decoder, mixed host + device_graph dispatch #682

Draft

melody-ren mentioned this pull request Jul 15, 2026

Route every decoder device pin through one resolver and two wrappers #698

Merged

anjbur added a commit to anjbur/cudaqx that referenced this pull request Jul 16, 2026

Revert "Fix GB200 gpu_roce decoding-server validation issues (NVIDIA#683

b67e000

)" This reverts commit 2cdc9df. Signed-off-by: Angela Burton <angelab@nvidia.com>

cketcham2333 mentioned this pull request Jul 16, 2026

Add realtime_decoding_demo example: drive the delivered decoding server from an FPGA or a QPU kernel #703

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix GB200 gpu_roce decoding-server validation issues#683

Fix GB200 gpu_roce decoding-server validation issues#683
vedika-saravanan merged 6 commits into
NVIDIA:mainfrom
vedika-saravanan:relaybp-gpuroce-gb200-fixes

vedika-saravanan commented Jul 13, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmhowe23 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

vedika-saravanan commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Runtime / performance impact

Self-review checklist

Before requesting review

Scope and size

Tests

Documentation

Code style

Dependencies

Uh oh!

copy-pr-bot Bot commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmhowe23 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vedika-saravanan commented Jul 13, 2026 •

edited

Loading