Skip to content

ci: enable NUMA affinity syscalls and add gated multi-GPU QEC test lane#648

Merged
bmhowe23 merged 7 commits into
NVIDIA:mainfrom
kvmto:ci-qec-affinity-multigpu
Jul 6, 2026
Merged

ci: enable NUMA affinity syscalls and add gated multi-GPU QEC test lane#648
bmhowe23 merged 7 commits into
NVIDIA:mainfrom
kvmto:ci-qec-affinity-multigpu

Conversation

@kvmto

@kvmto kvmto commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Why

#634 adds hardware affinity (CUDA device / NUMA / CPU pinning) for QEC decoders, including a concurrent multi-decoder pool that places each decoder on its own GPU. Its test suite cannot currently run in CI for two infrastructure reasons:

  1. NUMA syscalls are blocked in the job containers. Docker's default seccomp profile only admits set_mempolicy / get_mempolicy / mbind when the container holds CAP_SYS_NICE. Without it these syscalls return EPERM, so the hardware-affinity tests detect this and skip.
  2. No multi-GPU runner in the lane. The QEC GPU job runs on linux-<arch>-gpu-a100-latest-1 (single GPU), so every >= 2 GPU placement test skips.

What

One file changed (.github/workflows/lib_qec.yaml):

  • build-and-test: container gains options: --cap-add=SYS_NICE. This is the narrowest grant that unblocks the NUMA syscalls — note other NVIDIA repos' GPU CI (e.g. quantum-predecoder, Ising-Decoding) runs --security-opt seccomp=unconfined, which is strictly broader.
  • New multi-gpu-test job: runs the multi-GPU decoder placement tests on linux-amd64-gpu-a100-latest-2. Deliberately kept out of the per-PR critical path:
    • triggers only on merges to the default branch or manual workflow_dispatch;
    • needs: build-and-test and reuses its uploaded artifact (no rebuild);
    • timeout-minutes: 45 so a missing runner label becomes a visible failure, not an indefinite queue;
    • explicit nvidia-smi check that >= 2 GPUs are visible;
    • ctest --no-tests=error so an empty test filter fails loudly instead of green-washing.

The existing gpu-test job is untouched — PR CI keeps its current runner footprint and risk profile.

What we need from DevOps

  1. Does linux-amd64-gpu-a100-latest-2 exist in the runner pool? The -latest-<gpu-count> naming pattern is in production use elsewhere (linux-amd64-gpu-rtxpro6000-latest-2 in Ising-Decoding), but we could not confirm a 2-GPU A100 label from public config. If it does not exist, can one be provisioned — or should this job target a different 2-GPU pool? A workflow_dispatch run of this job is a safe empirical test.
  2. Is --cap-add=SYS_NICE acceptable on the fleet? It is one capability (scheduler/affinity/NUMA-placement control inside the container), much narrower than the seccomp=unconfined already used by sibling repos.

Merge ordering

The test names in the multi-gpu-test filter land with #634. Merging this PR after #634 keeps the lane green from day one; merging before it means the multi-gpu-test job fails on main (by design — --no-tests=error) until #634 lands.

🤖 Generated with Claude Code

@kvmto kvmto requested a review from anjbur July 2, 2026 15:31
anjbur added 5 commits July 2, 2026 09:59
Signed-off-by: Angela Burton <angelab@nvidia.com>
Signed-off-by: Angela Burton <angelab@nvidia.com>
Signed-off-by: Angela Burton <angelab@nvidia.com>
Signed-off-by: Angela Burton <angelab@nvidia.com>
@anjbur

anjbur commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Added some changes to use a multi-GPU ARM runner. I also changed the test filter to not fail when no tests are detected, since the tests are being added in a separate PR.

@anjbur anjbur requested a review from bmhowe23 July 3, 2026 00:31

@bmhowe23 bmhowe23 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. We may have to re-evaluate how often we run it if any of our per-PR jobs start stalling waiting too long for these runners.

@bmhowe23 bmhowe23 enabled auto-merge (squash) July 6, 2026 20:21
@bmhowe23 bmhowe23 merged commit 5819650 into NVIDIA:main Jul 6, 2026
21 checks passed
@bmhowe23 bmhowe23 deleted the ci-qec-affinity-multigpu branch July 6, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants