ci: enable NUMA affinity syscalls and add gated multi-GPU QEC test lane#648
Merged
Conversation
Signed-off-by: kvmto <kmato@nvidia.com>
Signed-off-by: Angela Burton <angelab@nvidia.com>
Signed-off-by: Angela Burton <angelab@nvidia.com>
Signed-off-by: Angela Burton <angelab@nvidia.com>
Signed-off-by: Angela Burton <angelab@nvidia.com>
Collaborator
|
Added some changes to use a multi-GPU ARM runner. I also changed the test filter to not fail when no tests are detected, since the tests are being added in a separate PR. |
bmhowe23
approved these changes
Jul 6, 2026
bmhowe23
left a comment
Collaborator
There was a problem hiding this comment.
This looks good to me. We may have to re-evaluate how often we run it if any of our per-PR jobs start stalling waiting too long for these runners.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#634 adds hardware affinity (CUDA device / NUMA / CPU pinning) for QEC decoders, including a concurrent multi-decoder pool that places each decoder on its own GPU. Its test suite cannot currently run in CI for two infrastructure reasons:
set_mempolicy/get_mempolicy/mbindwhen the container holdsCAP_SYS_NICE. Without it these syscalls return EPERM, so the hardware-affinity tests detect this and skip.linux-<arch>-gpu-a100-latest-1(single GPU), so every >= 2 GPU placement test skips.What
One file changed (
.github/workflows/lib_qec.yaml):build-and-test: container gainsoptions: --cap-add=SYS_NICE. This is the narrowest grant that unblocks the NUMA syscalls — note other NVIDIA repos' GPU CI (e.g. quantum-predecoder, Ising-Decoding) runs--security-opt seccomp=unconfined, which is strictly broader.multi-gpu-testjob: runs the multi-GPU decoder placement tests onlinux-amd64-gpu-a100-latest-2. Deliberately kept out of the per-PR critical path:workflow_dispatch;needs: build-and-testand reuses its uploaded artifact (no rebuild);timeout-minutes: 45so a missing runner label becomes a visible failure, not an indefinite queue;nvidia-smicheck that >= 2 GPUs are visible;ctest --no-tests=errorso an empty test filter fails loudly instead of green-washing.The existing
gpu-testjob is untouched — PR CI keeps its current runner footprint and risk profile.What we need from DevOps
linux-amd64-gpu-a100-latest-2exist in the runner pool? The-latest-<gpu-count>naming pattern is in production use elsewhere (linux-amd64-gpu-rtxpro6000-latest-2in Ising-Decoding), but we could not confirm a 2-GPU A100 label from public config. If it does not exist, can one be provisioned — or should this job target a different 2-GPU pool? Aworkflow_dispatchrun of this job is a safe empirical test.--cap-add=SYS_NICEacceptable on the fleet? It is one capability (scheduler/affinity/NUMA-placement control inside the container), much narrower than theseccomp=unconfinedalready used by sibling repos.Merge ordering
The test names in the
multi-gpu-testfilter land with #634. Merging this PR after #634 keeps the lane green from day one; merging before it means themulti-gpu-testjob fails on main (by design —--no-tests=error) until #634 lands.🤖 Generated with Claude Code