Self-hosted GPU runner out-of-memory & CUDA container cache invalidation during multi-worker CI matrix

[Tekhabeeb]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

ARC (Actions Runner Controller)

Discussion Details

Hi everyone,

We are configuring a self-hosted GitHub Actions runner (Ubuntu 22.04 LTS, NVIDIA RTX A2000, Docker v24.0.7) to run automated regression and liveness-detection test suites for our on-prem face recognition SDK
pipeline.

During the integration test step across a matrix of 4 concurrent test workers, the runner intermittently fails to mount the GPU container layer, throwing this exit code:

text
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: device error: unknown device id: gpu0: unknown
Error: Process completed with exit code 1.

Here is a stripped-down snippet from our .github/workflows/sdk-integration.yml:

yaml
jobs:
run-liveness-suite:
runs-on: [self-hosted, linux, x64, gpu]
strategy:
matrix:
worker-id: [1, 2, 3, 4]
steps:
- uses: actions/checkout@v4
- name: Run Test Container with NVIDIA Runtime
run: |
docker run --rm --gpus all
-v ${{ github.workspace }}/fixtures:/data/test-faces:ro
sdk-tester:local --worker-id ${{ matrix.worker-id }}

When jobs run sequentially (matrix.max-parallel: 1), everything passes without issue. However, running them concurrently causes nvidia-container-runtime to choke on resource allocation.

Has anyone encountered similar GPU isolation flaps on self-hosted runners when running multiple parallel Docker containers? Would partitioning via NVIDIA MIG or binding explicit NVIDIA_VISIBLE_DEVICES per runner instance be the recommended approach here?

[helmipradita]

Here is what is happening and how to properly configure your runner for concurrent test containers:

---

1. Does the RTX A2000 support NVIDIA MIG?

No. Multi-Instance GPU (MIG) is only supported on enterprise Hopper/Ampere datacenter architectures (such as the A100, A30, and H100). The RTX A2000 (GA106 silicon) does not have hardware MIG support, so hardware partitioning is not an option here.

---

2. Root Cause of unknown device id: gpu0

When 4 matrix workers execute docker run --rm --gpus all simultaneously on the same host, the NVIDIA Container Toolkit hook (nvidia-container-cli) executes concurrent NVML queries to discover and mount the GPU device nodes (/dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm).

Under rapid parallel invocations on Docker 24.x, nvidia-container-cli encounters a known lock contention in the device discovery hook, failing with:

nvidia-container-cli: device error: unknown device id: gpu0: unknown

This is why max-parallel: 1 succeeds reliably—the sequential execution avoids the NVML hook race.

---

3. Recommended Fixes

Fix A: Replace --gpus all with explicit device targeting and host IPC

Instead of using --gpus all (which triggers global NVML enumeration in every container), pass the explicit device index and enable host IPC sharing:

      - name: Run Test Container with NVIDIA Runtime
        run: |
          docker run --rm \
            --gpus device=0 \
            --ipc=host \
            -e NVIDIA_VISIBLE_DEVICES=0 \
            -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
            -v ${{ github.workspace }}/fixtures:/data/test-faces:ro \
            sdk-tester:local --worker-id ${{ matrix.worker-id }}

Fix B: Enable NVIDIA MPS (Multi-Process Service) for CUDA concurrency

Because you are running multiple parallel face-detection inference workloads on a single GPU, separate CUDA contexts will contend for GPU scheduling and VRAM (the RTX A2000 has 6 GB or 12 GB).

Enabling NVIDIA MPS allows multiple containers to multiplex into a single shared CUDA context, eliminating context-switching penalties and driver hook races:

1. On your self-hosted host machine, start the MPS daemon:

   sudo nvidia-smi -c EXCLUSIVE_PROCESS
   sudo nvidia-cuda-mps-control -d

2. In your workflow, mount the MPS control directory:

          docker run --rm \
            --gpus device=0 \
            --ipc=host \
            -v /tmp/nvidia-mps:/tmp/nvidia-mps \
            -e CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps \
            -v ${{ github.workspace }}/fixtures:/data/test-faces:ro \
            sdk-tester:local --worker-id ${{ matrix.worker-id }}

Fix C: Limit matrix concurrency if VRAM is constrained

If your face-recognition models require significant memory when loaded in parallel, 4 simultaneous instances might exceed the RTX A2000's memory ceiling. Setting max-parallel: 2 provides parallelism while maintaining a safe memory buffer:

    strategy:
      matrix:
        worker-id: [1, 2, 3, 4]
      max-parallel: 2