Prerequisites
Bug Description
Background
NVIDIA Fabric Manager (FM) is a userspace daemon (nv-fabricmanager) that is required on all multi-GPU NVSwitch systems (DGX/HGX A100, H100, H200, B200, B300). It serves two critical functions:
-
Initialization: Configures the NVSwitch memory fabric across all GPUs -- trains NVSwitch-to-NVSwitch and NVSwitch-to-GPU NVLink interconnects and registers GPUs with the fabric. Each GPU transitions through fabric.state: "Not Started" → "In Progress" → "Completed".
-
Runtime: Monitors the NVLink/NVSwitch fabric for errors, coordinates link retraining on transient failures, and manages CUDA job lifecycle on the fabric.
FM exposes a management API on 127.0.0.1:6666 and is deployed either as:
- A systemd service (
nvidia-fabricmanager.service) on bare-metal / host-driver installations
- A child daemon process inside the GPU Operator's
nvidia-driver-ctr container
Evidence
We have two nodes that are reporting this issue, but they are in different states
Node 1
In this node the fabric manager failed was killed and failed to start:
# systemctl status nvidia-fabricmanager.service
× nvidia-fabricmanager.service - NVIDIA fabric manager service
Loaded: loaded (/usr/lib/systemd/system/nvidia-fabricmanager.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Fri 2026-01-30 11:58:43 UTC; 2 weeks 4 days ago
CPU: 7.387s
Jan 30 11:58:43 node systemd[1]: nvidia-fabricmanager.service: Killing process 76454 (nv-fabricmanage) with signal SIGKILL.
Jan 30 11:58:43 node systemd[1]: nvidia-fabricmanager.service: Failed with result 'exit-code'.
Jan 30 11:58:43 node systemd[1]: Failed to start nvidia-fabricmanager.service - NVIDIA fabric manager service.
Jan 30 11:58:43 node systemd[1]: nvidia-fabricmanager.service: Consumed 7.387s CPU time.
Jan 30 11:59:08 node systemd[1]: /usr/lib/systemd/system/nvidia-fabricmanager.service:18: PIDFile= references a path below legacy directory /var/run/, updating /var/run/nvidia-fabricmanag>
and this resulted in the fabric status not being reported:
# nvidia-smi --format=csv --query-gpu=platform.module_id,index,serial,uuid,pci.bus_id,fabric.state,fabric.status
platform.module_id, index, serial, uuid, pci.bus_id, fabric.state, fabric.status
7, 0, 111, GPU-aaa, 00000000:53:00.0, In Progress, N/A
5, 1, 111, GPU-aaa, 00000000:64:00.0, In Progress, N/A
6, 2, 111, GPU-aaa, 00000000:75:00.0, In Progress, N/A
8, 3, 111, GPU-aaa, 00000000:86:00.0, In Progress, N/A
1, 4, 111, GPU-aaa, 00000000:97:00.0, In Progress, N/A
3, 5, 111, GPU-aaa, 00000000:A8:00.0, In Progress, N/A
4, 6, 111, GPU-aaa, 00000000:B9:00.0, In Progress, N/A
2, 7, 111, GPU-aaa, 00000000:CA:00.0, In Progress, N/A
which in turn resulted in failing workloads
Node 2
On this node, the fabric manager is running, and it is managed by GPU operator, but at runtime it has encountered an error causing the fabric to go into a degraded state:
# nvidia-smi --format=csv --query-gpu=platform.module_id,index,serial,uuid,pci.bus_id,fabric.state,fabric.status
platform.module_id, index, serial, uuid, pci.bus_id, fabric.state, fabric.status
3, 0, 111, GPU-aaa, 00000001:00:00.0, Completed, Success
2, 1, 111, GPU-aaa, 00000002:00:00.0, Completed, Unknown Error
4, 2, 111, GPU-aaa, 00000003:00:00.0, Completed, Unknown Error
1, 3, 111, GPU-aaa, 00000008:00:00.0, Completed, Unknown Error
5, 4, 111, GPU-aaa, 00000009:00:00.0, Completed, Unknown Error
6, 5, 111, GPU-aaa, 0000000A:00:00.0, Completed, Unknown Error
8, 6, 111, GPU-aaa, 0000000B:00:00.0, Completed, Unknown Error
7, 7, 111, GPU-aaa, 0000000C:00:00.0, Completed, Success
In both cases neither GPU health monitor nor Syslog health monitor picked up the fabric state and reported degradation
On the second node DCGM health checks did not report any failures
Component
Health Monitor
Steps to Reproduce
Hard to repo without fabric errors
Environment
- NVSentinel version: v0.8.0
- Kubernetes version: N/A
- Deployment method: helm
Logs/Output
No response
Prerequisites
Bug Description
Background
NVIDIA Fabric Manager (FM) is a userspace daemon (
nv-fabricmanager) that is required on all multi-GPU NVSwitch systems (DGX/HGX A100, H100, H200, B200, B300). It serves two critical functions:Initialization: Configures the NVSwitch memory fabric across all GPUs -- trains NVSwitch-to-NVSwitch and NVSwitch-to-GPU NVLink interconnects and registers GPUs with the fabric. Each GPU transitions through
fabric.state:"Not Started"→"In Progress"→"Completed".Runtime: Monitors the NVLink/NVSwitch fabric for errors, coordinates link retraining on transient failures, and manages CUDA job lifecycle on the fabric.
FM exposes a management API on
127.0.0.1:6666and is deployed either as:nvidia-fabricmanager.service) on bare-metal / host-driver installationsnvidia-driver-ctrcontainerEvidence
We have two nodes that are reporting this issue, but they are in different states
Node 1
In this node the fabric manager failed was killed and failed to start:
and this resulted in the fabric status not being reported:
which in turn resulted in failing workloads
Node 2
On this node, the fabric manager is running, and it is managed by GPU operator, but at runtime it has encountered an error causing the fabric to go into a degraded state:
In both cases neither GPU health monitor nor Syslog health monitor picked up the fabric state and reported degradation
On the second node DCGM health checks did not report any failures
Component
Health Monitor
Steps to Reproduce
Hard to repo without fabric errors
Environment
Logs/Output
No response