NVIDIA Open GPU Kernel Modules Version
570.133
Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
Operating System and Version
ubuntu22.04
Kernel Release
6.8.0
Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
Hardware: GPU
5090
Describe the bug
I successfully install this modules and nvidia-smi topo -m -p2p p shows p2p status is ok，but when i use the latest nccl==2.26.2 to test，it just run into error which some output like “: Test CUDA failure common.cu:297 'an illegal memory access was encountered'”，and the same shit happened when i use p2psimpletest
To Reproduce
root@lam-MU72-SU0-00:/opt/cuda-samples# cd Samples/5_Domain_Specific/p2pBandwidthLatencyTest/
root@lam-MU72-SU0-00:/opt/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest# ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090 D, pciBusID: c3, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090 D, pciBusID: ca, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1506.75 21.16
1 21.36 1540.93
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure /opt/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu:234: 'unspecified launch failure'
root@lam-MU72-SU0-00:/opt/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest# cd /opt/nccl-tests/
root@lam-MU72-SU0-00:/opt/nccl-tests# cd build/
root@lam-MU72-SU0-00:/opt/nccl-tests/build# ./all_reduce_perf -e 2^C
root@lam-MU72-SU0-00:/opt/nccl-tests/build# export NCCL_DEBUG=INFO
root@lam-MU72-SU0-00:/opt/nccl-tests/build# ./
all_gather_perf all_reduce_perf alltoall_perf broadcast_perf gather_perf hypercube_perf reduce_perf reduce_scatter_perf scatter_perf sendrecv_perf verifiable/
root@lam-MU72-SU0-00:/opt/nccl-tests/build# ./all_reduce_perf -b 2G -e 10G -f 2 -c 0 -g2
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
nThread 1 nGpus 2 minBytes 2147483648 maxBytes 10737418240 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 0 graph: 0
Using devices
Rank 0 Group 0 Pid 28273 on lam-MU72-SU0-00 device 0 [0000:c3:00] NVIDIA GeForce RTX 5090 D
Rank 1 Group 0 Pid 28273 on lam-MU72-SU0-00 device 1 [0000:ca:00] NVIDIA GeForce RTX 5090 D
lam-MU72-SU0-00:28273:28273 [0] NCCL INFO Bootstrap: Using eno1:10.0.0.171<0>
lam-MU72-SU0-00:28273:28273 [1] NCCL INFO cudaDriverVersion 12080
lam-MU72-SU0-00:28273:28273 [0] NCCL INFO NCCL version 2.26.2+cuda12.8
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal net plugin.
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO NET/IB : No device found.
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO NET/IB : Using [RO]; OOB eno1:10.0.0.171<0>
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO NET/Socket : Using [0]eno1:10.0.0.171<0>
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Using network Socket
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Using network Socket
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO ncclCommInitAll comm 0x62fb8090ffb0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId c3000 commId 0x5ff11f9dc5c16162 - Init START
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO ncclCommInitAll comm 0x62fb809979a0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId ca000 commId 0x5ff11f9dc5c16162 - Init START
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Bootstrap timings total 0.001470 (create 0.000038, send 0.000170, recv 0.000600, ring 0.000050, delay 0.000001)
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Bootstrap timings total 0.001310 (create 0.000096, send 0.000356, recv 0.000368, ring 0.000039, delay 0.000000)
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Setting affinity for GPU 1 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Setting affinity for GPU 0 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO comm 0x62fb809979a0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO comm 0x62fb8090ffb0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Channel 00/02 : 0 1
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Channel 01/02 : 0 1
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO P2P Chunksize set to 131072
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO P2P Chunksize set to 131072
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Check P2P Type intraNodeP2pSupport 1 directMode 1
lam-MU72-SU0-00:28273:28357 [0] NCCL INFO [Proxy Service] Device 0 CPU core 129
lam-MU72-SU0-00:28273:28359 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 131
lam-MU72-SU0-00:28273:28356 [1] NCCL INFO [Proxy Service] Device 1 CPU core 128
lam-MU72-SU0-00:28273:28358 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 130
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO CC Off, workFifoBytes 1048576
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO ncclCommInitAll comm 0x62fb809979a0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId ca000 commId 0x5ff11f9dc5c16162 - Init COMPLETE
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 2 total 0.41 (kernels 0.29, alloc 0.11, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO ncclCommInitAll comm 0x62fb8090ffb0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId c3000 commId 0x5ff11f9dc5c16162 - Init COMPLETE
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 2 total 0.41 (kernels 0.28, alloc 0.11, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
out-of-place in-place
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
lam-MU72-SU0-00:28273:28360 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer
lam-MU72-SU0-00:28273:28361 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
lam-MU72-SU0-00:28273:28360 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer
lam-MU72-SU0-00:28273:28361 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
lam-MU72-SU0-00:28273:28361 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
lam-MU72-SU0-00:28273:28360 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
lam-MU72-SU0-00: Test CUDA failure common.cu:297 'an illegal memory access was encountered'
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:407
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:594
.. lam-MU72-SU0-00 pid 28273: Test failure all_reduce.cu:90
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:625
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:1123
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:893
root@lam-MU72-SU0-00:/opt/nccl-tests/build# dmesg |grep -i iommu
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic root=UUID=2759d1d4-4464-4bc2-b464-1f0181b938c2 ro quiet splash intel_iommu=off vt.handoff=7
[ 1.617695] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic root=UUID=2759d1d4-4464-4bc2-b464-1f0181b938c2 ro quiet splash intel_iommu=off vt.handoff=7
[ 1.617805] DMAR: IOMMU disabled
[ 5.446956] iommu: Default domain type: Translated
[ 5.446956] iommu: DMA domain TLB invalidation policy: lazy mode
root@lam-MU72-SU0-00:/opt/nccl-tests/build# nvidia-smi
Thu Mar 27 19:54:46 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5090 D Off | 00000000:C3:00.0 Off | N/A |
| 0% 35C P0 57W / 575W | 0MiB / 32607MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 5090 D Off | 00000000:CA:00.0 Off | N/A |
| 0% 43C P0 84W / 575W | 0MiB / 32607MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvvs c3:00.0|grep -i wid
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvvs ca:00.0|grep -i wid
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vs c3:00.0|grep -i bar1
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvss c3:00.0|grep -i bar1
lspci: -s: Invalid slot number
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvvs c3:00.0|grep -i bar1
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvvs c3:00.0|grep -i bar
Vector table: BAR=0 offset=00b90000
PBA: BAR=0 offset=00ba0000
Capabilities: [134 v1] Physical Resizable BAR
BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [140 v1] Virtual Resizable BAR
BAR 2: current size: 256MB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB 256TB 512TB 1PB 2PB 4PB 8PB 16PB 32PB 64PB 128PB 256PB 512PB 1EB 2EB 4EB 8EB
Bug Incidence
Always
nvidia-bug-report.log.gz
。
More Info
No response
NVIDIA Open GPU Kernel Modules Version
570.133
Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
Operating System and Version
ubuntu22.04
Kernel Release
6.8.0
Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
Hardware: GPU
5090
Describe the bug
I successfully install this modules and nvidia-smi topo -m -p2p p shows p2p status is ok，but when i use the latest nccl==2.26.2 to test，it just run into error which some output like “: Test CUDA failure common.cu:297 'an illegal memory access was encountered'”，and the same shit happened when i use p2psimpletest
To Reproduce
root@lam-MU72-SU0-00:/opt/cuda-samples# cd Samples/5_Domain_Specific/p2pBandwidthLatencyTest/
root@lam-MU72-SU0-00:/opt/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest# ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090 D, pciBusID: c3, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090 D, pciBusID: ca, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1506.75 21.16
1 21.36 1540.93
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure /opt/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu:234: 'unspecified launch failure'
root@lam-MU72-SU0-00:/opt/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest# cd /opt/nccl-tests/
root@lam-MU72-SU0-00:/opt/nccl-tests# cd build/
root@lam-MU72-SU0-00:/opt/nccl-tests/build# ./all_reduce_perf -e 2^C
root@lam-MU72-SU0-00:/opt/nccl-tests/build# export NCCL_DEBUG=INFO
root@lam-MU72-SU0-00:/opt/nccl-tests/build# ./
all_gather_perf all_reduce_perf alltoall_perf broadcast_perf gather_perf hypercube_perf reduce_perf reduce_scatter_perf scatter_perf sendrecv_perf verifiable/
root@lam-MU72-SU0-00:/opt/nccl-tests/build# ./all_reduce_perf -b 2G -e 10G -f 2 -c 0 -g2
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
nThread 1 nGpus 2 minBytes 2147483648 maxBytes 10737418240 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 0 graph: 0
Using devices
Rank 0 Group 0 Pid 28273 on lam-MU72-SU0-00 device 0 [0000:c3:00] NVIDIA GeForce RTX 5090 D
Rank 1 Group 0 Pid 28273 on lam-MU72-SU0-00 device 1 [0000:ca:00] NVIDIA GeForce RTX 5090 D
lam-MU72-SU0-00:28273:28273 [0] NCCL INFO Bootstrap: Using eno1:10.0.0.171<0>
lam-MU72-SU0-00:28273:28273 [1] NCCL INFO cudaDriverVersion 12080
lam-MU72-SU0-00:28273:28273 [0] NCCL INFO NCCL version 2.26.2+cuda12.8
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal net plugin.
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO NET/IB : No device found.
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO NET/IB : Using [RO]; OOB eno1:10.0.0.171<0>
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO NET/Socket : Using [0]eno1:10.0.0.171<0>
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Using network Socket
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Using network Socket
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO ncclCommInitAll comm 0x62fb8090ffb0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId c3000 commId 0x5ff11f9dc5c16162 - Init START
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO ncclCommInitAll comm 0x62fb809979a0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId ca000 commId 0x5ff11f9dc5c16162 - Init START
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Bootstrap timings total 0.001470 (create 0.000038, send 0.000170, recv 0.000600, ring 0.000050, delay 0.000001)
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Bootstrap timings total 0.001310 (create 0.000096, send 0.000356, recv 0.000368, ring 0.000039, delay 0.000000)
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Setting affinity for GPU 1 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Setting affinity for GPU 0 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO comm 0x62fb809979a0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO comm 0x62fb8090ffb0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Channel 00/02 : 0 1
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Channel 01/02 : 0 1
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO P2P Chunksize set to 131072
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO P2P Chunksize set to 131072
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Check P2P Type intraNodeP2pSupport 1 directMode 1
lam-MU72-SU0-00:28273:28357 [0] NCCL INFO [Proxy Service] Device 0 CPU core 129
lam-MU72-SU0-00:28273:28359 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 131
lam-MU72-SU0-00:28273:28356 [1] NCCL INFO [Proxy Service] Device 1 CPU core 128
lam-MU72-SU0-00:28273:28358 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 130
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO CC Off, workFifoBytes 1048576
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO ncclCommInitAll comm 0x62fb809979a0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId ca000 commId 0x5ff11f9dc5c16162 - Init COMPLETE
lam-MU72-SU0-00:28273:28338 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 2 total 0.41 (kernels 0.29, alloc 0.11, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO ncclCommInitAll comm 0x62fb8090ffb0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId c3000 commId 0x5ff11f9dc5c16162 - Init COMPLETE
lam-MU72-SU0-00:28273:28337 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 2 total 0.41 (kernels 0.28, alloc 0.11, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
out-of-place in-place
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
lam-MU72-SU0-00:28273:28360 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer
lam-MU72-SU0-00:28273:28361 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
lam-MU72-SU0-00:28273:28360 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer
lam-MU72-SU0-00:28273:28361 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
lam-MU72-SU0-00:28273:28361 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
lam-MU72-SU0-00:28273:28360 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
lam-MU72-SU0-00: Test CUDA failure common.cu:297 'an illegal memory access was encountered'
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:407
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:594
.. lam-MU72-SU0-00 pid 28273: Test failure all_reduce.cu:90
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:625
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:1123
.. lam-MU72-SU0-00 pid 28273: Test failure common.cu:893
root@lam-MU72-SU0-00:/opt/nccl-tests/build# dmesg |grep -i iommu
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic root=UUID=2759d1d4-4464-4bc2-b464-1f0181b938c2 ro quiet splash intel_iommu=off vt.handoff=7
[ 1.617695] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic root=UUID=2759d1d4-4464-4bc2-b464-1f0181b938c2 ro quiet splash intel_iommu=off vt.handoff=7
[ 1.617805] DMAR: IOMMU disabled
[ 5.446956] iommu: Default domain type: Translated
[ 5.446956] iommu: DMA domain TLB invalidation policy: lazy mode
root@lam-MU72-SU0-00:/opt/nccl-tests/build# nvidia-smi
Thu Mar 27 19:54:46 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5090 D Off | 00000000:C3:00.0 Off | N/A |
| 0% 35C P0 57W / 575W | 0MiB / 32607MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 5090 D Off | 00000000:CA:00.0 Off | N/A |
| 0% 43C P0 84W / 575W | 0MiB / 32607MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvvs c3:00.0|grep -i wid
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvvs ca:00.0|grep -i wid
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vs c3:00.0|grep -i bar1
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvss c3:00.0|grep -i bar1
lspci: -s: Invalid slot number
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvvs c3:00.0|grep -i bar1
root@lam-MU72-SU0-00:/opt/nccl-tests/build# lspci -vvvs c3:00.0|grep -i bar
Vector table: BAR=0 offset=00b90000
PBA: BAR=0 offset=00ba0000
Capabilities: [134 v1] Physical Resizable BAR
BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [140 v1] Virtual Resizable BAR
BAR 2: current size: 256MB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB 256TB 512TB 1PB 2PB 4PB 8PB 16PB 32PB 64PB 128PB 256PB 512PB 1EB 2EB 4EB 8EB
Bug Incidence
Always
nvidia-bug-report.log.gz
。
More Info
No response