Hi team, I am setting MIG in a100 GPU but blocked at the very beginning which is the Enable MIG in the machine. could you please help look at this configuration problem when you have time? Appreciated!
here is the init nvidia-smi out put:
every time i config the gpus to the below and then reboot the config about mig is back to the init config.
nvidia-smi -a
Timestamp : Thu Nov 3 05:46:34 2022
Driver Version : 450.203.03
CUDA Version : 11.0
Attached GPUs : 8
GPU 00000000:07:00.0
Product Name : A100-SXM4-40GB
Product Brand : Tesla
Display Mode : Enabled
Display Active : Disabled
Persistence Mode : Enabled
MIG Mode
Current : Disabled
Pending : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : 1562021001122
GPU UUID : GPU-36c9561a-02ef-5c4d-3733-f0a56c42dc45
Minor Number : 0
VBIOS Version : 92.00.45.00.03
MultiGPU Board : No
Board ID : 0x700
GPU Part Number : 692-2G506-0200-002
Inforom Version
Image Version : G506.0200.00.04
OEM Object : 2.0
ECC Object : 6.16
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x07
Device : 0x00
Domain : 0x0000
Device Id : 0x20B010DE
Bus Id : 00000000:07:00.0
Sub System Id : 0x134F10DE
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 0 KB/s
Rx Throughput : 0 KB/s
Fan Speed : N/A
Performance State : P0
...
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
SRAM Correctable : 0
SRAM Uncorrectable : 0
DRAM Correctable : 0
DRAM Uncorrectable : 0
Aggregate
SRAM Correctable : 0
SRAM Uncorrectable : 0
DRAM Correctable : 0
DRAM Uncorrectable : 0
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows
Correctable Error : 0
Uncorrectable Error : 0
Pending : No
Remapping Failure Occurred : No
Bank Remap Availability Histogram
Max : 640 bank(s)
High : 0 bank(s)
Partial : 0 bank(s)
Low : 0 bank(s)
None : 0 bank(s)
Temperature
GPU Current Temp : 31 C
GPU Shutdown Temp : 92 C
GPU Slowdown Temp : 89 C
GPU Max Operating Temp : 85 C
Memory Current Temp : 30 C
Memory Max Operating Temp : 95 C
Power Readings
Power Management : Supported
Power Draw : 57.13 W
Power Limit : 400.00 W
Default Power Limit : 400.00 W
Enforced Power Limit : 400.00 W
Min Power Limit : 100.00 W
Max Power Limit : 400.00 W
Clocks
Graphics : 210 MHz
SM : 210 MHz
Memory : 1215 MHz
Video : 585 MHz
Applications Clocks
Graphics : 1095 MHz
Memory : 1215 MHz
Default Applications Clocks
Graphics : 1095 MHz
Memory : 1215 MHz
Max Clocks
Graphics : 1410 MHz
SM : 1410 MHz
Memory : 1215 MHz
Video : 1290 MHz
Max Customer Boost Clocks
Graphics : 1410 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : None
Hi team, I am setting MIG in a100 GPU but blocked at the very beginning which is the Enable MIG in the machine. could you please help look at this configuration problem when you have time? Appreciated!
here is the init
nvidia-smiout put:every time i config the gpus to the below and then reboot the config about mig is back to the init config.
nvidia-smi -a