BC4 Monitoring

Watching machine usage

The following command lists the usage of machines in the gpu partition in the comsm0018 account:

$ squeue -p gpu --states=running -A comsm0018

Check gpu node states

List the state of nodes in the gpu partition

$ sinfo -p gpu
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpu          up 7-00:00:00      1  down* gpu06
gpu          up 7-00:00:00     31   idle gpu[01-05,07-32]

Monitoring the queue backlog

$ squeue --Format 'username:10,account:12,priority,reservation:12,state:10,gres:10,minmemory:12,timeleft' --sort='P,b,m' --partition gpu
USER      ACCOUNT     PRIORITY            RESERVATION STATE     GRES      MIN_MEMORY  TIME_LEFT
el14718   betatest    0.00000023283064    (null)      RUNNING   gpu:1     1000M       1-21:23:36
el14718   betatest    0.00000023283064    (null)      RUNNING   gpu:1     1000M       1-21:23:36
el14718   betatest    0.00000023283064    (null)      RUNNING   gpu:1     1000M       1-21:23:36
el14718   betatest    0.00000023283064    (null)      RUNNING   gpu:1     1000M       1-21:23:06