Skip to content

BadisaMosesane/HPC_monitoring

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
## PREPARE AND SEND A REPORT echo "To: $CONTACT" >> $MAIL_FILE echo "Subject: LSST: new verify-worker nodes down according to Slurm" >> $MAIL_FILE echo "From: $FROM" >> $MAIL_FILE echo "" >> $MAIL_FILE echo "The following new nodes are down/down* according to Slurm (down = unavailable for use, down* = down and not responding):" >> $MAIL_FILE if [[ ! -f $NEW_NODES_FILE ]]; then touch $NEW_NODES_FILE fi for j in $(cat $NEW_NODES_FILE) do if [[ -f $DOWN_NODES_CURRENT_FILE && $(grep ${j} $DOWN_NODES_CURRENT_FILE) ]]; then echo "${j} down" >> $MAIL_FILE elif [[ -f $DOWNSTAR_NODES_CURRENT_FILE && $(grep ${j} $DOWNSTAR_NODES_CURRENT_FILE) ]]; then echo "${j} down*" >> $MAIL_FILE else exit 1 fi done for k in $(cat $STATE_CURRENT_FILE) do if [[ ! $(grep ${k} $NEW_NODES_FILE) ]]; then echo "${k}" >> $OLD_NODES_FILE fi done if [[ -f $OLD_NODES_FILE ]]; then echo "" >> $MAIL_FILE echo "The following other nodes are still down
## PREPARE AND SEND A REPORT echo "To: $CONTACT" >> $MAIL_FILE echo "Subject: LSST: new verify-worker nodes down according to Slurm" >> $MAIL_FILE echo "From: $FROM" >> $MAIL_FILE echo "" >> $MAIL_FILE echo "The following new nodes are down/down* according to Slurm (down = unavailable for use, down* = down and not responding):" >> $MAIL_FILE if [[ ! -f $NEW_NODES_FILE ]]; then touch $NEW_NODES_FILE fi for j in $(cat $NEW_NODES_FILE) do if [[ -f $DOWN_NODES_CURRENT_FILE && $(grep ${j} $DOWN_NODES_CURRENT_FILE) ]]; then echo "${j} down" >> $MAIL_FILE elif [[ -f $DOWNSTAR_NODES_CURRENT_FILE && $(grep ${j} $DOWNSTAR_NODES_CURRENT_FILE) ]]; then echo "${j} down*" >> $MAIL_FILE else exit 1 fi done for k in $(cat $STATE_CURRENT_FILE) do if [[ ! $(grep ${k} $NEW_NODES_FILE) ]]; then echo "${k}" >> $OLD_NODES_FILE fi done if [[ -f $OLD_NODES_FILE ]]; then echo "" >> $MAIL_FILE echo "The following other nodes are still down
 
 
README.md
README.md
 
 
compare_node_states.sh
compare_node_states.sh
 
 
no_down_nodes.sh
no_down_nodes.sh
 
 
send_mail.sh
send_mail.sh
 
 
slurm_monitor_nodes.sh
slurm_monitor_nodes.sh
 
 

Repository files navigation

HPC_monitoring

Using sinfo to monitor slurm nodes

Looking for nodes that are down - using slurm, then dump info for visualization on Grafana panel. This assumes using the TIG stack

About

Using sinfo to monitor slurm nodes

Resources

Readme
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages