You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Given an interval (e.g. 30 seconds), 1)list all the jobs, mpijobs, unified job; 2) find job associated pods, extract job status(pending/running/done/fail..) and associated pods resource utilizations (gpu, cpu, memory, network ...), 3) save to the info to MongoDB
Notes
1)avoid duplicated info due to custom job built on kubernetes job, e.g., mpijob will create job and then pods.
The text was updated successfully, but these errors were encountered:
GPU metrics are not merged to the Pod metrics. To get GPU utilization and GPU memory utilization, need to map from pod name to gpu process id. Refer to previous functions in app.py. A pod could use multiple GPUs and even cross nodes.
Given an interval (e.g. 30 seconds), 1)list all the jobs, mpijobs, unified job; 2) find job associated pods, extract job status(pending/running/done/fail..) and associated pods resource utilizations (gpu, cpu, memory, network ...), 3) save to the info to MongoDB
Notes
1)avoid duplicated info due to custom job built on kubernetes job, e.g., mpijob will create job and then pods.
The text was updated successfully, but these errors were encountered: