GPU_MONITOR

A lightweight script for GPU monitoring. Automatically emails the user when VRAM usage drops below a specified threshold and then exits the script.

一个轻量级 GPU 监控脚本，当显存使用情况低于预设阈值时将会自动发送邮件至指定邮箱，并退出脚本。

This script is mainly used to check whether the training is completed and whether the GPUs are idle.

本脚本主要用于查看训练是否完成/显卡是否闲置。

1. GPU 监控设置(GPU Monitoring Settings)

• 本脚本支持通过 gpu_id 监控指定GPU显，threshold是预设的显存阈值，默认为10MiB，check_interval为检测时间间隔，默认为300s。
• This script supports monitoring the VRAM of a specified GPU through the gpu_id. The threshold is the preset video memory threshold, with a default value of 10 MiB. check_interval is the detection time interval, with a default value of 300 seconds.

GPU_MONITOR/monitor.py

Lines 128 to 129 in 768c15b

	gpu_to_monitor = [0] # 修改为您需要监控的GPU ID列表
	monitor_gpu(threshold=10, check_interval=300, gpu_ids=gpu_to_monitor)# 启动GPU监控，只监控指定的GPU, 此处的阈值为10MB

---

2. 邮箱服务器配置(Email Server Configuration)

• 默认使用QQ邮箱，更换邮箱需修改 smtp_server 和 smtp_port
• Default QQ Mail. For other providers, update smtp_server & smtp_port

GPU_MONITOR/monitor.py

Lines 8 to 12 in 5c82370

	sender = 'youremails@qq.com'
	receiver = ['youremails@qq.com']
	password = 'yourpassword'
	smtp_server = 'smtp.qq.com'
	smtp_port = 465

---

3. 安全密码获取(Secure Password Acquisition)

• 必须使用邮箱安全密码（非登录密码）
• Use application-specific password (NOT login password)

Reference Blog

4. 运行(Run)

nohup python monitor.py >monitor.log 2>&1&