Nvidia multiple smi
Python bindings for pyNVML library over network
intended to work with python 3+
Based on pyNVML
- Allows you to get
nvidia-smioutput for multiple connected computers at once, and display it on a nice appindicator for Unity/Gnome users.
- Allows you to get a notification every time a new process is launched or finished. A minimum of 200MB memory use is needed for the notification to appear.
- This tool is aimed at small research teams, with multiple GPU-equipped computers, which you can manually ssh to. At a glance you can see every usage of your computer stock, and where you can launch your computation. It also provides some basis if you want to develop a tool to automatically launch your computation on the least busy computer of your network.
sudo python3 setup.py install
Server side services installation
To allow clients to access your computer's smi stats, simply run
But you can also enable it as a service that will be launched at boot.
- Ubuntu 15+ : the service file allow the server_smi to run automatically during boot
sudo systemctl daemon-reload sudo systemctl enable server_smi.service
sudo systemctl disable server_smi.service
- Ubuntu 14 : you have to daemonize the script and put it in init.d
sudo cp server_smi_daemon.sh /etc/init.d/. sudo chmod 0755 /etc/init.d/server_smi_daemon.sh sudo update-rc.d server_smi_daemon.sh defaults
sudo update-rc.d -f service_smi_daemon.sh remove
to run the appindicator
(gnome and unity environment only for the moment):
to run the client_smi (CLI version):
To add your own hosts, simply run a smi_client (CLI or gnome) once and add your entries in the json file that should be here:
Default ones are ml1 and ml2 with associated colors
Gpu usage stats:
Server-side, gpu usage history is stored in
~/.server_smi/history.csv if launched from CLI,
/etc/server_smi/history.csv if launched from systemctl/init.d. Usage is written on it every ~60 sec, feel free to make some data science with it.
To disable it, you can remove option
-s in server_smi.service (line 7) or server_smi_daemon.sh (line 6) before installing