NVIDIA GPU Monitoring Tools
NVML Go Bindings
NVIDIA Management Library (NVML) is a C-based API for monitoring and managing NVIDIA GPU devices. NVML go bindings are taken from nvidia-docker 1.0 with some improvements and additions. NVML headers are also added to the package to make it easy to use and build.
Three samples are included to demonstrate how to use the NVML API.
DCGM Go Bindings
NVIDIA Data Center GPU Manager (DCGM) is a set of tools for managing and monitoring NVIDIA GPUs in cluster environments. It's a low overhead tool suite that performs a variety of functions on each host system including active health monitoring, diagnostics, system validation, policies, power and clock management, group configuration and accounting.
DCGM go bindings makes administering and monitoring containerized GPU applications easy.
GPU metrics exporter for Prometheus leveraging NVIDIA Data Center GPU Manager (DCGM) is a simple shell script that starts nv-hostengine, reads GPU metrics every 1 second and converts it to a standard Prometheus format.
Find the installation and run instructions here.