Skip to content

release v1.5.0

Latest
Latest

Choose a tag to compare

View all tags
@spraveenio spraveenio released this 08 May 18:49
v1.5.0
1ebd041
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v1.5.0

  • Unix Domain Socket For IPC

    • The GPU Agent now uses Unix Domain Socket (/var/run/gpuagent.sock) for communication with the metrics exporter
    • Secure and Improved performance Lower latency compared to TCP/IP connections

  • New Metrics

    • GPU_PROCESS_CU_OCCUPANCY to report Compute Unit occupancy for each process using the GPU, with process_id label to differentiate between processes
    • GPU_ECC_DEFERRED_* for each ECC supported block

  • Profiler Configuration Enhancement

    • SamplingInterval to set sampling window for the profiler metrics
    • Default value is 1000 microseconds (1 millisecond)

  • Configurable Health Polling Rate

    • Added PollingRate field to HealthService configuration in CommonConfig
    • Supports duration formats: 30s, 5m, 1h, 1d, 23h10m15s
    • Default: 30 seconds, Min: 30 seconds, Max: 24 hours

  • Slinky

    • Slinky job reporting is now supported, with labels providing both Kubernetes and Slurm job information

  • KFD_PROCESS_ID Label Now Optional

    • The KFD_PROCESS_ID label is no longer mandatory and is disabled by default to reduce metric cardinality
    • This change helps optimize Prometheus storage and query performance for deployments that don't require process-level tracking
    • For users who need this label, simply add it to the ConfigMap configuration. See Configuration Documentation for details

Known Issues

  • Power metrics intermittently missing for individual GPUs on MI350X (SPX/NPS1 mode)
    • gpu_package_power and gpu_power_usage may be absent for one or more GPUs on
      MI350X nodes in SPX/NPS1 mode. The affected GPU is present in all other metrics
      (temperature, VRAM, clocks) in the same response.
    • Root cause: If the first metrics poll after exporter startup returns 0 for a
      GPU's power (which occurs transiently on MI350X when PMFW/SMU telemetry is not yet
      populated), the exporter suppresses that GPU's power metrics until the exporter is
      restarted.
    • Workaround: Restart the exporter pod after the GPU has been running for a few
      seconds so the first poll sees non-zero power.

Platform Support

ROCm 7.2.X MI2xx, MI3xx

Assets 2
Loading