Skip to content

fix(metrics): emit gpu_usage_percent as float for Cloud Monitoring (FILTER-585)#113

Merged
shingonoide merged 4 commits into
mainfrom
openfilter_gpu-usage-float-cast
Jun 29, 2026
Merged

fix(metrics): emit gpu_usage_percent as float for Cloud Monitoring (FILTER-585)#113
shingonoide merged 4 commits into
mainfrom
openfilter_gpu-usage-float-cast

Conversation

@shingonoide

@shingonoide shingonoide commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

📋 What does this PR do?

Casts the gpu_usage_percent system-health metric to float so it is always emitted as a floating-point value.

- self.gpu_usage_percent = gpu.get('gpu0', 0.0)
+ self.gpu_usage_percent = float(gpu.get('gpu0', 0.0))

🔍 Why is this needed?

NVML reports GPU utilization as an integer, so gpu_usage_percent was emitted as an integer whenever a GPU was active (the float default 0.0 only applies when no GPU is present). The Google Cloud Monitoring descriptor for this gauge is type-locked to DOUBLE, so every active-GPU point was rejected by the backend with:

value type for metric must be DOUBLE, but is INT64

and silently dropped (a partial, single-point rejection per export batch). The result is silent gaps in GPU-utilization data. Forcing the value to float makes the emitted points match the existing DOUBLE descriptor, restoring ingestion with no descriptor migration.

The change is intentionally narrow: the per-device <filter>_gpuN counters and the legitimately-integer gpu_accessible / camera_connected metrics are left untouched, since their descriptors are integer-typed and casting them would cause the opposite rejection.

🧪 How was it tested?

  • New unit test tests/test_gpu_usage_percent_float.py drives the GPU poll path with a mocked integer NVML reading and asserts gpu_usage_percent is a float equal to 55.0. It fails on the previous (un-cast) code and passes with this change.
  • Existing GPU / telemetry / timing suites remain green (tests/test_gpu.py, tests/test_telemetry.py, tests/test_timing_metrics.py).

🔗 Related Issues

FILTER-585

✅ Checklist

  • I have read and agreed to the terms of the LICENSE
  • I have read the CONTRIBUTING guide
  • I have followed the coding style
  • I have signed all commits in compliance with the DCO (git commit -s)
  • I have added or updated tests as needed
  • I have added or updated documentation as needed

View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

…ILTER-585)

NVML reports GPU utilization as an integer, so gpu_usage_percent was
emitted as INT64 whenever a GPU was active. The Cloud Monitoring
descriptor for this gauge is type-locked to DOUBLE, so every active-GPU
point was rejected with 'value type for metric must be DOUBLE, but is
INT64' and silently dropped. Cast the value to float so it matches the
descriptor. The per-device <filter>_gpuN counters and the INT64
gpu_accessible/camera_connected metrics are deliberately left untouched.

Adds a regression test that drives the GPU poll path with an integer
NVML reading and asserts a float result.

Signed-off-by: Rui Andrada <randrada@plainsight.ai>
@shingonoide shingonoide self-assigned this Jun 28, 2026
Signed-off-by: Rui Andrada <randrada@plainsight.ai>
…TER-585)

Signed-off-by: Rui Andrada <randrada@plainsight.ai>
@shingonoide shingonoide requested a review from lucasmundim June 28, 2026 21:50

@leandrobmarinho leandrobmarinho left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Summary: Casts gpu_usage_percent to float in Metrics.gpu_thread_func so an active GPU's integer gpu_util (gpu.py:71, gpu_util: int) no longer gets rejected by the DOUBLE-locked openfilter_gpu_usage_percent Cloud Monitoring descriptor (value type ... must be DOUBLE, but is INT64).

Verified the fix is correct and minimal: gpu['gpu0'] derives from the int gpu_util when a GPU is present, so the old code emitted an int; the cast forces DOUBLE. Scope is right — the per-device gpuN series and the legitimately INT64 descriptors (gpu_accessible, camera_connected) are untouched, and the downstream observability/client.py handles the value generically. No contract break, no reference/mutation concern.

FYI (non-blocking): test_gpu_usage_percent_is_float_when_gpu0_absent doesn't actually guard the regression — the no-gpu0 default 0.0 was already a float on both old and new code, so it passes either way; the gpu0-present test is the one that exercises the fix. No change needed to merge.

…-585)

Signed-off-by: Rui Andrada <randrada@plainsight.ai>

@lucasmundim lucasmundim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: the float() cast is correct and narrowly scoped to the gpu_usage_percent summary field that maps to Cloud Monitoring's DOUBLE descriptor (openfilter_gpu_usage_percent). Per-device gpuN series and the legitimately-INT64 flags are correctly left untouched. VERSION bump, RELEASE.md format, and the new test all check out. LGTM.

@shingonoide shingonoide merged commit c95c0ac into main Jun 29, 2026
12 checks passed
@shingonoide shingonoide deleted the openfilter_gpu-usage-float-cast branch June 29, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants