fix(pressure_reconciler): emit Reconciliation metric event#754
Merged
Conversation
cbartz
force-pushed
the
fix/pressure-reconciler-missing-reconciliation-metric
branch
2 times, most recently
from
ca738ef to
ad8f67e
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the planner-driven PressureReconciler to emit the Reconciliation metric event and update reconciliation-related Prometheus gauges, aligning the planner-based reconcile path with the legacy scaler behavior so dashboards don’t go empty.
Changes:
- Add reconciliation metric emission + Prometheus gauge updates to
PressureReconciler._handle_timer_reconcile. - Add/adjust unit test stubs and a new unit test to assert the reconciliation event is emitted with expected runner counts.
- Document the user-visible metrics fix in
docs/changelog.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| github-runner-manager/src/github_runner_manager/manager/pressure_reconciler.py | Adds reconciliation metric/gauge emission during timer reconcile. |
| github-runner-manager/tests/unit/manager/test_pressure_reconciler.py | Extends test scaffolding and adds a unit test for the reconciliation event. |
| docs/changelog.md | Notes the metrics behavior change in the user-facing changelog. |
github-runner-manager/src/github_runner_manager/manager/pressure_reconciler.py
Show resolved
Hide resolved
github-runner-manager/src/github_runner_manager/manager/pressure_reconciler.py
Outdated
Show resolved
Hide resolved
github-runner-manager/src/github_runner_manager/manager/pressure_reconciler.py
Show resolved
Hide resolved
PressureReconciler._handle_timer_reconcile never issued the Reconciliation metric event or updated Prometheus gauges (idle/busy runners count, reconcile duration). This caused the reconciliation metric to be missing when using the planner-driven code path. Add _issue_reconciliation_metric to PressureReconciler that emits the Reconciliation event and updates Prometheus gauges, matching the behavior of RunnerScaler.reconcile.
cbartz
force-pushed
the
fix/pressure-reconciler-missing-reconciliation-metric
branch
from
ad8f67e to
cdf4733
Compare
Use cleanup() metric stats to compute crashed_runners instead of hardcoding zero. Scale-down stats from soft_delete_runners are not included (it returns only an int), documented as an accepted trade-off.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes missing reconciliation observability in the planner-driven PressureReconciler path by emitting the Reconciliation metric event and updating the same Prometheus reconcile gauges used by RunnerScaler.reconcile, restoring expected dashboard signals.
Changes:
- Add
_issue_reconciliation_metric()and call it fromPressureReconciler._handle_timer_reconcile()to emit theReconciliationevent and update reconcile gauges/histogram. - Extend unit test stubs to model runner state/health and add a test asserting the
Reconciliationevent is issued. - Document the user-facing metrics fix in
docs/changelog.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| github-runner-manager/src/github_runner_manager/manager/pressure_reconciler.py | Emit Reconciliation metric event in timer reconcile and update reconcile Prometheus gauges/histogram. |
| github-runner-manager/tests/unit/manager/test_pressure_reconciler.py | Add runner/manager stubs needed for metric counting and a unit test for event emission. |
| docs/changelog.md | Add changelog entry for restored reconciliation metrics in pressure reconciler. |
github-runner-manager/src/github_runner_manager/manager/pressure_reconciler.py
Show resolved
Hide resolved
cbartz
force-pushed
the
fix/pressure-reconciler-missing-reconciliation-metric
branch
from
75830d2 to
95d7ef0
Compare
cbartz
requested review from
florentianayuwono,
javierdelapuente,
weiiwang01,
yanksyoon and
yhaliaw
as code owners
cbartz
changed the title
cbartz
changed the title
cbartz
deleted the
fix/pressure-reconciler-missing-reconciliation-metric
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Issue the reconciliation metric (and related prometheus metrics) in the pressure reconciler.
Rationale
The panels in the dashboards
metrics.jsonandmetrics_prometheus.jsonthat are based on the reconciliation metrics are currently non-functional when the pressure_reconciler mode is used.Checklist
urgent,trivial,complex).github-runner-manager/pyproject.toml.