feat(ironic): Add node state metrics from versioned notifications by nidzrai · Pull Request #1963 · rackerlabs/understack

nidzrai · 2026-04-22T12:44:10Z

Added power_state, provision_state, maintenance, and fault metrics by consuming
Ironic versioned notifications from a dedicated queue.
Added second consumer for ironic-hardware-exporter-states queue bound to
ironic_versioned_notifications.info routing key
Add parser for baremetal.node.power_set/provision_set events

log :

2026/04/22 18:26:54 HTTP server listening on :9608
2026/04/22 18:26:55 waiting for messages from queue: ironic-hardware-exporter-states
2026/04/22 18:26:55 waiting for messages from queue: ironic-hardware-exporter
2026/04/22 18:26:56 cached sensors node=Dell-93GSW04
2026/04/22 18:26:56 cached sensors node=Dell-F3GSW04
2026/04/22 18:26:56 cached sensors node=Dell-J3GSW04
2026/04/22 18:26:56 cached sensors node=Dell-24GSW04
2026/04/22 18:26:56 cached sensors node=Dell-53GSW04
2026/04/22 18:26:56 cached sensors node=Dell-B3GSW04
2026/04/22 18:26:56 cached sensors node=Dell-JBTSW04
2026/04/22 18:26:56 cached sensors node=Dell-D3GSW04
2026/04/22 18:27:02 GET /metrics — served 22 nodes
2026/04/22 21:42:11 cached state node=Dell-24GSW04 power=0x239aed3481d0 provision=0x239aed3481e0

❯ openstack baremetal node power off Dell-24GSW04
from /metrics
ironic_node_power_state{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",power_state="power off"} 1 ironic_node_provision_state{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",provision_state="available"} 1 ironic_node_maintenance{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3"} 0 ironic_node_fault{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",fault="none"} 0

❯ openstack baremetal node power on Dell-24GSW04
2026/04/22 21:44:28 cached state node=Dell-24GSW04 power=0x239aed2ab400 provision=0x239aed2ab410
ironic_node_power_state{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",power_state="power on"} 1 ironic_node_provision_state{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",provision_state="available"} 1 ironic_node_maintenance{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3"} 0 ironic_node_fault{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",fault="none"} 0

missing : conductor_host label on sensor metrics coz it does not exist with sensor data
WIP - deployment artifact and TLS

helm chart follows nautobotop go project

ctria · 2026-04-24T17:07:04Z

in that metric:

ironic_node_fault{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",fault="none"} 0

shouldn't it be 1 instead of 0? there are 1 entities with node_uuid=XXX, node_name=YYY, conductor_host="ZZZ" and fault "none".

My assumption here is that if that has a fault all the labels will remain same except "fault" which will now get the fault description. at THAT point fault="none" should drop to 0 (or disappear).

Do I read it wrongly?

nidzrai · 2026-04-25T20:16:05Z

in that metric:
ironic_node_fault{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",fault="none"} 0
shouldn't it be 1 instead of 0? there are 1 entities with node_uuid=XXX, node_name=YYY, conductor_host="ZZZ" and fault "none".

My assumption here is that if that has a fault all the labels will remain same except "fault" which will now get the fault description. at THAT point fault="none" should drop to 0 (or disappear).

Do I read it wrongly?

i treated fault more like a status/condition metric, where the numeric value indicates whether the node is faulted (0 = no, 1 = yes), rather than a pure state-label metric where the active label always has value 1.

nidzrai · 2026-04-25T20:20:15Z

in that metric:
ironic_node_fault{node_uuid="a8a8548c-fc07-4d9c-a5f2-5f2c6fe7992c",node_name="Dell-24GSW04",conductor_host="ironic-conductor.1327175-hp3",fault="none"} 0
shouldn't it be 1 instead of 0? there are 1 entities with node_uuid=XXX, node_name=YYY, conductor_host="ZZZ" and fault "none".

My assumption here is that if that has a fault all the labels will remain same except "fault" which will now get the fault description. at THAT point fault="none" should drop to 0 (or disappear).

Do I read it wrongly?

so I modeled ironic_node_fault as a boolean/status-style metric rather than a pure state-label metric.
The thought process was that fault is closer to a condition like “is the node faulted right now?” than to a normal lifecycle state like power_state or provision_state. With the current shape

ironic_node_fault{...,fault="none"} 0 the node is not faulted
ironic_node_fault{...,fault=""} 1 the node is faulted, and the label carries the reason

The main reason to chose that shape is query behavior with this model, expressions like sum(ironic_node_fault) directly answer “how many nodes are faulted right now?”. If fault="none" were emitted as 1, that kind of query would count healthy nodes as well, idk but this felt misleading to me.
There also isn’t a direct upstream fault metric in ironic-prometheus-exporter to mirror exactly, but the closest upstream precedent for status/health-style metrics is to encode the status in the metric value rather than using an always-1 state-label pattern.
That said, I agree your reading is reasonable, especially since power_state / provision_state in this exporter do use the value always 1, state in label model.

So if we want fault to follow the same pattern for consistency, I’m happy to change it. My choice here was mainly to avoid the “healthy nodes count as 1” downside of the alternative.

cardoe · 2026-05-04T14:22:54Z

Can we add some documentation to docs about the design of this code, how to build it and how to validate it. We should also have under the deployment docs some information on how to deploy it.

cardoe · 2026-05-04T14:25:51Z

+			store.UpdateNodeState(stateMsg)
+			log.Printf("cached state node=%s power=%v provision=%v",
+				stateMsg.NodeName, stateMsg.PowerState, stateMsg.ProvisionState)
+		}); err != nil {


Blocking issue: if the states consumer exits here, the process keeps running and /health remains 200. bothReady will flip /ready to 503, but Kubernetes will not restart the pod from readiness alone and the exporter will continue serving stale state metrics. The sensor consumer path exits fatally on failure; the states path should do the same or share a supervisor/reconnect loop so state collection recovers.

cardoe · 2026-05-04T14:25:58Z

+
+// nodeStateEventPrefixes are the baremetal.node events .
+// we use .end and .success variants so we  capture final state
+var nodeStateEventPrefixes = []string{


Because this parser also drives ironic_node_maintenance and ironic_node_fault, limiting the allow-list to power/provision events leaves those metrics stale when maintenance or fault changes through maintenance/update notifications. It also misses power_state_corrected, which the existing Nautobot sync path treats as a power-state event. Please include all Ironic node events that can change the fields exported here, or do not export fields this consumer cannot keep current.

will do these in follow up pr

cardoe · 2026-05-04T14:26:06Z

+	fmt.Fprintf(b, "# TYPE ironic_node_last_seen_timestamp_seconds gauge\n")
 	for _, n := range nodes {
-		fmt.Fprintf(&b, "ironic_node_last_seen_timestamp_seconds{node_uuid=%q,node_name=%q} %d\n",
+		fmt.Fprintf(b, "ironic_node_last_seen_timestamp_seconds{node_uuid=%q,node_name=%q} %d\n",


State events can create a cache entry before any hardware metrics arrive because UpdateNodeState does that. For those entries LastSeen is the Go zero time, so this loop emits ironic_node_last_seen_timestamp_seconds with -62135596800, which is a bogus hardware metrics timestamp. Skip zero LastSeen values or track a separate state timestamp instead.

will do this in follow up pr

cardoe · 2026-05-04T14:26:12Z

+		if n.PowerState == nil {
+			continue
+		}
+		fmt.Fprintf(b, "ironic_node_power_state{node_uuid=%q,node_name=%q,conductor_host=%q,power_state=%q} 1\n",


This current-state-only shape leaves the previous label series active in Prometheus until staleness kicks in. For example, after power off changes to power on, the old power_state="power off" sample can still match queries for several minutes. State metrics are safer if the exporter emits the complete enum as 0/1 for each node, or uses a numeric gauge without state labels.

will do these in follow up pr

cardoe · 2026-05-04T14:31:13Z

Suggestion: switch the exporter release tag namespace to ironic-hardware-exporter/vX.Y.Z and make the CI/release path target that format consistently.

Concrete shape I would suggest:

Change the workflow tag trigger from ironic-hardware-exporter-v* to ironic-hardware-exporter/v*.
Extract the release version from GITHUB_REF_NAME with ironic-hardware-exporter/v stripped, so image and chart versions remain X.Y.Z.
Validate the ref early so manual dispatches or accidental tags fail unless they use refs/tags/ironic-hardware-exporter/vX.Y.Z.
Rename the GoReleaser env var from CUSTOM_TAG to something like RELEASE_VERSION, since the value passed into image tags and OCI labels is the cleaned version, not the full git tag.
Consider setting a GoReleaser release.name_template such as ironic-hardware-exporter v{{ .Env.RELEASE_VERSION }} so the GitHub release title stays clean while the git tag remains namespaced.

This aligns the exporter with the component-style tag format already used elsewhere, for example ironic-ipxe/v*, while keeping published image tags and Helm chart versions as plain semver (X.Y.Z).

skrobul

lgtm with minor suggestions

skrobul · 2026-05-05T16:51:47Z

 	}
 }

+// todo: circle back here for parsing


leftover todo

skrobul · 2026-05-05T17:01:54Z

 	for _, n := range nodes {
 		for key, d := range n.Sensors.Drive {
 			val := 0.0
 			if d.State != nil && *d.State == "Enabled" {


Suggested change

if d.State != nil && *d.State == "Enabled" {

if d.State != nil && strings.EqualFold(*d.State, "Enabled") {

skrobul · 2026-05-05T17:19:37Z

+		if n.PowerState == nil && n.ProvisionState == nil {
+			continue
+		}


what is this check for? does the maintenance status change always arrive only with power info?
I feel like there is some story behind this, but couldn't figure out what it is.

The old check was replaced by HasStateData (set once on the first UpdateNodeState() call). The old PowerState == nil && ProvisionState == nil check would have silently dropped maintenance-only events maintenance.set arrives with both fields nil since Ironic only sends what changed. HasStateData correctly gates on "has any state event arrived for this node" rather than "does this node have both specific fields populated".

skrobul · 2026-05-05T18:26:49Z

+		if n.PowerState == nil && n.ProvisionState == nil {
+			continue
+		}


similar as above - why is this one gated?

was guarding maintenance and fault emission, but maintenance.set events (which only set maintenance=true) arrive with both power_state and provision_state nil. but replaced it with this was also replaced by HasStateData

Both checks are gone. transformer.go now has a single gate at the top of the stateFamilies loop: if !n.HasStateData { continue }. HasStateData is set by the first UpdateNodeState() call regardless of which fields the event carried, so maintenance-only events (nil power/provision state) too stays

nidzrai requested review from cardoe and skrobul April 22, 2026 13:23

nidzrai force-pushed the feat/ironic-hardware-exporter-states branch from 6d790ad to ed18e1b Compare April 24, 2026 15:05

nidzrai requested a review from a team April 27, 2026 16:04

nidzrai force-pushed the feat/ironic-hardware-exporter-states branch from ed18e1b to 22db994 Compare April 28, 2026 10:46

cardoe reviewed May 4, 2026

View reviewed changes

nidzrai added 8 commits May 5, 2026 22:48

add node state metrics from versioned notifications

68df14d

Fixed typos

df7e8c2

Lint fix

393e2c4

add TLS, deployment, and helm chart

04ef955

Added notes and fixed blocker issue

d29edd9

deploy guide

d47af12

Lint isuue fixed for a constant

cc94dfe

Refactor ironic-hardware-exporter cache, server, and release flow

6b8ec90

skrobul approved these changes May 5, 2026

View reviewed changes

removed todo and updated readme

4f4110e

nidzrai force-pushed the feat/ironic-hardware-exporter-states branch from 9a7f3b7 to 4f4110e Compare May 5, 2026 23:35

nidzrai added 4 commits May 6, 2026 16:25

Moved main.go to cmd and moved formatter and transformer to metric

7ca7a8d

remove local/ from PR — dev-only smoke test helper

35dd843

fixed version lint issue

abd356a

lint update

3ab4a72

nidzrai added this pull request to the merge queue May 6, 2026

Merged via the queue into main with commit 95d6c06 May 6, 2026
24 checks passed

nidzrai deleted the feat/ironic-hardware-exporter-states branch May 6, 2026 12:23

	if d.State != nil && *d.State == "Enabled" {
	if d.State != nil && strings.EqualFold(*d.State, "Enabled") {

Conversation

nidzrai commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ctria commented Apr 24, 2026

Uh oh!

nidzrai commented Apr 25, 2026

Uh oh!

nidzrai commented Apr 25, 2026

Uh oh!

cardoe commented May 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cardoe commented May 4, 2026

Uh oh!

skrobul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nidzrai commented Apr 22, 2026 •

edited

Loading