Failed to collect metric for exporter: nova

[qq346434990]

CentOS8 - Openstack yoga - nova

I have encountered a problem that is difficult to solve, I asked for help:
openstack-exporter: 1.6.0
problem: ERRO[0021] Failed to collect metric for exporter: nova, error: failed to collect metric: running_vms, error: CPUInfo has unexpected type: source="exporter.go:122"

[srkaviani]

I have this problem too. (msg="Failed to collect metric for exporter: nova, error: failed to collect metric: running_vms, error: CPUInfo has unexpected type: " source="exporter.go:123") after update to wallaby.

[Oxyash]

same problem on Zed
level=error msg="Failed to collect metric for exporter: nova, error: failed to collect metric: running_vms, error: CPUInfo has unexpected type: " source="exporter .go:123"

[adamoswick]

This seems to be a result of changes in the Nova API after version 2.87. The gophercloud code tries to fetch the cpu_info output from the os-hypervisor/detail API route. However, this doesn't exist anymore so the type is None causing an error from the SDK.

This causes the SDK to error and ultimately the Nova collector in openstack-exporter also errors meaning the metrics here fail. We've worked around this by setting prometheus_openstack_exporter_compute_api_version = 2.87 in Kolla Ansible (which sets OS_COMPUTE_API_VERSION=2.87 in the environment for openstack-exporter).

The 2.87 version of the Nova API still contains the cpu_info output meaning the error doesn't get thrown. Ideally, the Nova API changes following 2.87 need to be supported by the gophercloud SDK.

[srkaviani]

@adamoswick
Thanks. This was very helpful.

[samuelallan72]

On latest main I get this similar error output:

ts=2024-04-03T23:09:00.715Z caller=exporter.go:126 level=error err="Failed to collect metric for exporter" exporter=nova error="failed to collect metric: running_vms, error: CPUInfo has unexpected type: <nil>"

Confirming that setting OS_COMPUTE_API_VERSION=2.87 in the environment fixes the issue.

---

Also note that when this error happens, it also causes metrics to not be written for all the other metrics in the ListHypervisors function:

openstack-exporter/exporters/nova.go

Lines 200 to 225 in f1a2af1

	ch <- prometheus.MustNewConstMetric(exporter.Metrics["running_vms"].Metric,
	prometheus.GaugeValue, float64(hypervisor.RunningVMs), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))
	ch <- prometheus.MustNewConstMetric(exporter.Metrics["current_workload"].Metric,
	prometheus.GaugeValue, float64(hypervisor.CurrentWorkload), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))
	ch <- prometheus.MustNewConstMetric(exporter.Metrics["vcpus_available"].Metric,
	prometheus.GaugeValue, float64(hypervisor.VCPUs), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))
	ch <- prometheus.MustNewConstMetric(exporter.Metrics["vcpus_used"].Metric,
	prometheus.GaugeValue, float64(hypervisor.VCPUsUsed), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))
	ch <- prometheus.MustNewConstMetric(exporter.Metrics["memory_available_bytes"].Metric,
	prometheus.GaugeValue, float64(hypervisor.MemoryMB*MEGABYTE), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))
	ch <- prometheus.MustNewConstMetric(exporter.Metrics["memory_used_bytes"].Metric,
	prometheus.GaugeValue, float64(hypervisor.MemoryMBUsed*MEGABYTE), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))
	ch <- prometheus.MustNewConstMetric(exporter.Metrics["local_storage_available_bytes"].Metric,
	prometheus.GaugeValue, float64(hypervisor.LocalGB*GIGABYTE), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))
	ch <- prometheus.MustNewConstMetric(exporter.Metrics["local_storage_used_bytes"].Metric,
	prometheus.GaugeValue, float64(hypervisor.LocalGBUsed*GIGABYTE), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))
	ch <- prometheus.MustNewConstMetric(exporter.Metrics["free_disk_bytes"].Metric,
	prometheus.GaugeValue, float64(hypervisor.FreeDiskGB*GIGABYTE), hypervisor.HypervisorHostname, availabilityZone, aggregatesLabel(hypervisor.Service.Host, hostToAggrMap))