Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exporter crashes when collecting compute metrics #37

Closed
uxeng opened this issue Oct 9, 2019 · 5 comments
Closed

Exporter crashes when collecting compute metrics #37

uxeng opened this issue Oct 9, 2019 · 5 comments
Assignees
@niedbalski
Labels
bug Something isn't working
Milestone
0.6.0

Comments

@uxeng
Copy link

uxeng commented Oct 9, 2019

Looks like openstack_nova_agent_state is a metric that contains metadata that does not change over time (e.g. hostname). exporter does not handle this metric properly?

After running the exporter for few seconds/minutes, this error is shown when trying to collect the metrics (e..g wget http:/xxx:port/metrics)

An error has occurred while serving metrics:

collected metric "openstack_nova_agent_state" { label:<name:"adminState" value:"enabled" > label:<name:"hostname" value:"XXXXXXX" > label:<name:"service" value:"nova-conductor" > label:<name:"zone" value:"internal" > counter:<value:1 > } was collected before with the same name and label values
@niedbalski
Copy link
Member

@uxeng which version are you using? can you include the full logs?

@uxeng
Copy link
Author

uxeng commented Oct 9, 2019

Same errors using both snap versions edge and stable.

$ openstack --version
openstack 4.0.0

I dont see any errors on the console:

.........
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching domains information source="keystone.go:50"
INFO[0017] Collecting metrics for exporter: openstack_cinder, metric: snapshots source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching snapshots information source="cinder.go:116"
INFO[0017] Collecting metrics for exporter: openstack_neutron, metric: security_groups source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching list of network security groups source="neutron.go:127"
INFO[0017] Collecting metrics for exporter: openstack_cinder, metric: agent_state source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching services state information source="cinder.go:136"
INFO[0017] Collecting metrics for exporter: openstack_identity, metric: users source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching users information source="keystone.go:112"
INFO[0017] Collecting metrics for exporter: openstack_cinder, metric: volume_status source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Collecting metrics for exporter: openstack_neutron, metric: subnets source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching list of subnets source="neutron.go:147"
INFO[0017] Collecting metrics for exporter: openstack_neutron, metric: ports source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching list of ports source="neutron.go:167"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: running_vms source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching list of hypervisors source="nova.go:118"
INFO[0017] Collecting metrics for exporter: openstack_neutron, metric: agent_state source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching agents list source="neutron.go:75"
INFO[0017] Collecting metrics for exporter: openstack_neutron, metric: floating_ips source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching floating ips list source="neutron.go:56"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: vcpus_available source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: memory_used_bytes source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: flavors source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching list of flavors source="nova.go:161"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: availability_zones source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching list of availability zones source="nova.go:182"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: current_workload source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: vcpus_used source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: local_storage_used_bytes source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: security_groups source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching list of nova security groups source="nova.go:202"
INFO[0017] Collecting metrics for exporter: openstack_nova, metric: agent_state source="exporter.go:69"
INFO[0017] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0017] Fetching list of services source="nova.go:92"
INFO[0018] Collecting metrics for exporter: openstack_nova, metric: memory_available_bytes source="exporter.go:69"
INFO[0018] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0018] Collecting metrics for exporter: openstack_nova, metric: local_storage_available_bytes source="exporter.go:69"
INFO[0018] Refreshing auth client in case token has expired source="exporter.go:85"
INFO[0018] Collecting metrics for exporter: openstack_nova, metric: server_status source="exporter.go:69"
INFO[0018] Refreshing auth client in case token has expired source="exporter.go:85"
.....

niedbalski added a commit that referenced this issue Oct 9, 2019
@niedbalski
Added id to the nova_agent_state metric.
3de9565 
Its possible to have multiple nova-compute agents of the same
service on the same host.

Relates to #37

Signed-off-by: Jorge Niedbalski <jnr@metaklass.org>
niedbalski added a commit that referenced this issue Oct 9, 2019
@niedbalski
Add id to nova_agent_state
ab872e1 
Its possible to have multiple agents of the same
service on the same host.

Relates to #37

Signed-off-by: Jorge Niedbalski <jnr@metaklass.org>
@niedbalski niedbalski mentioned this issue Oct 9, 2019
Merged
niedbalski added a commit that referenced this issue Oct 9, 2019
@niedbalski
Add id to nova_agent_state (#40)
1cf1439 
Its possible to have multiple agents of the same
service on the same host.

Relates to #37

Signed-off-by: Jorge Niedbalski <jnr@metaklass.org>
@niedbalski
Copy link
Member

PR #40 should have fixed this issue by appending the service.ID to the list of labels. Please test it and let me know if it works as expected. @uxeng

@niedbalski
Copy link
Member

Are you able to reproduce this further with the latest master @uxeng ?

@niedbalski niedbalski self-assigned this Oct 10, 2019
@niedbalski niedbalski added the bug Something isn't working label Oct 10, 2019
@niedbalski niedbalski added this to the 0.6.0 milestone Oct 10, 2019
@uxeng
Copy link
Author

uxeng commented Oct 10, 2019 •
edited

Hi,
The issue is fixed!!!
thank you very much :)
(tested with edge)

/U

@uxeng uxeng closed this as completed Oct 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@niedbalski niedbalski

Labels
bug Something isn't working
Projects
None yet
Milestone
0.6.0
Development

No branches or pull requests
2 participants
@niedbalski @uxeng