Weird error at startup with Consul 1.9.1: error gathering metrics: 13 error(s) occurred

[pierresouchay]

Overview of the Issue

At startup, a weird message is issued when upgrading from 1.8.6 to 1.9.1:

agent: error gathering metrics: 13 error(s) occurred:
 * collected metric consul_version gauge:<value:nan >  has help "Represents the Consul version." but should have "consul_version"
 * collected metric consul_api_http summary:<sample_count:9 sample_sum:nan quantile:<quantile:0.5 value:nan > quantile:<quantile:0.9 value:nan > quantile:<quantile:0.99 value:nan > >  has help "Samples how long it takes to service the given HTTP request for the given verb and path." but should have "consul_api_http"
 * collected metric consul_client_api_catalog_service_nodes counter:<value:0 >  has help "Increments whenever a Consul agent receives a request to list nodes offering a service." but should have "consul_client_api_catalog_service_nodes"
 * collected metric consul_client_api_success_catalog_datacenters counter:<value:0 >  has help "Increments whenever a Consul agent successfully responds to a request to list datacenters." but should have "consul_client_api_success_catalog_datacenters"
 * collected metric consul_client_api_catalog_datacenters counter:<value:0 >  has help "Increments whenever a Consul agent receives a request to list datacenters in the catalog." but should have "consul_client_api_catalog_datacenters"
 * collected metric consul_consul_cache_fetch_success counter:<value:0 >  has help "Counts the number of successful fetches by the cache." but should have "consul_consul_cache_fetch_success"
 * collected metric consul_client_api_success_catalog_nodes label:<name:"node" value:"CZ3711L0SH" > counter:<value:40 >  has help "consul_client_api_success_catalog_nodes" but should have "Increments whenever a Consul agent successfully responds to a request to list nodes."
 * collected metric consul_client_api_catalog_services counter:<value:0 >  has help "Increments whenever a Consul agent receives a request to list services from the catalog." but should have "consul_client_api_catalog_services"
 * collected metric consul_client_api_success_catalog_services counter:<value:0 >  has help "Increments whenever a Consul agent successfully responds to a request to list services." but should have "consul_client_api_success_catalog_services"
 * collected metric consul_client_api_success_catalog_node_services counter:<value:0 >  has help "Increments whenever a Consul agent successfully responds to a request to list services in a node." but should have "consul_client_api_success_catalog_node_services"
 * collected metric consul_client_api_catalog_nodes counter:<value:0 >  has help "Increments whenever a Consul agent receives a request to list nodes from the catalog." but should have "consul_client_api_catalog_nodes"
 * collected metric consul_client_api_success_catalog_service_nodes label:<name:"node" value:"CZ3711L0SH" > counter:<value:1085 >  has help "consul_client_api_success_catalog_service_nodes" but should have "Increments whenever a Consul agent successfully responds to a request to list nodes offering a service."
 * collected metric consul_client_api_catalog_node_services label:<name:"node" value:"CZ3711L0SH" > counter:<value:3 >  has help "consul_client_api_catalog_node_services" but should have "Increments whenever a Consul agent successfully responds to a request to list nodes offering a service."

This also happens in dev mode on master:

error gathering metrics: 4 error(s) occurred:
* collected metric consul_fsm_ca summary:<sample_count:1 sample_sum:nan quantile:<quantile:0.5 value:nan > quantile:<quantile:0.9 value:nan > quantile:<quantile:0.99 value:nan > >  has help "Measures the time it takes to apply CA configuration operations to the FSM." but should have "consul_fsm_ca"
* collected metric consul_consul_fsm_intention summary:<sample_count:1 sample_sum:nan quantile:<quantile:0.5 value:nan > quantile:<quantile:0.9 value:nan > quantile:<quantile:0.99 value:nan > >  has help "Deprecated - use fsm_intention instead" but should have "consul_consul_fsm_intention"
* collected metric consul_fsm_intention label:<name:"op" value:"delete-all" > summary:<sample_count:1 sample_sum:0.04840100184082985 quantile:<quantile:0.5 value:nan > quantile:<quantile:0.9 value:nan > quantile:<quantile:0.99 value:nan > >  has help "consul_fsm_intention" but should have "Measures the time it takes to apply an intention operation to the FSM."
* collected metric consul_consul_fsm_ca summary:<sample_count:1 sample_sum:nan quantile:<quantile:0.5 value:nan > quantile:<quantile:0.9 value:nan > quantile:<quantile:0.99 value:nan > >  has help "Deprecated - use fsm_ca instead" but should have "consul_consul_fsm_ca"
    2020-12-28T18:01:03.363+0100 [DEBUG] agent.http: Request finished: method=GET url=/v1/agent/metrics?format=prometheus from=127.0.0.1:63455 latency=15.124411ms

Reproduction Steps

Steps to reproduce this issue, eg:

1. Create an agent with telemetry{prometheus_retention_time = "192h"}
2. Fetch metrics http://localhost:8500/v1/agent/metrics?format=prometheus
3. See error in logs

[jardleex]

@mkcp has created issue #9303 related to this error.
We're still on 1.8.3 and I'm hesitating to update to 1.9.0/1 due to this issues.

[pierresouchay]

@jardleex Thank you for the report, closing this issue as #9303 did exist before, let's continue the talk on #9303