Use peer id as metric label

[coignetp]

Hey!
I'm currently working on the consul integration at Datadog, and we want to scrape the prometheus endpoint to get consul metrics. In the current state several metrics are not consistently named, and contain parts that would be good as labels instead, and that makes scraping the endpoint and converting to Datadog metrics troublesome.
This PR use the peer ids as label and remove them from the metric names.

Example: consul_raft_replication_appendEntries_logs_684ad51c_6143_f2ea_aaaa_8ab6cc5ccccc -> consul_raft_replication_appendEntries_logs.
This PR affects:

• consul_raft_replication_installSnapshot
• consul_raft_replication_heartbeat
• consul_raft_replication_appendEntries_rpc
• consul_raft_replication_appendEntries_logs

This change is not backward compatible since the metric names will be different. The current metric parsing method won't work anymore.

Please tell me what you think about this change! 😄

P.S. : I'd like to do the same thing for other consul metrics (here for example https://github.com/hashicorp/consul/blob/5e5dbedd47a5f875b60e241c5555a9caab595246/agent/http.go#L258)

[hashicorp-cla]

All committers have signed the CLA.

[coignetp]

Is the CI failing because of my change? I'm not sure I understand the error here

[schristoff]

Howdy - this is great! Thank you so much.
Is there any way we could maintain the original metrics and add the new metrics as well to maintain backwards compatibility?

[coignetp]

I updated the PR to keep sending both metrics

[coignetp]

@s-christoff Should I do something more on this change? I'm not sure if the CI is failing because of this PR here🙂

[schristoff]

Howdy friend,
Thank you for your patience and hard work on this. It's started some internal discussions - which is great! We're (Consul) looking into deprecating other metadata-suffixed metrics in Consul and we are working through the details. We're going to figure some stuff out on our end, and will be looking to accept this soon.

[coignetp]

Hi @s-christoff, is there any news about this?

[coignetp]

Hi @s-christoff, sorry to bother you again, but how do you have any news here?

[schristoff]

lgtm - sorry for the delay, lots of internal conversations happening. :p