Prometheus metrics improvements

[myzie]

• Add username label to deconflict metrics that would otherwise have duplicate labels across different pools.
• Group metrics by name and only print HELP and TYPE once per metric name.
• Sort labels for a deterministic output.

This PR fixes #563

Without this change, having multiple users results in Prometheus processing errors downstream along the lines of:

second HELP line for metric name "pgcat_stats_total_xact_count"

Prometheus Reference

https://prometheus.io/docs/instrumenting/exposition_formats/#grouping-and-sorting

Grouping and sorting

All lines for a given metric must be provided as one single group, with the optional HELP and TYPE lines first (in no particular order). Beyond that, reproducible sorting in repeated expositions is preferred but not required, i.e. do not sort if the computational cost is prohibitive.

Each line must have a unique combination of a metric name and labels. Otherwise, the ingestion behavior is undefined.

Example setup

In the example below, I tested with an extra user defined in examples/docker/pgcat.toml, e.g.

[pools.postgres.users.0]
username = "postgres"
password = "postgres"
pool_size = 9

[pools.postgres.users.1]
username = "curtis"
password = "abcd1234"
pool_size = 9

Example Output: Before

Using pgcat_stats_total_xact_count as an example, to cull down the overall amount of text I'm pasting here.

$ cat metrics_before_fix.txt | grep pgcat_stats_total_xact_count

# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{role="primary",database="postgres",shard="0",host="postgres",pool="postgres"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{role="replica",host="postgres",database="postgres",pool="postgres",shard="0"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{shard="1",host="postgres",role="primary",database="postgres",pool="postgres"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{role="replica",pool="postgres",shard="1",database="postgres",host="postgres"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{shard="2",role="primary",host="postgres",pool="postgres",database="postgres"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{host="postgres",shard="2",pool="postgres",database="postgres",role="replica"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{host="postgres",role="primary",database="postgres",pool="postgres",shard="0"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{host="postgres",role="replica",database="postgres",pool="postgres",shard="0"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{host="postgres",shard="1",role="primary",database="postgres",pool="postgres"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{role="replica",database="postgres",shard="1",pool="postgres",host="postgres"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{database="postgres",shard="2",role="primary",host="postgres",pool="postgres"} 0
# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{database="postgres",pool="postgres",host="postgres",shard="2",role="replica"} 0

Example Output: After

$ cat metrics_after_fix.txt | grep pgcat_stats_total_xact_count

# HELP pgcat_stats_total_xact_count Total number of transactions started by the client
# TYPE pgcat_stats_total_xact_count counter
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="primary",shard="0",username="curtis"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="replica",shard="0",username="curtis"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="primary",shard="1",username="curtis"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="replica",shard="1",username="curtis"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="primary",shard="2",username="curtis"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="replica",shard="2",username="curtis"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="primary",shard="0",username="postgres"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="replica",shard="0",username="postgres"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="primary",shard="1",username="postgres"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="replica",shard="1",username="postgres"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="primary",shard="2",username="postgres"} 0
pgcat_stats_total_xact_count{database="postgres",host="postgres",pool="postgres",role="replica",shard="2",username="postgres"} 0

[drdrsh]

Changes were merged in #795