Add experimental configuration option to allow disabling legacy Prometheus metric names.

[reivilibre]

This is a WIP and might be the first step of #11106.

1. Use prometheus-client's exporter, not our own vendored one

Simply by changing over to prometheus-client's exporter, rather than our vendored one, the following metrics are changed:

Details

--- fm_leg_pc.txt2022-08-16 16:54:53.307316377 +0100
+++ fm_noleg_nocre.txt2022-08-16 16:54:57.379336740 +0100
-process_cpu_seconds
-python_gc_collections
-python_gc_objects_collected
-python_gc_objects_uncollectable
-synapse_admin_mau_current
-synapse_admin_mau_max
-synapse_admin_mau_registered_reserved_users
-synapse_background_process_db_sched_duration_seconds
-synapse_background_process_db_txn_count
-synapse_background_process_db_txn_duration_seconds
-synapse_background_process_ru_stime_seconds
-synapse_background_process_ru_utime_seconds
-synapse_background_process_start_count
-synapse_federation_client_events_processed
-synapse_federation_client_sent_edus
-synapse_federation_client_sent_pdu_destinations:count
-synapse_federation_client_sent_pdu_destinations_count_total
-synapse_federation_client_sent_pdu_destinations:total
-synapse_federation_client_sent_pdu_destinations_total
+synapse_federation_client_sent_pdu_destinations:count_total
-synapse_federation_client_sent_transactions
+synapse_federation_client_sent_pdu_destinations:total_total
-synapse_federation_server_number_inbound_pdu_pruned
-synapse_federation_server_received_edus
-synapse_federation_server_received_pdus
-synapse_federation_soft_failed_events
-synapse_handler_presence_bump_active_time
-synapse_handler_presence_federation_presence
-synapse_handler_presence_federation_presence_out
-synapse_handler_presence_notified_presence
-synapse_handler_presence_presence_updates
-synapse_handler_presence_timers_fired
-synapse_handlers_appservice_events_processed
-synapse_http_httppusher_badge_updates_failed
-synapse_http_httppusher_badge_updates_processed
-synapse_http_httppusher_http_pushes_failed
-synapse_http_httppusher_http_pushes_processed
-synapse_notifier_notified_events
-synapse_push_bulk_push_rule_evaluator_push_rules_invalidation_counter
-synapse_push_bulk_push_rule_evaluator_push_rules_state_size_counter
-synapse_replication_tcp_resource_federation_ack
-synapse_replication_tcp_resource_remove_pusher
-synapse_replication_tcp_resource_user_ip_cache
-synapse_replication_tcp_resource_user_sync
-synapse_state_res_cpu_for_biggest_room_seconds
-synapse_state_res_db_for_biggest_room_seconds
-synapse_storage_events_persisted_events
-synapse_storage_events_potential_times_prune_extremities
-synapse_storage_events_state_delta
-synapse_storage_events_state_delta_reuse_delta
-synapse_storage_events_state_delta_single_event
-synapse_storage_events_state_resolutions_during_persistence
-synapse_storage_events_times_pruned_extremities
-synapse_storage_transaction_time_count
-synapse_storage_transaction_time_sum
-synapse_user_logins
-synapse_user_registrations
-synapse_util_caches_cache
-synapse_util_caches_cache:evicted_size
-synapse_util_caches_cache_evicted_size
-synapse_util_caches_cache:hits
-synapse_util_caches_cache_hits
-synapse_util_caches_cache_max_size
-synapse_util_caches_cache_pending
-synapse_util_caches_cache:size
-synapse_util_caches_cache_size
-synapse_util_caches_cache:total
-synapse_util_caches_response_cache
-synapse_util_caches_response_cache:evicted_size
-synapse_util_caches_response_cache_evicted_size
-synapse_util_caches_response_cache:hits
-synapse_util_caches_response_cache_hits
-synapse_util_caches_response_cache:size
-synapse_util_caches_response_cache_size
-synapse_util_caches_response_cache:total
-synapse_util_metrics_block_count
-synapse_util_metrics_block_db_sched_duration_seconds
-synapse_util_metrics_block_db_txn_count
-synapse_util_metrics_block_db_txn_duration_seconds
-synapse_util_metrics_block_ru_stime_seconds
-synapse_util_metrics_block_ru_utime_seconds
-synapse_util_metrics_block_time_seconds

(N.B. I don't think the removal of both :hits and _hits is what we want — I suspect this is cutting them out because we need to patch up metrics to not contain colons ourselves.)

(N.B. +synapse_federation_client_sent_pdu_destinations:count_total and +synapse_federation_client_sent_pdu_destinations:total_total seem 'undesirable' to say the least :-)).

2. Massage metrics with colons in their names so that we return the underscored variant in the modern exporter and both in the legacy exporter.

We do this by renaming all colon-containing metrics to use underscores instead, but then patch them up in the legacy exporter so they appear to have the same name.

I have verified that this means that, in the legacy configuration, we have the same metrics as before this change.

3. Removing _created metrics from the prometheus-client exporter.

To do this normally, you have to set an environment variable and you need version ≥ 0.14.0, but I don't want to require users to set one themselves.

Instead, I poke the private flag in prometheus-client if it exists. I realise this is brittle so I ensure that the world won't end if it's not there, plus I add a test so that we'll notice.
I'm not super keen but I don't have any grand ideas either.

...

This PR should be commit-by-commit reviewable.

[clokep]

(I didn't review this, just commented on the phrasing of the PR.)

[DMRobertson]

A few thoughts. No major objections though. WDYT?

[DMRobertson]

SGTM!