HDDS-3236. Fix Dropwizard metrics mapping for latest Ratis metrics.

[elek]

What changes were proposed in this pull request?

Ratis use dropwizard metrics where the key parameters of the metrics (like group name or instance id) are part of the name of the metrics instead of using a tag.

For example

ratis.log_appender.851cb00a-af97-455a-b079-d94a77d2a936@group-C14654DE8C2C.follower_65f881ea-8794-403d-be77-a030ed79c341_match_index 

Instead of

ratis.log_appender_match_index{group="group-C14654DE8C2C",...} 

It makes hard to combine the same metrics (match_index) from different sources.

HDDS-2950 implemented a regexp based workaround, but the regexp doesn't match for the latest Ratis metrics.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3236

How was this patch tested?

Tested with kubernetes cluster generation ~200M keys.

An easier way to test is to start a docker compose cluster and check the /prom endpoint of one datanode:

docker-compose exec datanoe bash
curl localhost:9882/prom | grep index

[avijayanhwx]

LGTM +1.

[bharatviswa504]

Hi @elek Still I don't see a few of the ratis metrics.

Like numRetryCacheHits, numRequestQueueLimitHits and leaderNumResourceLimitHits

[elek]

numRetryCacheHits, numRequestQueueLimitHits and leaderNumResourceLimitHits

These (*Hits) are implemented on Ratis side in a way that you can see them only if they have non-zero values.

[arp7]

+1