Instrument DHT with metrics

[iand]

ETA: 2023-08-31

Currently go-libp2p-kad-dht exports the following metrics

• libp2p.io/dht/kad/received_messages - Total number of messages received per RPC
• libp2p.io/dht/kad/received_message_errors - Total number of errors for messages received per RPC
• libp2p.io/dht/kad/received_bytes - Total received bytes per RPC
• libp2p.io/dht/kad/inbound_request_latency - Latency per RPC
• libp2p.io/dht/kad/outbound_request_latency - Latency per RPC
• libp2p.io/dht/kad/sent_messages - Total number of messages sent per RPC
• libp2p.io/dht/kad/sent_message_errors - Total number of errors for messages sent per RPC
• libp2p.io/dht/kad/sent_requests - Total number of requests sent per RPC
• libp2p.io/dht/kad/sent_request_errors - Total number of errors for requests sent per RPC
• libp2p.io/dht/kad/sent_bytes - Total sent bytes per RPC
• libp2p.io/dht/kad/network_size - Network size estimation

We should replicate these or provide equivalents.

See also libp2p/go-libp2p-kad-dht#304 for more discussion

[iand]

See also this PR libp2p/go-libp2p-kbucket#37 which adds the following:

• libp2p.io/dht/k-bucket/utilization - Number of entries per k-bucket
• libp2p.io/dht/k-bucket/peer_latency - Recorded latency measurements for all peers added to each bucket, per bucket.
• libp2p.io/dht/k-bucket/peers_added - Number of peers added to each k-bucket (cumulative). Note that peers can be counted twice if added and removed.
• libp2p.io/dht/k-bucket/peers_rejected_high_latency - Number of peers rejected from k-buckets due to high latency.
• libp2p.io/dht/k-bucket/peers_rejected_capacity - Number of peers rejected from the routing table because their k-bucket was full.
• libp2p.io/dht/k-bucket/peers_refreshed - Number of peers moved to the front of a k-bucket that they were already in.
• libp2p.io/dht/k-bucket/peers_removed - Number of peers removed from each k-bucket (cumulative).

[iand]

Also query metrics:

• Query hop count