move metrics to a common interface/implementation

[srivastavankit]

allow reuse for http request / client module.

[stevededalus]

Need a copyright header here

[Raynos]

Honestly, I don't like this version of the code. I'd rather not add a lock when the existing code does not have a lock.

[srivastavankit]

@Raynos The existing version of the code pre-allocates all possible error codes as a metric. For an HTTP Client, this means we end up creating this map for each method on the client. Seems super expensive to do this much pre-allocation and premature optimization as well.

If we need to support lazy initialization of the map, we need to use mutex make sure multiple threads dont write to the map.

[Raynos]

@srivastavankit

~800 counters takes up 70KB.
Adding another ~75,000 counters takes up 40MB.

Using more memory to avoid per-request locks seems reasonable to me. That being said we should benchmark the performance difference and look at flamegraphs.

[Raynos]

Looking at benchmarks and flamegraphs

• https://people.uberinternal.com/~jakev/zanzibar-lock-metrics-as.metrics.svg
• https://people.uberinternal.com/~jakev/zanzibar-lock-metrics-master.svg

The lock does not impact performance at all but the extra sprintf causes a regression from 1.3% in Flush() -> 2% in Flush().

Since it's per-request critical path I would favor the more performant version.

[Raynos]

PR is stale. Will close for now. Feel free to re-open if still wanted.