Migrated IO metrics from Counter to ObservableCounter instrument (DB-85)

[hunter1703]

Changed : IO metrics using Counter instrument to ObservableCounter instrument

[linear]

DB-85 Convert eventstore-io counters to use ObservableCounter

There are currently two Counters in MetricsBootstrapper that we call Add on frequently (whenever we read or write events)

		var byteMetric = coreMeter.CreateCounter<long>("eventstore-io", unit: "bytes");
		var eventMetric = coreMeter.CreateCounter<long>("eventstore-io", unit: "events");

We want these to be ObservableCounters that work similarly to StatusMetric. We will probably want a new class (CounterMetric?) that creates the ObservableCounter just like StatusMetric does.

The trackers (TFChunkTracker and IndexTracker) will write to the CounterSubMetrics as they do currently, but the CounterSubMetrics will Interlocked.Increment a value that is then Observed by the CounterMetric that they are registered with.

The purpose of the trackers (TFChunkTracker and IndexTracker) is to provide a trivial API to encapsulate the operation of the metrics away from the user of the tracker.

The purpose of the CounterSubMetric is to contain the data and the tags to be observed when the metric is observed (so that it doesn't have to be repeated in each tracker)

The purpose of the CounterMetric is to create the ObservableCounter such that it observes each submetric registered to it.

The effect here is that when we read/write events we are only making Interlocked.Increment calls which, it turns out, are ~10x cheaper than calling Add on a counter with tags. The observation is likely more expensive, but we only incur this cost once per scrape (typically once every 15s)

[timothycoleman]

looks good 👍 can be squashed

[timothycoleman]

👍