Drafting possible API to generalise instrumentation code for ZIO apps.

[atooni]

This addresses #109.
As far as I can see this will require some refactoring of the existing instrumentation which is tightly couple to statsd.
I am starting to draft out a common instrumentation API that can be used in any ZIO app. The idea is to be able to report metrics to either Prometheus or statsd simply by sticking another Layer into the application.

Creating the PR early to share and discuss early.

An example of a very simple instrumented App is in the test trait and the test app - statsd and test app - empty and also test-app prometheus

• Reorganise code to make reporting backend pluggable, also see Reorganize Package Structure To Clarify API #103
• Technical proof - Sample app with simple counter reporting to
  • Empty backend
  • Datadog
  • Prometheus
• Define/Review Metrictypes for ZMX
  • Counter
  • Gauge
• Specify mapping of ZMX Metrictypes to
  • Statsd / Datadog
  • Prometheus
• Implement specification
• Documentation
• Instrumentation examples

See https://github.com/blended-zio/zio-zmx/tree/109-prometheus/docs/metrics for documentation related to the PR.

[atooni]

Example reporting to DataDog

Example reporting toPrometheus / Grafana

[atooni]

After quick review with @adamgfraser we think that we would like to get this PR to a mergable state rather sooner than later. To achieve that we would like to implement a unified for a minmal set of supported metrics which could be reported to either statsd or prometheus. Metrics that are special to DataDog statsd will be left to another PR.

We are leaning towards an approach that would make it easy to instrument an existing app without introducing the ZMetrics service into the requirements for all instrumented effects.

Further, we came up with the idea to have the instrumentation methods also as extensions to the ZIO itself, so that we could do the following instrumentation:

trait InstrumentedSample {

  def doSomething    = ZMetrics.count("myCounter")(ZIO.succeed(print(".")))
  def doSomething2   = ZIO.succeed(print(".")).counted("myCounter2")
  def countSomething = ZIO.foreach_(1.to(100))(_ => doSomething2.zip(doSomething))

  def program: ZIO[ZEnv with ZMetrics, Nothing, ExitCode] = for {
    _ <- countSomething.absorbWith(_ => new Exception("Boom")).orDie
  } yield ExitCode.success
}

Here, each execution of doSomething or domeSomething2 would be counted. The extension method might feel more natural for ZIO developers.

[atooni]

The API has now changed so that instrumented effects push Metric Events to a globally available Queue, the MetricsChannel. The MetricsChannel provides a Stream of metric events to an instrumentation, so that the handling of those events can be postponed until they reach the backend reporter. See here for preliminary documentation.

[CLAassistant]

All committers have signed the CLA.

[atooni]

In the last review with @adamgfraser we have agreed to finish the support for Counters and Gauges reporting to StatsD and Prometheus and leave the other metrics to a later PR. The reasoning is the amount of code changes in this PR. Merging the PR will allow others to contribute with more documentation and other metrics.

The PR comes with Basic testing and documentation. I will be working on some more docs, but before the merge I am planning nor more fundamental code changes.

[atooni]

[atooni]

I have now added the first round of documentation for the metrics supported so far. I would suggest to get this PR into a mergable state rather sooner than later, so that we can synchronise with other PR's and tackle more supported metrics.

[atooni]

Preview of available metrics captured with Prometheus visualised with Grafana

[adamgfraser]

Let's get this merged!