How do I instrument Elixir application with telemetry?

The PR checklist is saying I need to instrument my changes with telemetry. How do I do that? What does it mean?

Telemetry is an erlang application that is common in the BEAM community to use to allow hooking into code for metrics.

For a simple example on how to use telemetry:

:telemetry.execute([:wck, :users, :create, :success], %{count: 1}, %{user_id: 1})

:telemetry.exexcute/3 does the heavy lifting for us. The first argument is our event name, the second is the measurement, and finally a map of metadata.

What should I instrument?

In general, we should instrument actions that the client cares about. If they care, we care to know it's happening.

For instance, in WCK we have CRUD events around Orders, Restaurants, and Users. We additionally have specific actions around couriers, the order changing statuses, and Twilio API interactions. All of these we care about to know that the system is working as intended.

Event Names

Event naming should be in the format of [<application>, <resource>, <action>, <status>]. In our example above, the application is :wck, the resource is :users, the action is :create, and the status is :success.

resource may be expanded if there are sub-resources, for instance :couriers, :delivery if the courier has a delivery specific update.

For a few more key examples:

• [:wck, :users, :reset_password, :fail]
• [:wck, :orders, :update, :success]
• [:wck, :couriers, :delivery, :create, :success]

Measurements

For most of our cases, we'll want the measurement to be %{count: 1} and that's it. 90% of the time we only care that something is happening and this is the measurement we'll pass along.

If we need something like a duration, the measurement will be %{duration: native}. Where native is a native time measurement, this is a difference in monotonic time.

For example:

diff = timings[:request_end] - timings[:request_start]
microseconds = System.convert_time_unit(diff, :native, :microsecond)

Both values in timings are set via https://hexdocs.pm/elixir/1.12/System.html#monotonic_time/0

Metadata

If we're working on a resource that exists, we should pass along the id of the resource in the metadata. We may or may not use it, but it is good to have in place either way.

If we tried creating a resource but failed, we don't have a resource to pass the id along and this is generally when we skip sending metadata.

Exporting Metrics

We use PromEx for exporting metrics. The application should have a App.PromEx.Reporter to extend, if it was cloned from SteinExample. We will add our custom metrics to this reporter.

This uses Telemetry.Metrics under the hood. See docs for Telemetry.Metrics for other metric types. We will typically use counter as mentioned earlier.

For counters, the event name is the full event name plus the measurement key that should be used, most often this will be event_name ++ [:count]. Make sure to also include a description of what the event is measuring.

Below is an example of order metrics from WCK.

defmodule WCK.PromEx.Reporter do
  @moduledoc """
  Custom application metrics exported through PromEx
  """

  use PromEx.Plugin

  @impl true
  def event_metrics(_opts) do
    [
      courier_metrics(),
      order_metrics(),
      twilio_metrics(),
      doordash_metrics()
    ]
  end

  # ...

  defp order_metrics() do
    Event.build(
      :wck_order_metrics,
      [
        counter(
          [:wck, :orders, :create, :success, :count],
          description: "Number of successful orders"
        ),
        counter(
          [:wck, :orders, :create, :fail, :count],
          description: "Number of failed orders"
        ),
        counter(
          [:wck, :orders, :recreate, :count],
          description: "Number of successful order re-creates"
        ),
        counter(
          [:wck, :orders, :mode, :success, :count],
          description: "Number of successful order mode changes"
        ),
        counter(
          [:wck, :orders, :mode, :fail, :count],
          description: "Number of failed order mode changes"
        )
      ]
    )
  end
end