Introduction

This repository contains a practical example about how to use OpenTelemetry to add custom metrics to a .NET app, and how to visualize those metrics using Prometheus and Grafana.

Content

The repository contains the following applications.

• The BookStore WebAPI uses the OpenTelemetry OTLP exporter package (OpenTelemetry.Exporter.OpenTelemetryProtocol) to send the metric data to the OpenTelemetry Collector.
• The Prometheus server obtains the metric data from the OpenTelemetry Collector.
• The Grafana server comes preconfigured with a few dashboards to visualize the OpenTelemetry metrics emitted by the BookStore WebApi.

Application

The application is a BookStore API built using .NET 7. It allows us to do the following actions:

• Get, add, update and delete book categories.
• Get, add, update and delete books.
• Get, add, update and delete inventory.
• Get, add and delete orders.

For a better understanding, here's how the database diagram looks like:

OpenTelemetry Metrics

The application generates the following metrics:

BookStore custom metrics

Those are the business metrics instrumented directly on the application using the Metrics API incorporated into the .NET runtime itself.

The metrics generated are the following ones:

• BooksAddedCounter: It counts how many books are added to the store.
• BooksDeletedCounter: It counts how many books are deleted from the store.
• BooksUpdatedCounter: It counts how many books are updated.
• TotalBooksUpDownCounter: Total number of books that the store has at any given time.
• CategoriesAddedCounter: It counts how many book categories are added to the store.
• CategoriesDeletedCounter: It counts how many book categories are deleted from the store.
• CategoriesUpdatedCounter: It counts how many book categories are updated.
• TotalCategoriesGauge: Total number of book categories that the store has at any given time.
• OrdersPriceHistogram: Shows the price distribution of the orders.
• NumberOfBooksPerOrderHistogram: Shows the number of books distribution per order.
• OrdersCanceledCounter: It counts how many orders has been cancelled.
• TotalOrdersCounter: Total number of orders that the store has at any given time.

Http requests metrics

Those metrics are generated by the OpenTelemetry.Instrumentation.AspNetCore NuGet package.
This is an instrumentation library, which instruments .NET and collects metrics and traces about incoming web requests.

To start using the OpenTelemetry.Instrumentation.AspNetCore package you only need to add the AddAspNetCoreInstrumentation() extension method when setting up the .NET OpenTelemetry component. Here's an example:

builder.Services.AddOpenTelemetry().WithMetrics(opts => opts
        .AddAspNetCoreInstrumentation()
);

System.Runtime performance metrics

Those metrics are generated by the OpenTelemetry.Instrumentation.Runtime NuGet package. This is an instrumentation library, which instruments .NET Runtime and collect runtime performance metrics.

To start using the OpenTelemetry.Instrumentation.Runtime package you only need to add the AddRuntimeInstrumentation() extension method when setting up the .NET OpenTelemetry component. Here's an example:

builder.Services.AddOpenTelemetry().WithMetrics(opts => opts
    .AddRuntimeInstrumentation()
);

The OpenTelemetry.Instrumentation.Runtime package collects metrics about the following System.Runtime counters:

• process.runtime.dotnet.gc.collections.count: Number of garbage collections that have occurred since process start.
• process.runtime.dotnet.gc.objects.size: Count of bytes currently in use by objects in the GC heap that haven't been collected yet. Fragmentation and other GC committed memory pools are excluded.
• process.runtime.dotnet.gc.allocations.size: Count of bytes allocated on the managed GC heap since the process start
• process.runtime.dotnet.gc.committed_memory.size: The amount of committed virtual memory for the managed GC heap, as observed during the latest garbage collection.
• process.runtime.dotnet.gc.heap.size: The heap size (including fragmentation), as observed during the latest garbage collection.
• process.runtime.dotnet.gc.heap.fragmentation.size: The heap fragmentation, as observed during the latest garbage collection.
• process.runtime.dotnet.jit.il_compiled.size: Count of bytes of intermediate language that have been compiled since the process start.
• process.runtime.dotnet.jit.methods_compiled.count: The number of times the JIT compiler compiled a method since the process start.
• process.runtime.dotnet.jit.compilation_time: The amount of time the JIT compiler has spent compiling methods since the process start.
• process.runtime.dotnet.monitor.lock_contention.count: The number of times there was contention when trying to acquire a monitor lock since the process start.
• process.runtime.dotnet.thread_pool.threads.count: The number of thread pool threads that currently exist.
• process.runtime.dotnet.thread_pool.completed_items.count: The number of work items that have been processed by the thread pool since the process start.
• process.runtime.dotnet.thread_pool.queue.length: The number of work items that are currently queued to be processed by the thread pool.
• process.runtime.dotnet.timer.count: The number of timer instances that are currently active.
• process.runtime.dotnet.assemblies.count: The number of .NET assemblies that are currently loaded.
• process.runtime.dotnet.exceptions.count: Count of exceptions that have been thrown in managed code, since the observation started.

Some of the GC related metrics will be unavailable until at least one garbage collection has occurred.

Process metrics

Those metrics are generated by the OpenTelemetry.Instrumentation.Process NuGet package. This is an Instrumentation Library, which instruments .NET and collects telemetry about process behavior.

To start using the OpenTelemetry.Instrumentation.Process package you only need to add the AddProcessInstrumentation() extension method when setting up the .NET OpenTelemetry component. Here's an example:

builder.Services.AddOpenTelemetry().WithMetrics(opts => opts
        .AddProcessInstrumentation()
);

The OpenTelemetry.Instrumentation.Process package the following metrics of the running process:

• process.memory.usage: The amount of physical memory allocated for this process.
• process.memory.virtual: The amount of committed virtual memory for this process. One way to think of this is all the address space this process can read from without triggering an access violation; this includes memory backed solely by RAM, by a swapfile/pagefile and by other mapped files on disk.
• process.cpu.time: Total CPU seconds broken down by states.
• process.cpu.count: The number of processors (CPU cores) available to the current process.
• process.threads: Process threads count.

OpenTelemetry .NET Client

The app uses the following package versions:

<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.14" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.5.0-alpha.2" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.1.0-rc.2" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.5.0-alpha.2" />
<PackageReference Include="OpenTelemetry.Instrumentation.Process" Version="0.5.0-beta.2" />

External Dependencies

• OpenTelemetry Collector
• Prometheus
• MSSQL Server
• Grafana

How to run the app

The repository contains a docker-compose that starts up the BookStore app and also the external dependencies. The external dependencies (OpenTelemetry Collector, Prometheus, MSSQL Server and Grafana) are already preconfigured so you don't need to do any extra setup.

• The OpenTelemetry Collector is already configured to export the metrics to Prometheus.
• The MSSQL Server comes with the BookStore database schema configured.
• The Prometheus is already configured to receive the metric data from the OpenTelemetry Collector.
• The Grafana has the Prometheus connector already setup, it also contains 3 custom dashboards to visualize the OpenTelemetry metrics emitted by the BookStore app.

Just run docker-compose up and your good to go!

How to test the app

It requires to have cURL installed on your local machine.

This repository contains a seed-data.sh Shell script that will invoke some endpoints of the BookStore API via cURL.

The seed-data.sh script runs the following operations:

• Adds 8 book categories.
• Updates 3 book categories.
• Deletes 2 book categories.
• Adds 17 books into the store.
• Updates 4 existing books.
• Deletes 2 existing books.
• Adds inventory for every book on the store.
• Creates 10 orders.
• Cancels 3 existing orders.

The purpose behind this script is to generate a decent amount of metrics that later can be visualized in Grafana.

Output

If you open Grafana you're going to see those 3 dashboards.

If you open those dashboards after running the seed-data.sh script from the previous section, you're going to see something like this.

BookStore custom metrics dashboard

.NET performance counters & process metrics dashboard

Http Requests dashboard

Changelog

04/09/2023

• Update application to .NET 7.
• Add a new middleware into the app that simulates latency.
• Update OpenTelemetry packages to the latest version.
• Fix breaking changes on the app due to the OpenTelemetry packages version upgrade.
• Fix a few broken panels on the Grafana dashboard due to the OpenTelemetry packages version upgrade.
• Install and configure 'OpenTelemetry.Instrumentation.Process' package to import CPU and memory metrics.
• Replace the 'TotalBooksGauge' metric with the 'TotalBooksUpDownCounter' metric. Update Grafana, Prometheus and OTEL Collector images used on the docker-compose to the newest versions.