Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
19 contributors

Users who have contributed to this file

@SergeyKanzhelev @lmolkova @MikeGoldsmith @CodeBlanch @SychevIgor @ysma500 @z1c0 @TimothyMothra @sywhang @lukasz-pyrzyk @lizthegrey @c24t @JamesNK @EGHornbostel @eddynaka @cijothomas @bruno-garcia @andy-g @mgodse
769 lines (603 sloc) 38.2 KB

OpenTelemetry .NET SDK - distributed tracing and stats collection framework

.NET Channel: Gitter chat

Community Channel: Gitter chat

We hold regular meetings. See details at community page.

Approvers (@open-telemetry/dotnet-approvers):

Find more about the approver role in community repository.

Maintainers (@open-telemetry/dotnet-maintainers):

Find more about the maintainer role in community repository.

Build Status

OpenTelemetry is a toolkit for collecting application performance and behavior data.

The library is in Alpha stage. The library is expected to move to GA stage after v1.0.0 major release.

Please join gitter for help or feedback on this project.

We encourage contributions. Use tags up-for-grabs and good first issue to get started with the project. Follow CONTRIBUTING guide to report issues or submit a proposal.

Packages

Nightly builds

Myget feeds:

API and implementation

Package MyGet (CI) NuGet (releases)
OpenTelemetry MyGet Nightly NuGet Release
OpenTelemetry.Api MyGet Nightly NuGet Release

Data Collectors

Package MyGet (CI) NuGet (releases)
ASP.NET Core MyGet Nightly NuGet Release
ASP.NET (WebForms, MVC, & WebAPI) MyGet Nightly NuGet Release
.NET Core HttpClient, .NET Framework HttpClient & HttpWebRequest, Microsoft.Data.SqlClient, System.Data.SqlClient, & Azure SDKs MyGet Nightly NuGet Release
StackExchange.Redis MyGet Nightly NuGet Release

Exporters Packages

Package MyGet (CI) NuGet (releases)
Zipkin MyGet Nightly NuGet release
Prometheus MyGet Nightly NuGet release
Application Insights MyGet Nightly NuGet release
Stackdriver MyGet Nightly NuGet release
Jaeger MyGet Nightly NuGet release
LightStep MyGet Nightly NuGet release
Honeycomb NuGet release
NewRelic NuGet release
Console MyGet Nightly

OpenTelemetry Tracing QuickStart: Collecting Data

You can use OpenTelemetry API to instrument code and report data. Check out Tracing API overview to learn more about distributed tracing.

In the examples below we demonstrate how to create and enrich spans though OpenTelemetry API.

OpenTelemetry also provides auto-collectors for ASP.NET Core & ASP.NET incoming requests and HttpClient (.NET Core & .NET Framework), SqlClient, & Azure SDK outgoing requests. See the configuration section for examples detailing how to enable auto-collection.

Obtaining tracer

Applications should follow configuration section to find out how to create/obtain tracer.

Libraries must take dependency on OpenTelemetry API package only and should never instantiate tracer or configure OpenTelemetry. Libraries will be able to obtain global tracer that may either be noop (if user application is not instrumented with OpenTelemetry) or real tracer implemented in the SDK package.

Create basic span

To create the most basic span, you only specify the name. OpenTelemetry SDK collects start/end timestamps, assigns tracing context and assumes status of this span is OK.

var span = tracer.StartSpan("basic span");
// ...
span.End();

Create nested spans

In many cases you want to collect nested operations. You can propagate parent spans explicitly in your code or use implicit context propagation embedded into OpenTelemetry and .NET.

Explicit parent propagation and assignment

var parentSpan = tracer.StartSpan("parent span");

// explicitly assigning parent here
var childSpan = tracer.StartSpan("child span", parentSpan);

childSpan.End();
parentSpan.End();

Implicit parent propagation and assignment

// calling StartActiveSpan starts a span and puts parentSpan into the ambient context
// that flows in async calls.   When child is created, it implicitly becomes child of current span
using (tracer.StartActiveSpan("parent span", out _))
{
    var childSpan = tracer.StartSpan("child span");

    childSpan.End();
}

// parent span is ended when StartActiveSpan result is disposed

Span with attributes

Attributes provide additional context on span specific to specific operation it tracks such as HTTP/DB/etc call properties.

// spans have Client, Server, Internal, Producer and Consumer kinds to help visualize them
var span = tracer.StartSpan("span with attributes", SpanKind.Client);

// attributes specific to the call
span.SetAttribute("db.type", "redis");
span.SetAttribute("db.instance", "localhost:6379[0]");
span.SetAttribute("db.statement", "SET");
span.End();

Span with links

Links allow to create relationships between different traces i.e. allow spans to have multiple relatives. They are typically used to trace batching scenarios where multiple traces are merged into another one.

Links affect sampling decision and should be added before sampling decision is made (i.e. before span starts).

SpanContext link1 = ExtractContext(eventHubMessage1);
SpanContext link2 = ExtractContext(eventHubMessage2);

var span = tracer.StartSpan("span with links", SpanKind.Server, DateTime.UtcNow, new [] {link1, link2});
span.End();

Span with events

Events are timed text (with optional attributes) annotations on the span. Events can be added to current span (or any running span).

using (tracer.StartActiveSpan("incoming HTTP request", SpanKind.Server, out var span))
{
    span.AddEvent("routes resolved");
}

// span is ended when StartActiveSpan result is disposed

Context propagation out of process

When instrumenting transport-layer operations, instrumentation should support context propagation.

// this extracts W3C trace-context from incoming HTTP request
// context may be valid if present and correct in the headers
// or invalid if there was no context (or it was not valid)
// instrumentation code should not care about it
var context = tracer.TextFormat.Extract(incomingRequest.Headers, (headers, name) => headers[name]);

var incomingSpan = tracer.StartSpan("incoming http request", context, SpanKind.Server);

var outgoingRequest = new HttpRequestMessage(HttpMethod.Get, "http://microsoft.com");
var outgoingSpan = tracer.StartSpan("outgoing http request", SpanKind.Client);

// now that we have outgoing span, we can inject it's context
// Note that if there is no SDK configured, tracer is noop -
// it creates noop spans with invalid context. we should not propagate it.
if (outgoingSpan.Context.IsValid)
{
    tracer.TextFormat.Inject(
        outgoingSpan.Context,
        outgoingRequest.Headers,
        (headers, name, value) => headers.Add(name, value));
}

// make outgoing call
// ...

outgoingSpan.End();
incomingSpan.End();

Auto-collector implementation for Activity/DiagnosticSource

System.Diagnostics.Activity is similar to OpenTelemetry Span. HttpClient, ASP.NET Core, SqlClient, & Azure SDKs use them to expose diagnostics events and context.

Leaving aside subscription mechanism, here is an example how you may implement callbacks for Start/Stop Activity

void StartActivity()
{
    var span = tracer.StartSpanFromActivity("GET api/values", Activity.Current);

    // extract other things from Activity and set them on span (tags to attributes)
    // ...
}

void StopActivity()
{
    var span = tracer.CurrentSpan;
	
	span.End();
    if (span is IDisposable disposableSpan)
    {
        disposableSpan.Dispose();
    }
}

Configuration

Configuration is done by user application: it should configure exporter and may also tune sampler and other properties.

Basic Configuration

  1. Install packages to your project: OpenTelemetry OpenTelemetry.Exporter.Zipkin

  2. Create TracerFactory

    using (TracerFactory.Create(builder => builder
            .UseZipkin(options => {})
            .SetResource(Resources.CreateServiceResource("http-client-test")))
    {
        // ...
    }

Configuration with Microsoft.Extensions.DependencyInjection

  1. Install packages to your project: OpenTelemetry.Hosting to provide AddOpenTelemetry helper method OpenTelemetry.Collector.AspNetCore to collect incoming HTTP requests OpenTelemetry.Collector.Dependencies to collect outgoing HTTP requests, SqlClient calls, and Azure SDK calls

  2. Make sure TracerFactory, is registered in DI.

    services.AddOpenTelemetry(builder =>
    {
        builder
            .SetSampler(new AlwaysSampleSampler())
            .UseZipkin(options => {})
    
            // you may also configure request and dependencies collectors
            .AddRequestCollector()
            .AddDependencyCollector()
            .SetResource(Resources.CreateServiceResource("my-service"))
    });

Configuration with ASP.NET (Full .NET Framework) running in IIS or IIS Express (if supported)

  1. Add a reference to the OpenTelemetry.Collector.AspNet package. Add any other collectors & exporters you will need.

  2. Add the Microsoft telemetry module in your Web.config:

    <system.webServer>
        <modules>
          <add name="TelemetryCorrelationHttpModule" type="Microsoft.AspNet.TelemetryCorrelation.TelemetryCorrelationHttpModule, Microsoft.AspNet.TelemetryCorrelation" preCondition="integratedMode,managedHandler"/>
        </modules>
    </system.webServer>
  3. Configure OpenTelemetry in your application startup:

    public class WebApiApplication : HttpApplication
    {
        private TracerFactory tracerFactory;
    
        protected void Application_Start()
        {
            this.tracerFactory = TracerFactory.Create(builder =>
            {
                builder
                    .UseJaeger(c =>
                    {
                        c.AgentHost = "localhost";
                        c.AgentPort = 6831;
                    })
                    .AddRequestCollector()
                    .AddDependencyCollector();
            });
        }
    
        protected void Application_End()
        {
            this.tracerFactory?.Dispose();
        }
    }

Using StackExchange.Redis collector

Outgoing http calls to Redis made using StackExchange.Redis library can be automatically tracked.

  1. Install package to your project: OpenTelemetry.Collector.StackExchangeRedis

  2. Configure Redis collector

    // connect to the server
    var connection = ConnectionMultiplexer.Connect("localhost:6379");
    
    using (TracerFactory.Create(b => b
                .SetSampler(new AlwaysSampleSampler())
                .UseZipkin(options => {})
                .SetResource(Resources.CreateServiceResource("my-service"))
                .AddCollector(t =>
                {
                    var collector = new StackExchangeRedisCallsCollector(t);
                    connection.RegisterProfiler(collector.GetProfilerSessionsFactory());
                    return collector;
                })))
    {
    
    }

You can combine it with dependency injection as shown in previous example.

Custom samplers

You may configure sampler of your choice

 using (TracerFactory.Create(b => b
            .SetSampler(new ProbabilitySampler(0.1))
            .UseZipkin(options => {})
            .SetResource(Resources.CreateServiceResource("my-service")))
{

}

You can also implement custom sampler by implementing ISampler interface

class MySampler : Sampler
{
    public override string Description { get; } = "my custom sampler";

    public override Decision ShouldSample(SpanContext parentContext, ActivityTraceId traceId, ActivitySpanId spanId, string name,
        IDictionary<string, object> attributes, IEnumerable<Link> links)
    {
        bool sampledIn;
        if (parentContext != null && parentContext.IsValid)
        {
            sampledIn = (parentContext.TraceOptions & ActivityTraceFlags.Recorded) != 0;
        }
        else
        {
            sampledIn = Stopwatch.GetTimestamp() % 2 == 0;
        }

        return new Decision(sampledIn);
    }
}

OpenTelemetry QuickStart: Exporting Data

Using the Jaeger exporter

The Jaeger exporter communicates to a Jaeger Agent through the compact thrift protocol on the Compact Thrift API port. You can configure the Jaeger exporter by following the directions below:

  1. Get Jaeger.
  2. Configure the JaegerExporter
    • ServiceName: The name of your application or service.
    • AgentHost: Usually localhost since an agent should usually be running on the same machine as your application or service.
    • AgentPort: The compact thrift protocol port of the Jaeger Agent (default 6831)
    • MaxPacketSize: The maximum size of each UDP packet that gets sent to the agent. (default 65000)
  3. See the sample for an example of how to use the exporter.
using (var tracerFactory = TracerFactory.Create(
    builder => builder.UseJaeger(o =>
    {
        o.ServiceName = "jaeger-test";
        o.AgentHost = "<jaeger server>";
    })))
{
    var tracer = tracerFactory.GetTracer("jaeger-test");
    using (tracer.StartActiveSpan("incoming request", out var span))
    {
        span.SetAttribute("custom-attribute", 55);
        await Task.Delay(1000);
    }
}

Using Zipkin exporter

Configure Zipkin exporter to see traces in Zipkin UI.

  1. Get Zipkin using getting started guide.
  2. Configure ZipkinTraceExporter as below:
  3. See sample for example use.
using (var tracerFactory = TracerFactory.Create(builder => builder
    .UseZipkin(o =>
    {
        o.ServiceName = "test-zipkin";
        o.Endpoint = new Uri(zipkinUri);
    })))
{
    var tracer = tracerFactory.GetTracer("zipkin-test");

    // Create a scoped span. It will end automatically when using statement ends
    using (tracer.WithSpan(tracer.StartSpan("Main")))
    {
        Console.WriteLine("About to do a busy work");
        for (var i = 0; i < 10; i++)
        {
            DoWork(i, tracer);
        }
    }
}

Using Prometheus exporter

Configure Prometheus exporter to have stats collected by Prometheus.

  1. Get Prometheus using getting started guide.
  2. Start PrometheusExporter as below.
  3. See sample for example use.
var exporter = new PrometheusExporter(
    new PrometheusExporterOptions()
    {
        Url = "http://+:9184/metrics/"
    },
    Stats.ViewManager);

exporter.Start();

try
{
    // record metrics
    statsRecorder.NewMeasureMap().Put(VideoSize, values[0] * MiB).Record();
}
finally
{
    exporter.Stop();
}

Using the NewRelic exporter

The New Relic OpenTelemetry Trace Exporter is a OpenTelemetry Provider that sends data from .NET applications to New Relic. It uses the NewRelic SDK to send Traces to the New Relic backend

Please refer to the New Relic Exporter Documentation

Using Stackdriver Exporter

This sample assumes your code authenticates to Stackdriver APIs using service account with credentials stored in environment variable GOOGLE_APPLICATION_CREDENTIALS. When you run on GAE, GKE or locally with gcloud sdk installed - this is typically the case. There is also a constructor for specifying path to the service account credential. See sample for details.

  1. Add Stackdriver Exporter package reference.
  2. Enable Stackdriver Trace API.
  3. Enable Stackdriver Monitoring API.
  4. Instantiate a new instance of StackdriverExporter with your Google Cloud's ProjectId
  5. See sample for example use.

Advanced configuration

You may want to filter on enrich spans and send them to multiple destinations (e.g. for debugging or telemetry self-diagnostics purposes). You may configure multiple processing pipelines for each destination like shown in below example.

In this example

  1. First pipeline sends all sampled in spans to Zipkin
  2. Second pipeline sends spans to ApplicationInsights, but filters them first with custom built FilteringSpanProcessor
  3. Third pipeline adds custom DebuggingSpanProcessor that simply logs all calls to debug output
using (var tracerFactory = TracerFactory.Create(builder => builder
    .UseZipkin(o =>
    {
        o.Endpoint = new Uri(zipkinUri);
    })
    .UseApplicationInsights(
        o => o.InstrumentationKey = "your-instrumentation-key",
        p => p.AddProcessor(nextProcessor => new FilteringSpanProcessor(nextProcessor)))
    .AddProcessorPipeline(pipelineBuilder => pipelineBuilder.AddProcessor(_ => new DebuggingSpanProcessor()))))
    .SetResource(Resources.CreateServiceResource("test-zipkin"))

{
    // ...
}

Traces

var spanExporter = new StackdriverTraceExporter(projectId);

using var tracerFactory = TracerFactory.Create(builder => builder.AddProcessorPipeline(c => c.SetExporter(spanExporter)));
var tracer = tracerFactory.GetTracer("stackdriver-test");

using (tracer.StartActiveSpan("/getuser", out TelemetrySpan span))
{
    span.AddEvent("Processing video.");
    span.PutHttpMethodAttribute("GET");
    span.PutHttpHostAttribute("localhost", 8080);
    span.PutHttpPathAttribute("/resource");
    span.PutHttpStatusCodeAttribute(200);
    span.PutHttpUserAgentAttribute("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0");

    Thread.Sleep(TimeSpan.FromMilliseconds(10));
}

Metrics

var metricExporter = new StackdriverExporter(
    "YOUR-GOOGLE-PROJECT-ID",
    Stats.ViewManager);
metricExporter.Start();

Using Application Insights exporter in Console App

  1. Create Application Insights resource.
  2. Set instrumentation key via telemetry configuration object (new TelemetryConfiguration("iKey")). This object may be injected via dependency injection as well.
  3. Instantiate a new instance of ApplicationInsightsExporter.
  4. See sample for example use.
using var tracerFactory = TracerFactory.Create(builder => builder
    .SetResource(Resources.CreateServiceResource("my-service"))
    .UseApplicationInsights(config => config.InstrumentationKey = "instrumentation-key"));
var tracer = tracerFactory.GetTracer("application-insights-test");

using (tracer.StartActiveSpan("incoming request", out var span))
{
    span.AddEvent("Start processing video.");
    Thread.Sleep(TimeSpan.FromMilliseconds(10));
    span.AddEvent("Finished processing video.");
}

Using Application Insights exporter in Web App

  1. Create Application Insights resource.
  2. In Startup class, ConfigureServices method add a call of AddOpenTelemetry
  3. Configurate Application Insights inside AddOpenTelemetry with call UseApplicationInsights
  4. Set instrumentation key via telemetry configuration object telemetryConfiguration.InstrumentationKey = instrumentationKey;
  5. See sample for example use.
 services.AddOpenTelemetry((sp, builder) =>
            {
                    builder.UseApplicationInsights(telemetryConfiguration =>
                    {
                        var instrumentationKey = this.Configuration.GetValue<string>("ApplicationInsights:InstrumentationKey");
                        telemetryConfiguration.InstrumentationKey = instrumentationKey;
                    })
                    .AddRequestCollector()
                    .AddDependencyCollector();
            });

Using LightStep exporter

Configure LightStep exporter to see traces in LightStep.

  1. Setup LightStep using getting started guide
  2. Configure LightStepTraceExporter (see below)
  3. See sample for example use
using (var tracerFactory = TracerFactory.Create(
    builder => builder.UseLightStep(o =>
        {
            o.AccessToken = "<access-token>";
            o.ServiceName = "lightstep-test";
        })))
{
    var tracer = tracerFactory.GetTracer("lightstep-test");
    using (tracer.StartActiveSpan("incoming request", out var span))
    {
        span.SetAttribute("custom-attribute", 55);
        await Task.Delay(1000);
    }
}

Implementing your own exporter

Tracing

Exporters should subclass SpanExporter and implement ExportAsync and Shutdown methods. Depending on user's choice and load on the application ExportAsync may get called concurrently with zero or more spans. Exporters should expect to receive only sampled-in ended spans. Exporters must not throw. Exporters should not modify spans they receive (the same span may be exported again by different exporter).

It's a good practice to make exporter IDisposable and shut it down in IDispose unless it was shut down explicitly. This helps when exporters are registered with dependency injection framework and their lifetime is tight to the app lifetime.

class MyExporter : SpanExporter
{
    public override Task<ExportResult> ExportAsync(IEnumerable<Span> batch, CancellationToken cancellationToken)
    {
        foreach (var span in batch)
        {
            Console.WriteLine($"[{span.StartTimestamp:o}] {span.Name} {span.Context.TraceId.ToHexString()} {span.Context.SpanId.ToHexString()}");
        }

        return Task.FromResult(ExportResult.Success);
    }

    public override Task ShutdownAsync(CancellationToken cancellationToken)
    {
        return Task.CompletedTask;
    }
}

Users may configure the exporter similarly to other exporters. You should also provide additional methods to simplify configuration similarly to UseZipkin extension method.

var exporter = new MyExporter();
using (var tracerFactory = TracerFactory.Create(
    builder => builder.AddProcessorPipeline(b => b.SetExporter(new MyExporter())))
{
    // ...
}

Versioning

This library follows Semantic Versioning.

GA: Libraries defined at a GA quality level are stable, and will not introduce backwards-incompatible changes in any minor or patch releases. We will address issues and requests with the highest priority. If we were to make a backwards-incompatible changes on an API, we will first mark the existing API as deprecated and keep it for 18 months before removing it.

Beta: Libraries defined at a Beta quality level are expected to be mostly stable and we're working towards their release candidate. We will address issues and requests with a higher priority. There may be backwards incompatible changes in a minor version release, though not in a patch release. If an element is part of an API that is only meant to be used by exporters or other OpenTelemetry libraries, then there is no deprecation period. Otherwise, we will deprecate it for 18 months before removing it, if possible.

You can’t perform that action at this time.