An oversimplified, but entirely repeatable, example of distributed tracing with Linkerd, OpenTelemetry and Jaeger in Kubernetes using ASP.Net Core, gRPC, Redis and SQL Server.
The purpose of this demo is to show you how to add distributed tracing into your .Net Core APIs and micro-services with the OpenTelemetry Beta for .Net and make it interact with the existing tracing capabilities and tooling in Linkerd 2.
This is not intended to be an in-depth discussion of the OpenTelemetry project, Linkerd or Kubernetes. I've got just enough detail in the code and setup to give you a good idea of how to make it work for yourself.
Contributions welcome: If you have suggestions on how this example can be improved, I welcome any and all feedback. This is a nights and weekends thing for me, so please don't be offended if I don't reply immediately.
- Do you do micro-services?
- Do you run in Kubernetes?
- Do you write your code in .Net Core?
- Does your code often run in environments you can't attach a debugger to?
- Does anything ever behave unexpectedly, in said environments?
They say a picture is worth a thousand words. How about this one?
Would you like to have this view when service A is not talking to service B, or when service C is taking so long your users are dying of old age? We've all been there, this tooling and pattern is what keeps me sane and productive.
Starting with a new (empty) Kubernetes cluster, will avoid unnecessary complications.
I ran my tests on AKS, but there is nothing really Azure specific. This should work on any cloud or even bare-metal Kubernetes cluster. However, the LoadBalancer service for the ingress controller, and PersistentVolumeClaim for SQL Server, will require additional setup on bare-metal.
I'm executing all of these commands from the setup folder in this repository.
I'm doing a textbook Linkerd 2 install with:
$ linkerd install | kubectl apply -f -Check out "https://linkerd.io/2/getting-started" for a detailed guide on Linkerd 2 setup.
This consists of OpenCensus Collector and Jaeger.
$ linkerd upgrade --config 02-linkerd-addon.yaml | kubectl apply -f -See "https://linkerd.io/2/tasks/distributed-tracing" for more information.
3) Install Nginx Ingress Controller.
There are many ways to skin this animal, I've included the YAML I used for my demo cluster.
The Namespace definition in this YAML contains the same annotations as in step 6 (below). If you don't use my YAML, you'll need to add these annotations to your ingress-nginx Namespace. Also, the ConfigMap definition contains values for enable-opentracing:, zipkin-collector-host: and zipkin-sample-rate: that are required for this to work.
$ kubectl apply -f 03-nginx-ingress.yamlAsk for the newly assigned external IP for the Ingress service and test it via HTTP, you should get a 404.
$ kubectl get svc -n ingress-nginxThere are at least two ways to do this:
My preferred method is to run the Linkerd dashboard, then browse to "http://localhost:50750/jaeger".
$ linkerd dashboard
^ (CTRL+C when done)The second option is to port-forward the service directly, then browse to "http://localhost:16686/".
$ kubectl port-forward -n linkerd svc/linkerd-jaeger 16686
^ (CTRL+C when done)This Namespace definition contains the annotations for Linkerd to automatically watch any new resources in the Namespace and inject the Linkerd proxy as well as participate in distributed tracing spans.
$ kubectl apply -f 06-demo-ns.yaml
$ kubectl config set-context --current --namespace=demoAlternatively, you could add Linkerd annotations to your "default" Namespace.
$ kubectl annotate ns default \
linkerd.io/inject=enabled \
config.linkerd.io/trace-collector=linkerd-collector.linkerd:55678 \
config.alpha.linkerd.io/trace-collector-service-account=linkerd-collectorI will never hard-code an "sa" password, so this requires 2 commands.
$ kubectl create secret generic mssql --from-literal=SA_PASSWORD=<your-sa-password>
$ kubectl apply -f 07-mssql.yaml8) Create the database with "mssql-cli".
My command below uses local port 14330 (in case you have SQL Server installed locally). You will have to wait for the mssql-dev-0 pod to fully start before you can complete this step. It may take a minute to provision the persistent volume.
This requires two shell prompts, one for the port-forward, and one for mssql-cli. You could also port-forward in one shell prompt and run the script in any SQL GUI tool.
$ kubectl port-forward pod/mssql-dev-0 14330:1433
^ (CTRL+C when done)$ mssql-cli -S 127.0.0.1,14330 -U sa -P <your-sa-password> -i 08-create-db.sqlThis one is easy, as I'm only using Redis as an ephemeral cache and not persisting anything to disk.
$ kubectl apply -f 09-redis.yamlThis is one large script to reduce the number of steps.
$ kubectl apply -f 10-sample-apps.yamlThis will allow you to access your WebApi from outside the cluster. This also causes Nginx to initiate the traces that we will propagate.
My example Ingress route is for http://greeter.demo which requires you to modify your hosts file to point this hostname to the external IP you discovered in step 4 (above). Another option would be to point an actual DNS address at your IP and modify the "host" line in the Ingress definition
$ kubectl apply -f 11-ingress.yamlGive the sample apps a minute to start up, then browse to "http://greeter.demo/". You should get Swagger docs to implement a client or you can just play with the API right through the browser.
Repeating the process from Step 5 will let you see your traces.
You can look in your SQL Database at any time with the following commands. This requires port-forwarding and therefor two command prompts.
$ kubectl port-forward pod/mssql-dev-0 14330:1433
^ (CTRL+C when done)$ mssql-cli -S 127.0.0.1,14330 -U sa -P <your-sa-password> -d GreetingDb -Q "SELECT * FROM dbo.Greetings"I know what you're thinking. This example is intended for developers, but most of what you've read to this point has been: infrastructure, configuration and deployment. We all love DevOps these days, don't we?!?
The good news is that adding OpenTelemetry to your code is simple and probably takes less time than the scratch Kubernetes configuration does.
NOTE: My code uses Serilog to enhance the default ASP.Net Core logging. This is not a requirement for tracing, I just like it better. I especially love it when combined with Seq by Datalust. I highly recommend the combination of Serilog and Seq for distributed systems. For an excellent introduction to Serilog and Seq, check out C# Logging with Serilog and Seq - Structured Logging Made Easy by IAmTimCorey on YouTube.
You need to install some OpenTelemetry NuGet packages into your project(s). The simplest way to see the packages needed is to look in my two sample service projects.
This is my WebApi that calls the gRPC service.
This is my gRPC service that also talks to Redis and SQL Server:
Again, the simple solution is to look in the samples.
The important bits to pay attention to are:
// create and register an activity source
var activitySource = new ActivitySource(Program.Name);
services.AddSingleton(activitySource);This creates and registers an ActivitySource with DI. The Activity Source is the key to having control of your internal tracing detail. The framework will largely take care of RPC tracing once configured properly.
Don't worry about Program.Name, that was a convenient place for me to put a constant that I'll use a few times.
// TextFormat Defaults to W3C - Enable B3 via configuration
string tracingFormat = Configuration["Tracing:Format"]?.ToLowerInvariant();
if (tracingFormat == "b3m") // B3 (multi) headers come from Nginx
OpenTelemetry.Sdk.SetDefaultTextMapPropagator(new B3Propagator(singleHeader: false));
else if (tracingFormat == "b3s")
OpenTelemetry.Sdk.SetDefaultTextMapPropagator(new B3Propagator(singleHeader: true));This is how we configure OpenTelemetry.Net to use B3m headers instead of the default W3C headers. This is required to work with Nginx and Linkerd.
// Configure OpenTelemetry
services.AddOpenTelemetryTracing(builder =>
{
// register the activity source
builder.AddSource(activitySource.Name);
// Add ASP.Net Core Request Handling
builder.AddAspNetCoreInstrumentation(options =>
{
options.EnableGrpcAspNetCoreSupport = true;
});
...This sets up the OpenTelemetry framework and registers our ActivitySource. This also adds the automatic handling of incoming ASP.Net Core Requests (MVC, WebApi & gRPC) and will accept inbound trace headers.
// Add gRPC Propagation
builder.AddGrpcClientInstrumentation();This is only in the WebApi and tells the framework to propagate child spans via HTTP Headers on outbound requests with GrpcClient.
// add automatic instrumentation for Sql Server
builder.AddSqlClientInstrumentation(options =>
{
options.SetTextCommandContent = true; // probably not in production?
options.EnableConnectionLevelAttributes = true;
});
// add automatic instrumentation for Redis (needs ConnectionMultiplexer)
builder.AddRedisInstrumentation(redisProtoCache.Connection);These bits are only in the gRPC service, since only it talks to Redis and SQL Server. This is pretty self-explanatory. These providers allow capture and tracing inside calls to Sql and Redis.
// export to Zipkin receiver
string zipkinUrl = Configuration.GetConnectionString("Telemetry");
if (!string.IsNullOrEmpty(zipkinUrl))
{
builder.AddZipkinExporter(options =>
{
options.ServiceName = Program.Name;
options.Endpoint = new Uri(zipkinUrl);
});
}
// enable console output by configuration
if (Configuration.GetValue<bool>("Tracing:Console", false))
builder.AddConsoleExporter();Finally, this is in both services. This is how we configure OpenTelemetry to write the spans out to the Zipkin receiver and optionally, the console. The appsettings.json files in both service projects have the default Telemetry connection string you'll use with Linkerd.
The ActivitySource and Activity classes are the key to internal tracing spans and adding data. To start using them, add the ActivitySource via DI. This works in Controllers and GrpcServices.
public class GreetingController : ControllerBase
{
private readonly ILogger<GreetingController> _logger;
private readonly Greeter.GreeterClient _greeterClient;
private readonly ActivitySource _activitySource;
public GreetingController(ActivitySource activitySource, Greeter.GreeterClient greeterClient, ILogger<GreetingController> logger)
{
_activitySource = activitySource;
_greeterClient = greeterClient;
_logger = logger;
}
...Of course I'm going to point you to the actual code again for more info.
webapi/Controllers/GreetingController.cs
grpc/Services/GreeterService.cs
Here are a few snippets that are interesting from GreeterService.cs and deserve a bit of explanation.
if (context.Status.StatusCode != StatusCode.OK)
{
// record validation failure as trace event
Activity.Current?.AddEvent(new ActivityEvent(context.Status.ToString()));
...The snippet above adds an ActivityEvent to the current Activity to include details of input validation failure in the trace.
// try to get our reply from the cache
using (Activity tryCacheActivity = _activitySource.StartActivity("Check-Cache"))
{
tryCacheActivity?.AddTag("cache.searchKey", request.Name);
HelloReply cacheHit = await _cache.GetProto<HelloReply>(request.Name);
if (cacheHit != null)
{
tryCacheActivity?.AddTag("cache.result", "hit");
return cacheHit;
}
tryCacheActivity?.AddTag("cache.result", "miss");
}This snippet creates a new "Check-Cache" activity before going out to Redis and tags on the hit/miss result, as well as the key searched, to the trace. You saw this pointed out in the screenshot above.
using (Activity insertActivity = _activitySource.StartActivity("Insert-Sql-Record"))
{
int recId = await _repository.InsertGreeting(record, context.CancellationToken);
insertActivity?.AddTag("record.id", recId.ToString());
_logger.LogInformation("Inserted new greeting record {@record}", record);
}This Activity tracks the round-trip to the database and includes the new record Identity in the trace info. This can be a sanity saver when you need to look for a data inconsistency caused by a service error.
If you want to see the actual B3 headers that make this all work, check out the included "setup/extras/echoheaders.yaml".
$ kubectl apply -f extras/echoheaders.yamlThe included ingress rule maps "echo.greeter.demo", so you'll need to add that to your hosts file also. Give it the same IP as you have from setup step 4. If you open a browser to "http://echo.greeter.demo/" you'll see something like the following.
...
Request Headers:
accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding=gzip, deflate
accept-language=en-US,en;q=0.9
cache-control=max-age=0
host=echo.greeter.demo
l5d-dst-canonical=echoheaders.demo.svc.cluster.local:80
upgrade-insecure-requests=1
user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36
x-b3-flags=0
x-b3-parentspanid=529cef1c2e371ecf
x-b3-sampled=1
x-b3-spanid=3551dac81b7ba8d4
x-b3-traceid=178f408e6509b549
x-forwarded-for=127.0.0.1
x-forwarded-host=echo.greeter.demo
x-forwarded-port=80
x-forwarded-proto=http
x-real-ip=127.0.0.1
x-request-id=c739708686ab84579ba692d33ae5231e
x-scheme=http
...
These are all the HTTP headers that Nginx sent to your Pod, simply echoed back to you. The ones that start with x-b3-* are the relevant headers for tracing. See openzipkin/b3-propagation on GitHub for a lot more detail.
There are a few ways to get this kind of capability, and many different tools and techniques. This is simply my recipe.
I hope you enjoy!