Not getting any trace headers when I send requests from an instrumented service

[rcdailey]

I have an ASP.NET Core .NET 7 project set up that uses YARP. Using a Swagger-generated C# client library, I'm making a REST API call to another service. Even though I've integrated the latest prerelease packages into my code, I am not seeing any trace headers attached to the outgoing request. I'll dump some code below. It's been frustrating to get any of this working due to the lack of documentation, unclear documentation, or just out of date information.

Code I use in program.cs to set everthing up:

private static void SetupOpenTelemetry(IServiceCollection services)
{
    const string serviceName = "Monitor.ApiProxy";
    var resourceBuilder = ResourceBuilder.CreateDefault()
        .AddService(serviceName, GitVersionInformation.SemVer);

    var activitySource = new ActivitySource(serviceName, GitVersionInformation.SemVer);

    services.AddOpenTelemetry()
        .WithTracing(builder => builder
            .AddSource(activitySource.Name)
            .SetResourceBuilder(resourceBuilder)
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddConsoleExporter()
            .AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri("http://otel:4317");
            }));

    services.AddSingleton(activitySource);
}

I also added this to Program.cs to try to get propagation working (which is what I assume is needed to get the headers, but this is just trial & error on my part):

// See: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Extensions.Propagators
Sdk.SetDefaultTextMapPropagator(new TraceContextPropagator());

Lastly, in the class that uses the swagger C# client class (which itself has an HttpClient), I tried using the ActivitySource to start a new activity, thinking it might help, but it doesn't. For that matter, I thought this was supposed to be built in so I get an implicit start activity for every HTTP request I send out:

// In my composition root, set up the C# client object with http client factory:
services.AddHttpClient<IDeploymentService, DeploymentService>();

// Try starting an activity before I use the object to send a request:
using var activity = _activity.StartActivity("GetMappings");

My testing was done by:

1. Setting breakpoints in the receive callback in the destination service to inspect headers
2. Using Fiddler to inspect traffic

Here are the packages I'm using. Again, this is .NET 7.

<PackageVersion Include="Castle.Core" Version="5.1.1" />
<PackageVersion Include="GitVersion.MsBuild" Version="5.12.0" />
<PackageVersion Include="JetBrains.Annotations" Version="2022.3.1" />
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Versioning" Version="5.0.0" />
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Versioning.ApiExplorer" Version="5.0.0" />
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="7.0.0" />
<PackageVersion Include="OpenTelemetry.Exporter.Console" Version="1.4.0-rc.4" />
<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.4.0-rc.4" />
<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.4.0-rc.4" />
<PackageVersion Include="OpenTelemetry.Extensions.Propagators" Version="1.4.0-rc.4" />
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.13" />
<PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.13" />
<PackageVersion Include="Serilog.AspNetCore" Version="6.1.0" />
<PackageVersion Include="System.IO.Abstractions" Version="17.1.1" />
<PackageVersion Include="TestableIO.System.IO.Abstractions.Extensions" Version="1.0.39" />
<PackageVersion Include="Yarp.ReverseProxy" Version="2.0.0-rc.1.23068.3" />

What am I missing?

[cijothomas]

Yarp. See https://cloud-native.slack.com/archives/C01N3BC2W7Q/p1675781903348719

[rcdailey]

I'm not able to view that link. I do not have an account in that slack workspace.

[cijothomas]

Its a long thread, so can't paste it all. Here the most relevant link from that discussion:
microsoft/reverse-proxy#1497 (comment)

[rcdailey]

Thanks for the link. Everything I'm reading there says what I'm doing should work, so I'm not really clear on what you are suggesting. Also worth pointing out that the specific HTTP call here is being done out of band: It's a call made from scratch as part of the configuration process of YARP, not really a request being proxied. I assume that has some significance.

Your response, at the very least, hints to me that I probably need to engage the YARP developers for more information. Hopefully I can get their help in another thread I started over there.

Thanks again for the help.

[cijothomas]

If you learn something, please post back, so others can see as well. Thanks!