Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry.OpenTelemetry: The Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to initialize the Sentry SDK. #2738

Closed
sinanbozkus opened this issue Oct 18, 2023 · 21 comments · Fixed by #2911 or #2918
Assignees
Labels
Bug Something isn't working

Comments

@sinanbozkus
Copy link

Package

Other

.NET Flavor

.NET

.NET Version

7.0.12

OS

Linux

SDK Version

7.0.0

Self-Hosted Sentry Version

No response

Steps to Reproduce

When I was using "Sentry.OpenTelemetry" 3.39.1 or older everything was ok. After updating 3.40.0 or 3.40.1 I got this error.

The Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to initialize the Sentry SDK.

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
            tracerProviderBuilder
                .AddSource("Development-Application.Api")
                .ConfigureResource(resource => resource.AddService("Development-Application.Api"))
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddSentry() // <-- Configure OpenTelemetry to send trace information to Sentry
    );

builder.WebHost.UseSentry(o =>
{
    o.Dsn = "XXXXXXXXX";
    o.TracesSampleRate = 1.0;
    o.UseOpenTelemetry();
});

Expected Result

No error

Actual Result

The Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to initialize the Sentry SDK.initialized

@jamescrosswell
Copy link
Collaborator

Thanks for the report @sinanbozkus !

Are you able to try switching those statments, so the Sentry registration comes first?

builder.WebHost.UseSentry(o =>
{
    o.Dsn = "XXXXXXXXX";
    o.TracesSampleRate = 1.0;
    o.UseOpenTelemetry();
});

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
            tracerProviderBuilder
                .AddSource("Development-Application.Api")
                .ConfigureResource(resource => resource.AddService("Development-Application.Api"))
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddSentry() // <-- Configure OpenTelemetry to send trace information to Sentry
    );

@sinanbozkus
Copy link
Author

sinanbozkus commented Oct 18, 2023

, so the Sentry registration comes first

services.AddOpenTelemetry()
            .WithTracing(builder =>
                {
                    builder.AddSentry();
                    builder.AddSource("Development-Application.Api"))
                        .ConfigureResource(resource => resource.AddService("Development-Application.Api"));
                    // builder.AddSentry(); // or here
                    builder.AddAspNetCoreInstrumentation();
                    builder.AddHttpClientInstrumentation();
                    builder.AddConsoleExporter();
                }
            );

I've tried like this but I got the same error.

@jamescrosswell
Copy link
Collaborator

Hi @sinanbozkus , can you try as I suggested (builder.WebHost.UseSentry comes first then builder.Services.AddOpenTelemetry)?

@sinanbozkus
Copy link
Author

sinanbozkus commented Oct 18, 2023

I've tried, it's the same.

@jamescrosswell
Copy link
Collaborator

Hm, the error itself is coming from here:

_options = hub.GetSentryOptions();
if (_options is not { })
{
throw new InvalidOperationException(
"The Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to " +
"initialize the Sentry SDK.");
}

Typically that would indicate that SentrySdk.CurrentHub had not been initialized correctly.

You could try adding o.Debug = true to your UseSentry... then check the debug output to see if there are any warnings or errors that might indicate what's going wrong with initialization.

If not, would it be possible to get you to share a minimal reproducible example in a GitHub repo that we could clone?

@raulidavid
Copy link

I have the same problem the Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to initialize the Sentry SDK.

Follow the instructions @jamescrosswell

@jamescrosswell
Copy link
Collaborator

@raulidavid I'm unable to reproduce this... the Sample app provided with Sentry works fine for me.

Are you able to provide a copy of what you're seeing in the Debug window when you run the application with Debug = true in your SentryOptions?

@raulidavid
Copy link

raulidavid commented Oct 19, 2023

Version 3.40.1 Error
<PackageVersion Include="Sentry.AspNetCore" Version="3.39.1" /> <PackageVersion Include="Sentry.OpenTelemetry" Version="3.39.1" /> <PackageVersion Include="Sentry.AspNetCore" Version="3.40.1" /> ERROR <PackageVersion Include="Sentry.OpenTelemetry" Version="3.40.1" /> ERROR

@jamescrosswell
Copy link
Collaborator

Version 3.40.1 Error <PackageVersion Include="Sentry.AspNetCore" Version="3.39.1" /> <PackageVersion Include="Sentry.OpenTelemetry" Version="3.39.1" /> <PackageVersion Include="Sentry.AspNetCore" Version="3.40.1" /> ERROR <PackageVersion Include="Sentry.OpenTelemetry" Version="3.40.1" /> ERROR

@raulidavid I think we're maybe not on the same wavelength... I mean the output that you see in the Debug console in your IDE. For example, this is what I see when I run the application in debug mode with SentryOptions.Debug = true:

/usr/local/share/dotnet/dotnet /Users/jamescrosswell/code/sentry-dotnet/samples/Sentry.Samples.OpenTelemetry.AspNetCore/bin/Debug/net7.0/Sentry.Samples.OpenTelemetry.AspNetCore.dll
  Debug: Logging enabled with ConsoleDiagnosticLogger and min level: Debug
  Debug: Initializing Hub for Dsn: 'https://b887218a80114d26a9b1a51c5f88e0b4@o447951.ingest.sentry.io/6601807'.
  Debug: Using 'GzipBufferedRequestBodyHandler' body compression strategy with level Optimal.
  Debug: Starting BackgroundWorker.
  Debug: BackgroundWorker Started.
  Debug: New scope pushed.
  Debug: Registering integration: 'AutoSessionTrackingIntegration'.
  Debug: Registering integration: 'AppDomainUnhandledExceptionIntegration'.
  Debug: Registering integration: 'AppDomainProcessExitIntegration'.
  Debug: Registering integration: 'UnobservedTaskExceptionIntegration'.
  Debug: Registering integration: 'SentryDiagnosticListenerIntegration'.
  Debug: New scope pushed.
   Info: Replacing current logger with: 'MelDiagnosticLogger'.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: https://localhost:5092
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/jamescrosswell/code/sentry-dotnet/samples/Sentry.Samples.OpenTelemetry.AspNetCore
info: Sentry.ISentryClient[0]
      Sentry trace header is null. Creating new Sentry Propagation Context.
info: Sentry.ISentryClient[0]
      Envelope queued up: 'dda2b958555848daa8874b20301f6dc3'
info: Sentry.ISentryClient[0]
      Envelope 'dda2b958555848daa8874b20301f6dc3' successfully sent.

@leonid-lm
Copy link

Here is an example of such behavior, I'm limited to using the "Microsoft.Extensions.Hosting.Host" to ensure that I can configure the custom tracing context, that will clean up health check routes from all traces. Unfortunately, I get the same error. That's why my setup is another one than in your sample. If you can give me a hint on how could I make a customization for tracing context using WebApplication.CreateBuilder(args), I would be really glad... Thanks for your support.

using System;
using System.IO;
using System.Reflection;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Profile.Host.Config;
using Sentry;
using Sentry.OpenTelemetry;
using Serilog;
using Hosting = Microsoft.Extensions.Hosting.Host;

namespace Sample.Host
{
    public static class Program
    {
        private const string EnvVariableName = "ASPNETCORE_ENVIRONMENT";
        
        public static void Main(string[] args)
        {
            var envName = Environment.GetEnvironmentVariable(EnvVariableName);
            var settingsFileName = !string.IsNullOrEmpty(envName)
                ? $"appsettings.{envName}.json"
                : $"appsettings.json";
            IConfiguration configuration = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile(settingsFileName)
                .AddEnvironmentVariables()
                .Build();

            Log.Logger = new LoggerConfiguration()
                .ReadFrom.Configuration(configuration)
                .CreateLogger();

            BuildWebHost(args).Build().Run();
        }

        public static IHostBuilder BuildWebHost(string[] args)
        {
            var builder = Hosting.CreateDefaultBuilder(args);
            
            builder.ConfigureServices(t => t.AddOpenTelemetry()
                    .WithTracing(g => g
                    .AddSource(Startup.ServiceName)
                    .ConfigureResource(resource => resource.AddService(serviceName: Startup.ServiceName,
                        serviceVersion: Assembly.GetExecutingAssembly().GetName().Version?.ToString()))
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddConsoleExporter()
                    .AddSentry())
                .WithMetrics(m => m
                    .AddHttpClientInstrumentation()
                    .AddAspNetCoreInstrumentation()
                    .AddConsoleExporter()));
            
            builder.ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseSentry(options =>
                    {
                        options.TracesSampler = context =>
                        {
                            if (context.CustomSamplingContext.TryGetValue("__HttpPath", out var value))
                            {
                                string path = (string)value;
                                if (path!.Equals(Startup.HealthPath) || path.Equals(Startup.ReadinessPath) ||
                                    path.Equals(Startup.LivenessPath))
                                    return 0.0;
                            }

                            return options.TracesSampleRate;
                        };
                        options.UseOpenTelemetry();
                        options.AddEntityFramework();
                    });
                    webBuilder.UseStartup<Startup>();
                })
                .UseSerilog();
            
            return builder;
        }
    }
}

@leonid-lm
Copy link

leonid-lm commented Oct 20, 2023

Even when I migrated my code to the .Net 6.0 style. It still doesn't work (even with WebApplication.CreateBuilder(args)).

@jamescrosswell
Copy link
Collaborator

@leonid-lm again, would it be possible to get a copy of the debug output so that we can get some visibility into what is and what isn't working during SDK initialization? See my reply to @raulidavid above.

@shokkueibu
Copy link

I was having this issue and in my case it was an existing line in my project clearing the logging providers that was also clearing the SentryOptions.
Moving the UseSentry call below the logging configuration did the trick.

  (...)
  .ConfigureLogging(logging =>
  {
    logging.ClearProviders();
    logging.SetMinimumLevel(LogLevel.Trace);
  })
  .UseSentry(options =>
  {
    options.UseOpenTelemetry();
  })
  (...)

@leonid-lm
Copy link

I was able to identify the issue's root cause. For some reason existing initialization of the sentry goes into conflict with Serilog initialization. The current issue is easily reproducible if you just add the following code:

var builder = WebApplication.CreateBuilder(args);

var startup = new Startup(builder.Configuration);
startup.ConfigureServices(builder.Services);
builder.Services.AddOpenTelemetry()
    .WithTracing(traceBuilder =>
    {
        traceBuilder
            .AddSource(Startup.ServiceName)
            .ConfigureResource(resource => resource.AddService(serviceName: Startup.ServiceName,
                serviceVersion: Assembly.GetExecutingAssembly().GetName().Version?.ToString()))
            .AddAspNetCoreInstrumentation(option => option.RecordException = true)
            .AddHttpClientInstrumentation(option => option.RecordException = true)
            .AddSentry();
    });

builder.Host.UseSerilog((context, configuration) =>
    configuration.ReadFrom.Configuration(context.Configuration));

builder.WebHost.UseSentry(options =>
{
    options.Debug = builder.Environment.IsDevelopment();
    options.TracesSampler = context =>
    {
        if (context.CustomSamplingContext.TryGetValue("__HttpPath", out var value))
        {
            string path = (string)value;
            if (path!.Equals(Startup.HealthPath) || path.Equals(Startup.ReadinessPath) ||
                path.Equals(Startup.LivenessPath))
                return 0.0;
        }
        return options.TracesSampleRate;
    };
    options.AddEntityFramework();
     options.UseOpenTelemetry();
});

var app = builder.Build();
startup.Configure(app, app.Environment);

app.Run();

If you try to remove the following lines, it works:

builder.Host.UseSerilog((context, configuration) =>
configuration.ReadFrom.Configuration(context.Configuration));

So, if you would like to have the Sentry traces enabled, you need to get rid of Serilog.

@jamescrosswell
Copy link
Collaborator

jamescrosswell commented Nov 26, 2023

If you try to remove the following lines, it works:

builder.Host.UseSerilog((context, configuration) =>
configuration.ReadFrom.Configuration(context.Configuration));

So, if you would like to have the Sentry traces enabled, you need to get rid of Serilog.

It's a bit tricky to tell without seeing the config that is being passed to Serilog, but I suspect what's happening is that the call to UseSentry, under the hood, eventually resolves to this:

// The earliest we can hook the SDK initialization code with the framework
// Initialization happens at a later time depending if the default MEL backend is enabled or not.
// In case the logging backend was replaced, init happens later, at the StartupFilter
_ = builder.ConfigureLogging((context, logging) =>
{
logging.AddConfiguration();

Sentry's initialization is being delayed when the Microsoft Extension Logging backend has been replaced (by Serilog) and so when AddSentry is called on the trace builder, Sentry hasn't yet been initialized.

@leonid-lm are you able to share a sanitized version of the relevant bits of your serilog/sentry config?

@knd775
Copy link

knd775 commented Nov 27, 2023

...are you able to share a sanitized version of the relevant bits of your serilog/sentry config?

I'm having the same issue, and I've removed all Serilog config except just builder.Host.UseSerilog();

Here's my sentry config:

static void ConfigureSentry(WebApplicationBuilder builder)
{
    builder.Logging.AddOpenTelemetry(b =>
    {
        b.IncludeFormattedMessage = true;
        b.IncludeScopes = true;
        b.ParseStateValues = true;
        b.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("xxxxxx"));
    });

    builder.Services
        .AddOpenTelemetry()
        .WithTracing(b =>
        {
            b.AddSource("xxxxxx");
            b.ConfigureResource(r => r.AddService("xxxxxx"));
            b.AddAspNetCoreInstrumentation();
            b.AddHttpClientInstrumentation();
            b.AddHotChocolateInstrumentation();
            b.AddSentry(); // Removing this line "fixes" the issue
        }
    );

    builder.WebHost.UseSentry(o =>
    {
        o.Dsn = builder.Configuration.GetSetting("Sentry:Dsn");
        o.TracesSampleRate = 1.0;
        o.Environment = builder.Configuration.GetSetting("Sentry:Environment");
        o.UseOpenTelemetry();
        o.SetBeforeSendTransaction((sentryTransaction, hint) =>
        {
            // Filter out requests to BCP
            if (MatchBCPRegex().IsMatch(sentryTransaction.Name))
                return null;

            return sentryTransaction;
        });
    });
}

@jamescrosswell
Copy link
Collaborator

I'm having the same issue, and I've removed all Serilog config except just builder.Host.UseSerilog();

OK, progress. I'm able to replicate the same by adding this to Sentry.Samples.OpenTelemetry.AspNetCore.csproj

builder.Host.UseSerilog();

Now all that remains is to fix the problem 😜

@leonid-lm
Copy link

Thanks a lot... I had a really simple config for sentry and serilog:

  "Sentry": {
    "Dsn": "<LINK>",
    "SendDefaultPii": true,
    "IncludeActivityData": true,
    "AutoSessionTracking": true,
    "AttachStackTrace": true,
    "EnableTracing": true,
    "Debug": false,
    "DiagnosticLevel": "Error",
    "TracesSampleRate": 1.0,
    "Environment": "Integration"
  },
"Serilog": {
    "Using":  [ "Serilog.Sinks.Console", "Serilog.Sinks.Seq" ],
    "LevelSwitches": { "$controlSwitch": "Information" },
    "MinimumLevel": {
      "ControlledBy": "$controlSwitch",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning",
        "Hangfire": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "theme": "Serilog.Sinks.SystemConsole.Themes.AnsiConsoleTheme::Code, Serilog.Sinks.Console",
          "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} <s:{SourceContext}>{NewLine}{Exception}"
        }
      },
      {
        "Name": "Seq",
        "Args": {
          "serverUrl": "http://seq.logs.svc.cluster.local:5341",
          "apiKey": "<KEY>"
        }
      },
      {
        "Name": "Sentry",
        "Args": {
          "minimumBreadcrumbLevel": "Debug",
          "minimumEventLevel": "Error"
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId", "WithEventType" ],
    "Properties": {
      "Application": "app-api",
      "Environment": "Integration"
    }
  },

@jamescrosswell
Copy link
Collaborator

@knd775 and @leonid-lm are you able to try with version 3.41.3?

@knd775
Copy link

knd775 commented Nov 29, 2023

are you able to try with version 3.41.3?

Working for me! Thank you!

@jamescrosswell
Copy link
Collaborator

Great, I'll mark this as closed then. Any other issues I think we open a new ticket 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment