OpenTelemetry against Jaeger works in Simple but not Batch

[jowliwon]

Have a pretty much working environment to use for tracing, OpenTelemetry against Jaeger Tracing. I sort of read that ExportProcessorType.Batch is preferred and more efficient then ExportProcessorType.Simple. How ever, when running in .NET Framework 4.8 application, ExportProcessorType.Batch dose not seem to give any results being logged. I did a data packet capture with Wireshark and found out data did not even get sent when running with ExportProcessorType.Batch.

Is there something with this configuration that is missing to have this as ExportProcessorType.Batch instead of ExportProcessorType.Simple? Or is it not even possible to run in Batch?

public TracerProvider GetTracerProvider(string host, int port)
    {           
        BackendServiceResource = ResourceBuilder.CreateDefault()
             .AddService(Process.GetCurrentProcess().ProcessName)               
             .AddAttributes(new[]
             {
                new KeyValuePair<string, object>("MachineName", Environment.MachineName),
                new KeyValuePair<string, object>("UserName", Environment.UserName),
             });

        return Sdk.CreateTracerProviderBuilder()
        .SetResourceBuilder(BackendServiceResource)
        .SetSampler(new AlwaysOnSampler())
        .SetErrorStatusOnException(true)
        .AddSource(ActivitySource.Name)
        .AddConsoleExporter()
        .AddJaegerExporter(jeager =>
        {
            jeager.AgentHost = host;
            jeager.AgentPort = port;

            jeager.MaxPayloadSizeInBytes = 4096;

            jeager.ExportProcessorType = ExportProcessorType.Batch;
            jeager.BatchExportProcessorOptions = new BatchExportProcessorOptions<Activity>()
            {
                MaxQueueSize = 2048,
                ScheduledDelayMilliseconds = 5000,
                ExporterTimeoutMilliseconds = 30000,
                MaxExportBatchSize = 512,
            };
        })
        .Build();
    }

[alanwest]

Is there something with this configuration that is missing to have this as ExportProcessorType.Batch instead of ExportProcessorType.Simple? Or is it not even possible to run in Batch?

Yes, it is possible. The Jaeger exporter defaults to ExportProcessorType.Batch.

We have a sample ASP.NET application targeting .NET Framework 4.8 which can be configured to use the Jaeger exporter using the batch processor. The exporter can be configured here (change console to jaeger):

opentelemetry-dotnet/examples/AspNet/Web.config

Lines 8 to 10 in 2df1a62

	<add key="UseExporter" value="console"/>
	<add key="JaegerHost" value="localhost"/>
	<add key="JaegerPort" value="6831"/>

We'd need a more complete repro in order to assist any further.

[jowliwon]

We have a sample ASP.NET application targeting .NET Framework 4.8 which can be configured to use the Jaeger exporter using the batch processor. The exporter can be configured here (change console to jaeger):

Well, I'm not running this as an ASP.NET application. Might that be part of the issue? I have a console application right now where I test this out, as the end goal for this is not to have it for web purposes, but for an .NET framework 4.8 based application.

We'd need a more complete repro in order to assist any further.

I'll see if I can include a more complete code to reproduce this.

[jowliwon]

I have this hooked up with Fody and MethodBoundaryAspect.Fody in the end. So you can probably ignore some of the interface referenced "On..." code that is found in the end of the OpenTelemetryAdapter.cs. How ever, this is what has been built to use OT to hook up with Jaeger. It loggs to console, but dose not try to send anything by the UDP port it has been assigned when set to Batch.

OpenTelemetryAdapter.cs -> OpenTelemetryEngine -> Startup.cs

OpenTelemetryAdapter.cs

using MethodBoundaryAspect.Fody.Attributes;
using OpenTelemetry.Trace;
using Debug.TraceLog.Adapter.OpenTelemetry.System;
using System;
using System.Diagnostics;

namespace Debug.TraceLog.Adapter.OpenTelemetry
{
    /// <summary>
    /// Adapter class with the intent to keep connection though OpenTelemetry and 
    /// extend upon the Activity class without involving the legacy Activity class.
    /// </summary>
    public class OpenTelemetryAdapter : ITraceLogHost.ITraceLogHost
    {
        /// <summary>
        /// Open telemetry engine
        /// </summary>
        #pragma warning disable IDE0052
        private OpenTelemetryEngine Engine { get; set; }
        #pragma warning restore IDE0052

        private static readonly Lazy<OpenTelemetryAdapter> instance = new Lazy<OpenTelemetryAdapter>(() => new OpenTelemetryAdapter());

        /// <summary>
        /// Instance of the engine of Open Telemetry
        /// </summary>
        public static OpenTelemetryAdapter Instance => instance.Value;

        /// <summary>
        /// Construct 
        /// </summary>
        private OpenTelemetryAdapter()
        {
            this.Engine = OpenTelemetryEngine.Instance;
        }

        /// <summary>
        /// Fires on start of execution
        /// </summary>
        /// <example>
        /// Make sure to start up with this line to keep track of what activity to use
        ///  Activity activity = arg.MethodExecutionTag as Activity;
        /// </example>
        /// <param name="arg">Arguments of this execution</param>
        public void OnEntry(MethodExecutionArgs arg)
        {
            //Activity activity = arg.MethodExecutionTag as Activity;

        }

        /// <summary>
        /// Fires on exit of execution
        /// </summary>
        /// <example>
        /// Make sure to start up with this line to keep track of what activity to use
        ///  Activity activity = arg.MethodExecutionTag as Activity;
        /// </example>
        /// <param name="arg">Arguments of this execution</param>
        public void OnExit(MethodExecutionArgs arg)
        {
           // Activity activity = arg.MethodExecutionTag as Activity;

        }

        /// <summary>
        /// Fires on all exceptions that are not handled by already implemented try-catch 
        /// </summary>
        /// <example>
        /// Make sure to start up with this line to keep track of what activity to use
        ///  Activity activity = arg.MethodExecutionTag as Activity;
        /// </example>
        /// <param name="arg">Arguments of this execution</param>
        public void OnException(MethodExecutionArgs arg)
        {
            Activity activity = arg.MethodExecutionTag as Activity;

            activity.SetStatus(Status.Error);
            activity.RecordException(arg.Exception);
        }
    }
}

OpenTelemetryEngine.cs

using OpenTelemetry.Trace;
using System;
using System.Diagnostics;

namespace Debug.TraceLog.Adapter.OpenTelemetry.System
{
    /// <summary>
    ///  Adapter class with the intent to keep connection with OpenTelementry and Jeager.
    /// </summary>
    public class OpenTelemetryEngine
    {
        private readonly string serveradress = "127.0.0.1";
        private readonly int serverPort = 6831;

        private readonly ActivitySource source = new ActivitySource("Sample.DistributedTracing", "1.0.0");

        private Startup Startup { get; set; }

        /// <summary>
        /// Instance of the tracer provider class of OpenTelemetry
        /// </summary>
        public TracerProvider Provider { get; }
        /// <summary>
        /// Instance of the OpenTelemetry tracer class
        /// </summary>
        public Tracer OpenTelemetryTracer { get; }

        private static readonly Lazy<OpenTelemetryEngine> instance = new Lazy<OpenTelemetryEngine>(() => new OpenTelemetryEngine());

        /// <summary>
        /// Instance of the engine of OpenTelemetry
        /// </summary>
        public static OpenTelemetryEngine Instance => instance.Value;

        /// <summary>
        /// Constructor starting up and accessing Tracing related objects and functions
        /// </summary>
        private OpenTelemetryEngine()
        {
            Startup = new Startup(source);
            Provider = Startup.GetTracerProvider(serveradress, serverPort);
            OpenTelemetryTracer = Provider.GetTracer(source.Name);            
        }

    }
}

Startup.cs

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using System.Diagnostics;
using System.Collections.Generic;
using System;

namespace Debug.TraceLog.Adapter.OpenTelemetry.System
{
    /// <summary>
    /// Starts up the OpenTelemetry and JeagerTracing connection
    /// </summary>
    public class Startup
    {
        private ActivitySource ActivitySource { get; }
        private ResourceBuilder BackendServiceResource { get; set; }

        /// <summary>
        /// Constructor
        /// </summary>
        /// <param name="source"></param>
        public Startup(ActivitySource source)
        {
            ActivitySource = source;
        }

        /// <summary>
        /// Gets the current trace provider, in our case Jaeger
        /// </summary>
        /// <param name="host">Server address for Jaeger</param>
        /// <param name="port">Server port for Jaeger</param>
        /// <returns>Returns trace provider</returns>
        public TracerProvider GetTracerProvider(string host, int port)
        {           
            BackendServiceResource = ResourceBuilder.CreateDefault()
                 .AddService(Process.GetCurrentProcess().ProcessName)               
                 .AddAttributes(new[]
                 {
                    new KeyValuePair<string, object>("MachineName", Environment.MachineName),
		            new KeyValuePair<string, object>("UserName", Environment.UserName),
                 });

            return Sdk.CreateTracerProviderBuilder()
            .SetResourceBuilder(BackendServiceResource)
            .SetSampler(new AlwaysOnSampler())
            .SetErrorStatusOnException(true)
            .AddSource(ActivitySource.Name)
            .AddConsoleExporter()
            .AddJaegerExporter(jeager =>
            {
                jeager.AgentHost = host;
                jeager.AgentPort = port;

                jeager.MaxPayloadSizeInBytes = 4096;

                jeager.ExportProcessorType = ExportProcessorType.Batch;
                jeager.BatchExportProcessorOptions = new BatchExportProcessorOptions<Activity>()
                {
                    MaxQueueSize = 2048,
                    ScheduledDelayMilliseconds = 5000,
                    ExporterTimeoutMilliseconds = 30000,
                    MaxExportBatchSize = 512,
                };
            })
            .Build();
        }
    }
}

[alanwest]

Is your application short running? It could be that not enough time passes for the batch to be sent. If this is the case there's a couple ways you can solve it. Either delay the termination of your application or disposing your tracer provider as I do in this minimal example:

using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;

namespace ConsoleApp1
{
    internal class Program
    {
        static void Main(string[] args)
        {
            using (var tracerProvider = Sdk.CreateTracerProviderBuilder()
                    .AddSource("MySource")

                    // Using all the default options. This will use the batch processor.
                    .AddJaegerExporter()
                    .Build())
            {
                using (var activitySource = new ActivitySource("MySource"))
                {
                    using (var activity = activitySource.StartActivity("Root"))
                    {
                        activity?.Stop();
                    }
                }
            }
        }
    }
}

[jowliwon]

Iv read your thought about being to short. Im not sure how it could be to short as there is a task delay of 500 ms.
Anyway, Iv made a stripped down version of the same OT Jaeger trace I'm running. This is a .NET Framework 4.8 console application. I don't have the possibility to run either Core or Standard when it comes to .NET for our project.
Anyway, should be able to reproduce and see what's going on with out much problem, simple copy-paste of what is bellow.
As it's still the same issue with nothing posting in Batch but works in Simple.

Im running with the latest OT 1.1.0 you get by NuGet in VS.

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace OTJaeger
{
    class Program
    {
        private static ResourceBuilder BackendServiceResource { get; set; }

        private static readonly ActivitySource ActivitySource = new ActivitySource("Sample.DistributedTracing");

        private static readonly string serveradress = "127.0.0.1";
        private static readonly int serverPort = 6831;

        public static TracerProvider Provider { get; set; }

        public static Tracer OpenTelemetryTracer { get; set; }

        static void Main(string[] args)
        {
            Provider = GetTracerProvider(serveradress, serverPort);
            OpenTelemetryTracer = Provider.GetTracer(ActivitySource.Name);

            MakeActivity("Test 1");
            MakeActivity("Test 2");
            MakeActivity("Test 3");
            MakeActivity("Test 4");
            MakeActivity("Test 5");

        }

        static void MakeActivity(string name)
        {
            Activity activity = ActivitySource.StartActivity(name);
            activity.AddTag("machine.name", Environment.MachineName);
            activity.AddTag("user.name", Environment.UserName);

            Task.Delay(500);

            activity.Stop();
        }

        /// <summary>
        /// Gets the current trace provider, in our case Jaeger
        /// </summary>
        /// <param name="host">Server address for Jaeger</param>
        /// <param name="port">Server port for Jaeger</param>
        /// <returns>Returns trace provider</returns>
        public static TracerProvider GetTracerProvider(string host, int port)
        {
            BackendServiceResource = ResourceBuilder.CreateDefault()
                 .AddService(Process.GetCurrentProcess().ProcessName)
                 .AddAttributes(new[]
                 {
                    new KeyValuePair<string, object>("MachineName", Environment.MachineName),
                    new KeyValuePair<string, object>("UserName", Environment.UserName),
                 });

            return Sdk.CreateTracerProviderBuilder()
            .SetResourceBuilder(BackendServiceResource)
            .SetSampler(new AlwaysOnSampler())
            .SetErrorStatusOnException(true)
            .AddSource(ActivitySource.Name)
            .AddConsoleExporter()
            .AddJaegerExporter(jeager =>
            {
                jeager.AgentHost = host;
                jeager.AgentPort = port;

                jeager.MaxPayloadSizeInBytes = 4096;

                jeager.ExportProcessorType = ExportProcessorType.Batch;
                jeager.BatchExportProcessorOptions = new BatchExportProcessorOptions<Activity>()
                {
                    MaxQueueSize = 2048,
                    ScheduledDelayMilliseconds = 5000,
                    ExporterTimeoutMilliseconds = 30000,
                    MaxExportBatchSize = 512,
                };
            })
            .Build();
        }


    }
}

[alanwest]

ScheduledDelayMilliseconds is set to 5000 milliseconds - this is the interval that batches will be sent. Your application only runs for ~2500 milliseconds. Not enough time for the batch processor to do its work.

Invoking Provider.Dispose() will cause the batch processor to flush prior to termination.

[jowliwon]

So, what dose this mean? Shorter ScheduledDelayMilliseconds will allow for more rapid response? Is it only the termination of the trace provider?

[jowliwon]

@alanwest your short example with the using works. How ever, I need to be able to use two methods for activities, one for starting and one for stopping. How would this be solved if disposing is the only issue here that stands in the way for batch to work properly?

[jowliwon]

@alanwest Iv understod now that by disposing the trace provider before the application terminates, everything will work as it should. Because you are right, this application is very fast and dose not get correct closure for the trace provider.

All that is left now is to see if this will work in a real case as well where the application that uses this process wont get terminated abruptly. But that's another chapter.

So I guess this question can be closed now as all questions about this so far are solved.

Thank you for your help and insight!

[alanwest]

Great, glad you got things working!