Manually send custom metrics (Agent API or integration with a Metrics Library)

[gregkalapos]

We should offer users the ability to manually send metrics.

I think there would be 2 approaches:

• Integrate with an existing metrics library (e.g. AppMetrics, Metrics.NET)
• Extend the public agent API

1. step: decide which approach to take
2. step: implement.

[oleh-zheleznyak]

For my project having custom metrics is really important, and it is also available in other APM products on the market.
It would be really awesome to have this!

From a consumers point of view, in a ASP .NET Core application, I would expect to be able to register a source of IMetricsProvider (it could be a list, a factory, etc) with my DI container by using a simple extension method on IServiceCollection for example services.AddElasticApmMetrics(IReadOnlyCollection<IMetricsProvider> metricProviders)
(That collection/factory could later be injected into a new MetricsCollector class ctor)

Without custom metrics key stakeholders in our company will most likely choose another product.

Thanks a lot for the great effort you're doing with APM agent for dotnet!

[andrewharry]

Could this be done using Agent.PayloadSender.QueueMetrics(IMetricSet metrics);

[andrewharry]

Otherwise the IMetricsCollector.MetricsProviders needs to be exposed via a public API

[andrewharry]

https://github.com/workbetter-com-au/apm-agent-dotnet-metrics
Attempted a quick manual 'send metric'. Couldn't get it to work. No errors were logged.
I will pull into the project the actual Elastic.Apm to troubleshoot why it didn't work

[gregkalapos]

Couldn't get it to work. No errors were logged.

Let me try to help.

The agent currently sends data to the APM Server when one of these 2 things happen:

• A transaction ends (Transaction.End() is called either manually or by auto instrumentation)
• There are at least 20 events recorded (1 event is e.g. a span, or a metricset with 1 timespan)

In your sample you only record metrics, so probably that's the issue - you need at least 20 of those to make the agent send it to the server. You can either force it by also capturing the transaction (just turn on the agent normally with UseElasticApm) - or simply change the code temporarily - or send at least 20 metrics to fill the queue.

Probably we should add a config setting for this... that'd be a nicer way of doing this.

Also I'd like to point out that in PayloadSenderV2 we already do the big part of the work (serialization and sending to the server) on a separate thread, the payloadSender.QueueMetrics(metrics) already returns very fast. I'm not sure your BatchWorker is needed.

Hope this helps.

[andrewharry]

I think I have identified the issue
https://github.com/elastic/apm-agent-dotnet/blob/master/src/Elastic.Apm/Report/PayloadSenderV2.cs#L151

The payloadsender specifically checks if the metric is of type Metrics.MetricSet
where as I don't have access to the metricset class (Internal)

If this was changed to check for the Interface instead - IMetricSet - Maybe it would work then?

[andrewharry]

I wasn't too familiar with the internal workings of the payloadsender. I was reproducing the MetricCollector behaviour with the BatchWorker pattern. In general, I'm eager for the internal classes to be opened up a lot more :) - But I get the logic of not exposing potentially breaking changes early in the process

[gregkalapos]

If this was changed to check for the Interface instead - IMetricSet - Maybe it would work then?

We should rather make MetricSet public - at least that was my plan, its just a plain C# type to transfer data, I think it's a good candidate to use it on the API surface.

[andrewharry]

That would be ideal. I also hit issues with the TraceParent being internal. And the internal constructor on the DistributedTracingData object

[gregkalapos]

I also hit issues with the TraceParent being internal. And the internal constructor on the DistributedTracingData object

What usecase was that?

[andrewharry]

I also hit issues with the TraceParent being internal. And the internal constructor on the DistributedTracingData object

What usecase was that?

I have put together a project in .NET Framework showing how I have integrated with the Elastic.Apm dotnet client
https://github.com/workbetter-com-au/apm-agent-dotnet-framework

[SergeyKleyman]

@andrewharry Thank you - I will definitely take a look at your implementation of the integrations and consider if and how to make them available with the agent out of the box.

[marcoregueira]

I'm closing #708. As suggested this seems a better place for the matter. Just for the record, my suggestion was to allow registration of more (and custom) IMetricsProvider. The OTB functionality is great, but there is no need in hitting you when somebody wants this or that, counter, or even a custom one; I found they're pretty easy to implement if they can be added to the queue...

For me having a pluggable or custom PayloadSenderV2 would be interesting too. I would like to route some of the events to some other system or even add new ones (such as log entries) to the switch in ProcessQueue. (I know that's out of scope for you, as it is paired with the APM service ingestion, but having it open would not harm). Well, by now I have a fork of the agent I can live with, but asking is free :)))

Thank you

[glucaci]

I did a PR to have access to MetricSet and looking forward to write a Reporter for Metrics.NET.

In the same time I thought maybe will be a good idea to create a separate package for Metrics, Elastic.Apm.Metrics and maybe also for Server communication Elastic.Apm.Reporter (IPayloadSender). The classes are already splitted on folder level but will be nice to have separate nuget packages for 3rd party integrators libraries.

[Aaronontheweb]

Any update on this? We'd be interested in contributing App.Metrics support for Elastic.Apm but it looks like this isn't feasible today.

[kbalys]

@gregkalapos - any update on this issue? I am really looking forward to this feature as I would like to send multiple custom metrics using APM.

[Matiszak]

Since kibana apm seems to have support for opentelemetry protocol I think you can try opentelemetry dotnet client which supports metrics in beta release and setup otpl protocol exporter to kibana apm.

[fuzolan]

It would be nice to have support for https://docs.microsoft.com/en-us/dotnet/core/diagnostics/event-counters

[BenWolstencroft]

Is anyone working on this feature?

[smarandav]

Is there a demo project that can be used to send a custom metric to the APM Server?

[kbalys]

@gregkalapos are there plans in the nearest future to implement this feature?

[kbalys]

@gregkalapos, could you share updates or plans regarding this feature?

[johanknirk]

I'm also curious if you could get this working by using .net telemetry exporter
https://opentelemetry.io/docs/instrumentation/net/exporters/

to setup exporting to OpenTelemetry intake API:
https://www.elastic.co/guide/en/apm/guide/current/api-otlp.html

Anybody tested this ?