Proposal: Adding profiling as a support event type

[MovieStoreGuy]

Profiling events

There is a shifting concept that performance monitoring and application monitoring (the idea of tracking the time spent in functions and or methods, vs how long it takes to serve a request) are near identical and come under the realm of Observability (understanding how your service is performing).

How is this different from tracing

Conventional tracing looks at showing the user's request flow through the application to show time spent in different operations. However, this can miss any background operations that indirectly impact the user request flow.

ie. If I take a rate limiting service that has a background sync to share state among other nodes:

func ShouldRateLimit(next http.Handler) http.Handler {
   return http.HandlerFunc(w http.ResponseWriter, r *http.Request) {
         span, ctx := otel.SpanFromContext(r.Context())
         defer span.Finish()
         key, err := ratelimit.GetKey(r)
        
        if limits.Key(key).Exceed() {
             // return 429 status code
        }
        next.ServeHTTP(w,r)
   })
}

func (l *limits) SyncLimits() {
    l.cache.RLock()
    defer l.cache.RUnlock()
    for _, limit := limits.cache {
          // publish data to each node or distributed cache
          // Update internal values with shared updates
    }
}

In the above example, I can clearly see how the function ShouldRateLimit impacts the requests processing time considering the context used as part of the request can be used to link spans together but there is a hidden cost here with SyncLimits that currently can not be exposed due to the fact it runs independently from in bound requests and thus can not / should not share the same context.

Now, the SyncLimits function could implement metrics to help expose runtime performance issues but could be problematic due to:

• As a developer, I need to know what to start observing in order to diagnose
• The problem may disappear due to the nature of the issue (race conditions, Heisenbug)
• Measure performance of my function comparatively to my entire application
• Can not easily measure deadlocks / livelocks without elaborate code orchestration

Suggestion

At least within the golang community, https://github.com/google/pprof has been the leading tool in order to facilitate these kinds of questions while also offering first part support within Go. Moreover, AWS also have their own solution https://aws.amazon.com/codeguru/ that offers something similar for JVM based applications.

Desired outcomes of data:

• Show cumulative runtime of functions (could also derive percentage from this data)
• Map resource usage (CPU, Memory, and I/O) to internal methods / functions

Desired outcomes of orchestration:

• Low friction with adding profiling support (As an example, pprof adds a single handler to perform software based profiling)
• Should not require major modifications of existing code to work (should not require adding functions that would complicate existing logic)

I understand that software based profiling is not 100% accurate as per the write up here https://go.googlesource.com/proposal/+/refs/changes/08/219508/2/design/36821-perf-counter-pprof.md however, this could give an amazing insight into hidden application performance that could help increase reliability, performance and discover resource issues that were hard to discover with the existing events being emitted.

[jkwatson]

FYI, JFR is probably the top JVM profiling tool, as it's built-in to the JVM these days.

[MovieStoreGuy]

That is awesome to know @jkwatson :D I don't often work with JVM based languages but I will 100% have a look :D

[iNikem]

@jkwatson @MovieStoreGuy The top profiling tool for JVM is async-profiler :)

[jkwatson]

@jkwatson @MovieStoreGuy The top profiling tool for JVM is async-profiler :)

The docs on that are seriously out of date...they still reference JFR as a commercial product. I guess that's true if you're profiling java 7, but I don't wish that on anyone.

[iNikem]

What docs are out of date?

[jkwatson]

well, now I can't find the ones I was just looking at, so /shrug.
Also, this probably isn't the place to argue about specific profiling tools. :)

[MovieStoreGuy]

I agree with @jkwatson, I appreciate bring JVM tools to my intention, that is not the focus of this proposal :)

[rakyll]

We're interested in being able to collect CPU, memory, contention and other profiles with OpenTelemetry and have representations of profiles in OTLP and support in the collector. We are currently also looking into existing data model alternatives such as pprof as an option given its wide use in open source and language support.

We want to enable cases where we can use OpenTelemetry attributes to label profiles as well. pprof has support for labelling (an example can be seen at https://rakyll.org/profiler-labels/).

As of today, it's very difficult for our users to enable profiling at a later time, especially in production. They need to add CodeGuru Profiler libraries, rebuild and redeploy. As more and more of them are linking OpenTelemetry for other telemetry collection, we want to enable cases where we can enable profile collection dynamically in runtime. This use case will require the OpenTelemetry client libraries to speak to the collector (or another control plane) to enable/disable collection.

[thegreystone]

Not sure if this will help, but I thought I'd chip in with what we're doing at Datadog. For the continuous profiler (which is integrating with our tracer), we're using our own profiling libraries for most platforms, and our own agent using JFR on the JVM. For the JVM we've added our own profiling events for various different kinds of profiling (e.g. rate limited exception profiling). For non-JVM languages we're partly using pprof as the serialization format (some data doesn't fit well into the model, so it's currently an archive with multiple files in it). For the JVM we're using JFR for the serialization format.

There are a few interesting initiatives for JFR in recent and upcoming versions - such as a new allocation profiler in JDK 16, and much faster stack trace capturing (I believe JDK 17). We (Datadog), are also considering contributing an all new, full process, proper CPU profiler and some neat new capabilities allowing you to, for example, easily implement your own dynamic wall clock profiler.

[MovieStoreGuy]

It has been sometime since I have opened this, but I'd like to know how I could speed up anything that is required to make this part of the default otel offering.

[tedsuo]

Hi @MovieStoreGuy. We're pretty heads down getting metrics and logs completed, as well as expanding and improving library instrumentation. There probably will not be a lot of bandwidth from the current community until these components are stable; apologies in advance, it will probably be slow going. However, profiling is definitely top priority after metrics and logs!

If you, @thegreystone, and others are interested in contributing work towards this project, I would suggest the following steps, which any new signal would need to take:

1. Create a prototype in (ideally) two or three languages.
2. Write an OTEP with the proposed specification, based on those prototypes.

If there is a group willing to put in the time to prototype, we can help by creating a OTel SIG for this work (a repo plus a slack channel for discussion). But again, I'm concerned that the spec reviewers and language maintainers are fully committed, so there may not be a lot of bandwidth for review or assistance until we clear the deck. I hate saying "next year" but six months to complete metrics and the remaining current initiatives is probably realistic. If there are well thought out proposals and prototypes by then, it would definitely give this project a speed boost. :)

[aalexand]

We (owners of https://github.com/google/pprof repo) would be curious what it would take to standardize on the profile.proto as the wire format for profiling data in OTel.

@thegreystone RE "some data doesn't fit well into the model, so it's currently an archive with multiple files in it" - do you mind elaborating on that?

[alolita]

Is pprof being evaluated? It would be great to have a formal issue in the community repo. Ty!

[MovieStoreGuy]

Hey @alolita ,

Which community repo are you referring to?