New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow the OpenTelemetry tracer to be used with the profiler #3819
Comments
@phillipuniverse Did you find any way to workaround this? (we are looking to do exactly the same) |
@shahargl not yet but I hope to make some investments in this area within the next month! |
Potentially relevant is #4170 where the DD tracer now has knowledge of the OpenTelemetry |
Looks like #4170 was reverted, base issue at #3872. I went down a pretty long and involved process trying to build a bridge between the OpenTelemetry tracer and the Datadog tracer. I couldn't quite get the stack profiles to work though and I think in general it's just too shaky. I'm going to try a different approach where I do the following:
This has other downsides though. I would like to avoid the Datadog instrumentation to not worry about disabling exports of spans, but if I don't do instrumentation I will have to manually establish the tracer in Django/FastAPI/pika/Celery/etc. entrypoints in my services. I need to think more about this. At any rate, here is a gist to my sort-of working code. Profiles do show up but are not linked to spans so it doesn't truly work right. |
Below is a lot of research that I did into this which has been dead-ends so far. At this point I would need some additional advice from Datadog folks in what exactly needs to be set where in the internals of the Datadog tracer to get things synced correctly between DD and otel. I'm going to pause work on this until I get additional feedback. Summary of the issues below:
I think probably the final conclusion of this is that the bridge will work like the existing opentracing implementattion. @Kyle-Verhoog it looks like you had were one of the initial drivers there, got some pointers for important pieces of that bridge that could apply to OpenTelemetry support? Specifically, what needs to be synced on the Datadog side? A couple of other thoughts:
I have an update but it's not really good news. I started going down the path of using the Datadog instrumentation and disabling export. I found a good workaround in this issue for how to disable the trace writer which works great! But it's still not quite giving me what I want because I still have to sync the Datadog Span/Trace with the ids of the otel Span/Trace. Everything has to line up for profiler events to be associated in the APM. I couldn't find a great way to do that but maybe something like monkeypatching the So back to my gist. I think I have a better bridge between otel and dd through the OpenTelemetry context api. That is a low-level function that is truly activates/de-activates the span. My goal with this is to take the otel context api and use that to sync in changes to the DD context api. This works slightly better. In the gist I include a monkeypatch to the DD
Another weird thing is that it is constantly ping-ponging between span ids well after the request is over, like it's continuing to get profile events for a previous span even after that span is done. My guess is that I need to be stopping/detaching something but I don't know where. Then I went and tried to validate what happens without OpenTelemetry and just manually patch. So I included this: config.env = ENVIRONMENT # the environment the application is in
config.service = SERVICE # name of your application
config.version = VERSION # version of your application
patch(django=True) And voila, I got span ids and they were all hooked up into the Datadog APM correctly. I also only see this logging when I would expect, in the context of an API request, not persistently like before.
|
@phillipuniverse sorry for the long delay responding here, would you be willing to send me an email at brett.langdon@datadoghq.com I'd love to try and schedule a call with you to discuss this topic! |
Whoops! The new OpenTelemetry integration does not yet link profiles with otel spans. |
Manually doing what the Github Action was supposed to do (fixed in #7835): This issue has been automatically closed after six months of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error. |
Which version of dd-trace-py are you using?
1.1.3
Which version of pip are you using?
22.0.3
Which version of the libraries are you using?
OpenTelemetry 1.12.0rc1
How can we reproduce your problem?
I use OpenTelemetry for all span/trace instrumentation. I would like to use the traces/spans from OpenTelemetry to give context to the ddtrace Profiler. Here's an example of what I would like to do:
Alternatively, is there a bridge maybe for the default Datadog tracer from
ddtrace.tracer
to simply inherit the trace/span information from OpenTelemetry?The text was updated successfully, but these errors were encountered: