FastAPI trace root not being carried in context with other spans because of certain dependency

[emhagman]

Summary of problem

When installing dramatiq[watch] specifically, ddtrace no longer carries context of root trace.

This does not happen if the dramatiq[watch] feature flag is installed so it's fixable for now by separating local vs production requirements.txt but it seems like a larger issue worth looking into if just having a certain dependency installed and imported breaks things.

Which version of dd-trace-py are you using?

2.5.2

Which version of pip are you using?

23.3.1

Which libraries and their versions are you using?

`pip freeze`

annotated-types==0.6.0 anyio==4.2.0 async-timeout==4.0.3 attrs==23.2.0 bytecode==0.15.1 cattrs==23.2.3 click==8.1.7 ddsketch==2.0.4 ddtrace==2.5.2 Deprecated==1.2.14 dramatiq==1.16.0 envier==0.5.1 exceptiongroup==1.2.0 fastapi==0.99.1 gevent==23.9.1 greenlet==3.0.3 h11==0.14.0 idna==3.6 importlib-metadata==6.11.0 opentelemetry-api==1.22.0 prometheus-client==0.19.0 protobuf==4.25.2 pydantic==1.10.14 pydantic_core==2.16.1 redis==5.0.1 six==1.16.0 sniffio==1.3.0 starlette==0.27.0 typing_extensions==4.9.0 uvicorn==0.27.0.post1 watchdog==3.0.0 watchdog-gevent==0.1.1 wrapt==1.16.0 xmltodict==0.13.0 zipp==3.17.0 zope.event==5.0 zope.interface==6.1

How can we reproduce your problem?

Note, I used patch_all here but we use ddtrace-run in production and can reproduce the same exact issue no matter how things are patched.

from fastapi import FastAPI
from ddtrace import tracer
import uvicorn
import os

from ddtrace import patch_all
patch_all()

os.environ["DD_ENV"] = "dev"

tracer.configure(enabled=True, hostname="localhost")

import dramatiq # remove this line and tracing works

app = FastAPI()

@app.get("/test")
def test():
    with tracer.trace("test_span") as span:
        span.set_tag("test_tag", "test_value")
        return {"test": "test"}


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

What is the result that you get?

Without import dramatiq and dramatiq[watch] or with just dramatiq[redis] and no watch feature

With import dramatiq and dramatiq[watch]

What is the result that you expected?

The context should be carried over with the span not starting a new trace.

MORE CONTEXT

To help debug more, adding the following line AFTER THE IMPORTS:

tracer.configure(context_provider=DefaultContextProvider())

... makes the tracing tied together again. I believe this has something to do with contextvars since both libraries use them but I could be wrong.

It also may be related to one of these libraries since that is the diff between dramatiq[redis] and dramatiq[redis,watch]:

problematic libraries

gevent==23.9.1
greenlet==3.0.3
watchdog==3.0.0
watchdog-gevent==0.1.1
zope.event==5.0
zope.interface==6.1

dramatiq[redis] pip freeze

async-timeout==4.0.3
dramatiq==1.16.0
prometheus-client==0.19.0
redis==5.0.1

dramatiq[redis,watch] pip freeze

async-timeout==4.0.3
dramatiq==1.16.0
gevent==23.9.1
greenlet==3.0.3
prometheus-client==0.19.0
redis==5.0.1
watchdog==3.0.0
watchdog-gevent==0.1.1
zope.event==5.0
zope.interface==6.1

[emmettbutler]

Thank you for the report, @emhagman. We'll look into it. If you're up for it, you could help the investigation by creating a Dockerfile that can be used to replicate the issue.

[emhagman]

Thank you for the report, @emhagman. We'll look into it. If you're up for it, you could help the investigation by creating a Dockerfile that can be used to replicate the issue.

Yeah sure thing, can do.

[github-actions]

This issue has been automatically closed after a period of inactivity. If it's a
feature request, it has been added to the maintainers' internal backlog and will be
included in an upcoming round of feature prioritization. Please comment or reopen
if you think this issue was closed in error.