Using ddtrace-run with Prefect: "Unable to serialize unknown type: <class 'uuid.UUID'>" / runs stuck on "Submitting"

[zzstoatzz]

this was drafted by claude at my direction

TL;DR — If flow runs hang on "Submitting" and your worker logs show
PydanticSerializationError: Unable to serialize unknown type: <class 'uuid.UUID'>,
the most likely cause is a zero-code APM/tracing agent monkeypatching your
process at import time — most commonly Datadog's ddtrace-run. Dropping the
wrapper (prefect flow-run execute instead of ddtrace-run prefect flow-run execute) resolves it. If you want tracing, Prefect is OpenTelemetry-native —
wire your spans through OTel/OTLP instead (details below).

Symptom

The run never leaves Submitting in the UI, and the flow process fails while trying
to advance its own state:

{"type":"PydanticSerializationError",
 "message":"Unable to serialize unknown type: <class 'uuid.UUID'>",
 "traceback":"... flow_engine.py ... begin_run ... set_state ... propose_state_sync ...
   _flow_runs/client.py ... set_flow_run_state ...
   state=state_create.model_dump(mode=\"json\", serialize_as_any=True) ..."}

Serializing the StateCreate payload fails, so the transition never reaches the API
and the run is stranded on Submitting.

Root cause

A pydantic defer_build interaction. Prefect's state schemas (StateDetails, etc.)
are built lazily, and the client serializes state with
model_dump(mode="json", serialize_as_any=True). If a nested model's serializer is
left in pydantic's incomplete (MockValSer) state, serialize_as_any=True falls
back to inference serialization, which then can't serialize the uuid.UUID fields
in StateDetails → Unable to serialize unknown type: <class 'uuid.UUID'>.

Prefect already guards against this on its own — it rebuilds these schemas at import
time (see #18053 /
#20127). The failure reappears
when something else perturbs pydantic's build state after Prefect's rebuild.

Two things make this sharper:

• serialize_as_any got stricter in pydantic 2.12. Cases that older pydantic
  tolerated now raise PydanticSerializationError: Unable to serialize unknown type
  (see pydantic #12348,
  #12379,
  #12382). Prefect currently
  allows pydantic 2.12, so an env on 2.12 is more exposed.
• Import-time monkeypatching by an APM agent can leave a nested serializer
  incomplete during the prefect flow-run execute subprocess startup.

This is a class of tools, not just ddtrace

ddtrace-run is the one we've seen reported, but the trigger is "zero-code agent
that wraps your entrypoint and rewrites/patches imports before your app runs."
Other tools in the same category that could interact the same way:

• Datadog — ddtrace-run (confirmed)
• OpenTelemetry — opentelemetry-instrument (also uses runtime monkeypatching)
• New Relic — newrelic-admin run-program / newrelic.agent.initialize()
• Elastic APM, Sentry auto-instrumentation, Dynatrace OneAgent, etc.

None of these are "wrong" — but wrapping the Prefect engine subprocess entrypoint
(prefect flow-run execute) with a global bootstrap is the risky part, because that's
exactly where the deferred pydantic schemas get built.

The better way: lean on Prefect's native OpenTelemetry support

You almost certainly reached for ddtrace-run to get traces of your flow runs.
You don't need to wrap the engine to get that — Prefect already speaks OpenTelemetry:

• It emits OTel spans for flow and task runs.
• It propagates W3C trace context across runs (that's the traceparent you'll
  notice on flow-run labels / StateDetails), so parent/child runs stitch together.

Because Prefect uses the global OpenTelemetry API, any OTel SDK + exporter you
configure in the environment picks up Prefect's spans. To land them in Datadog:

1. Configure an OTel SDK + OTLP exporter via standard env vars
   (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_EXPORTER=otlp, …).
2. Send OTLP to the Datadog Agent's OTLP ingest endpoint (or the OTel Collector
   with the Datadog exporter). See Datadog's
   OTLP ingestion
   docs.

This gives you Datadog traces (including Prefect's own flow/task spans) without
globally rewriting the engine process.

If you specifically want ddtrace instrumentation of your own libraries, prefer
scoped, programmatic patching inside your flow code
(import ddtrace; ddtrace.patch(requests=True, ...)) over the global ddtrace-run
wrapper around the engine entrypoint.

Workarounds (if you just need it working now)

- ddtrace-run prefect flow-run execute
+ prefect flow-run execute

• Drop the global APM wrapper from the flow-run entrypoint (reported to fix it).
• Or pin pydantic<2.12 while you migrate your tracing setup.

If you hit this

Please share your prefect version, your pydantic and APM-agent
(ddtrace/opentelemetry-*/newrelic/…) versions, and the exact command that
launches the flow-run process. A version-pinned reproduction would help us decide
whether Prefect can harden the serialization path further so it's robust even under
aggressive import-time monkeypatching.

[rehan243]

oh nice rundown on the ddtrace-run monkeypatching thing, that's a detail a lot of folks miss. I've hit that exact serialization error with uuid.UUID when Datadog's wrapper sneaks in early on some jobs, especially when mixing multiple tracing agents. The thing is, monkeypatching at import time nukes a bunch of assumptions about type handling, in my case, it also broke pydantic's handling of custom types (ran into MockValSer getting stuck, just like you described).

honestly, switching to OpenTelemetry with Prefect's native hooks was way smoother. I get traces into Grafana using otel-collector and didn't need any hacky wrappers. For anyone running flows at scale (I'm doing ~3000/day), the stuck-on-Submitting bug is brutal since you don't always notice until jobs pile up. Curious if you've seen similar issues with other APMs, or is this mainly Datadog? Also, are you tracing just the orchestrator or the workers too? I found worker-level spans give way more actionable stuff, but depends how noisy you want your traces.