fix several async transport issues

[mobuchowski]

Summary

This fixes 3 bugs in the Datadog transport that caused Airflow scheduler CPU to climb until it hit its limit (issue #4683). The transport built an async HTTP worker thread for every instance, even when nothing routed to it, and that thread polled constantly even when idle. A separate bug also leaked memory when a run's START event failed.

What was wrong

DatadogTransport built an AsyncHttpTransport on every construction, whether or not any event ever used it. Building AsyncHttpTransport starts a daemon thread. With the default routing rules, most events never use the async path, so most of these threads did nothing but consume CPU.

Each idle thread polled its queue every 0.01 seconds, forever. On a production scheduler that built one client per event, this added up: 50 instances used 29.5% CPU while idle, 400 instances used 167.6%.

A third bug caused a memory leak: when a run's START event failed after all retries, any completion events already queued for that run stayed in memory. Nothing released them.

What changed

We made 3 changes to client/python/src/openlineage/client/transport/:

• datadog.py now builds the async transport only when an event actually needs it, not on every construction. Most deployments will never build it at all.
• async_http.py now backs off when idle. The worker thread sleeps for 0.01 seconds after the queue is empty, then doubles that wait each time it stays empty, up to 1 second. It resets to 0.01 seconds as soon as new work arrives.
• async_http.py now clears out a run's pending completion events if its START event fails after all retries, and logs a warning when it does. Before this change, those events stayed in memory indefinitely.

Checklist

• AI was used in creating this PR

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.72%. Comparing base (b54853f) to head (0a336a7).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4694   +/-   ##
=======================================
  Coverage   73.72%   73.72%           
=======================================
  Files          21       21           
  Lines        2356     2356           
=======================================
  Hits         1737     1737           
  Misses        619      619           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.