3.9.3

Release Notes

Bug Fixes

• Algoliasearch: Fix for potential dangling reference that could trhow an exception. -
• Tracing: Resolves a bug where os.system or subprocess.Popen could return the wrong exception type.
• AAP: This fix resolves an issue where the new ATO SDK track_user was reporting differently email, name, scope and role of the tracked user.

3.9.2

Bug Fixes

• This fix resolves an issue in which traced nested generator functions
  had their execution order subtly changed in a way that affected the
  stack unwinding sequence during exception handling. The issue was
  caused by the tracer's use of simple iteration via
  for v in g: yield v during the wrapping of generator functions where
  full bidrectional communication with the sub-generator via
  yield from g was appropriate. See PEP380 for an explanation of how
  these two generator uses differ.

3.9.1

Bug Fixes

• LLM Observability:
  • Fix an issue where the trace ID exported from export_span was incorrect.

3.9.0

New Features

• DSM
  • Add support for context extraction for SQS -> Lambda messsage events.
• Add support for Python 3.13 on Windows.
• azure_functions
  • Add distributed tracing support for http triggers.
• Code Security (IAST)
  • Unvalidated Redirect detection for Django, Flask and FastAPI applications, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Management documentation for more information about this feature.
• CI Visibility
  • This introduces report links to the pytest plugin. At the end of a test session, ddtrace shows links to the Datadog Test Optimization pages with the test results for the current commit and for the current CI job (provided that the CI environment variables with the current job and pipeline ID are available).
• AAP
  • This introduces the capability for the waf to decide of the sampling priority of the trace in case of a security event.
• litellm
  • Adds APM and LLM Observability tracing support for LiteLLM's synchronous and asynchronous completion and text_completion router methods.
• LLM Observability
  • add processor capability to process span inputs and outputs. See usage documentation [here](https://docs.datadoghq.com/llm_observability/setup/sdk/python/#span-processing).
  • Propagate the ml_app of the most recent LLM Observability span (or the global ml_app) when injecting distributed headers. In distributed services, uses the ml_app from the distributed trace headers.
  • This introduces tracing for system prompts in the OpenAI Agents SDK.
  • This introduces tracing for the content of tool call outputs passed to LLM spans for the OpenAI Agents integration.
• dynamic instrumentation
  • Add support for excluding identifiers from redaction with DD_DYNAMIC_INSTRUMENTATION_REDACTION_EXCLUDED_IDENTIFIERS

Upgrade Notes

• AAP
  • Upgrade the WAF value regex obfuscator.

Deprecation Notes

• tracing
  • Deprecates support for DD_TRACE_GLOBAL_TAGS in favor of DD_TAGS.
  • Removes support for APM Legacy App Analytics. There are still some remnants of App Analytics in the codebase, but they are not functional and output a deprecation warning.
• Deprecates starting the serverless mini agent from the tracer. Use the datadog-serverless-compat package instead.

Bug Fixes

• LLM Observability

  • This fix resolves an issue where error type was being set to the full error message for OpenAI Agents SDK errors, resulting in long error types.
  • This fix resolves an issue where LLM interactions were not being traced when a non-default base URL was provided for the Anthropic, Bedrock, LangChain, Open AI, and Azure Open AI integrations.
  • This fix resolves an issue where parsing token usage from langchain AI message types causes an attribute error.
  • Fixes an issue where using astream_events on a compiled graph would raise a KeyError.

• CI Visibility

  • This fix resolves an issue where the DD_CIVISIBILITY_ITR_ENABLED was not honored properly.
  • This fix resolves an issue where running from a GitHub action triggered on a tag push would cause the branch name to be null, causing errors when fetching Test Optimization settings from the backend.

• crewai

  • This fix resolves an issue where using crewai>=1.120.0 with ddtrace caused a TypeError to be thrown due to empty task contexts.

• tracing

  • Fixes a bug in distributed tracing where pickling ddtrace.trace.Context fails in coroutines. This regression was introduced in v3.7.0.
  • fix issue where Trace Agent sampling rates were not being applied when using a TraceFilter and tracer.configure.
  • Resolves the "sample_before_fork was unregistered without first being registered" warning by removing tracer at_exit hooks from the product protocol, ensuring hooks are registered and unregistered exactly once.
  • Fixes an issue where truncation of span attributes longer than 25000 characters would not consistently count the size of UTF-8 multibyte characters, leading to a unicode string is too large error.

• openai

  • Resolves an issue where streamed completions and chat completions immediately returning None would result in unfinished spans.

• dynamic instrumentation

  • fixes an issue where only module scopes were being uploaded, preventing method probes from being created.

• langgraph

  • Fixes an issue where using astream_events on a compiled graph would cause missing spans.

3.9.0rc2

Bug Fixes

• LLM Observability
  • This fix resolves an issue where LLM interactions
    were not being traced when a non-default base URL was provided for the
    Anthropic, Bedrock, LangChain, Open AI, and Azure Open AI
    integrations.

3.9.0rc1

New Features

• CI Visibility
  • This introduces report links to the pytest plugin. At the end of a test session, ddtrace shows links to the Datadog Test Optimization pages with the test results for the current commit and for the current CI job (provided that the CI environment variables with the current job and pipeline ID are available).
• azure_functions
  • This introduces tracing support for service bus triggers.

Bug Fixes

• AAP
  • Fixed API Security sampling when running in standalone mode (without APM tracing enabled).
• dynamic instrumentation
  • Fixed an issue with a GC-based lookup for code origin resolution resulting in high performance overhead.
• tracing
  • Fixes support for wrapping generator and async generator functions with tracer.wrap(). Previously, calling tracer.current_span() inside a wrapped generator function would return None, leading to AttributeError when interacting with the span. Additionally, traces reported to Datadog showed incorrect durations, as span context was not maintained across generator iteration. This change ensures that tracer.wrap() now correctly handles both sync and async generators by preserving the tracing context throughout their execution and finalizing spans correctly. Users can now safely use tracer.current_span() within generator functions and expect accurate trace reporting.
  • This fix resolves an issue where the library fails to decode a supported sampling mechanism, resulting in the log line: "failed to decode _dd.p.dm: ..."

Other Changes

• Adds a new configuration option DD_TRACE_SAFE_INSTRUMENTATION_ENABLED to enable safer patching of integrations.

  When enabled, the tracer will check if the installed version of an integration is compatible with the explicit version range supported by that integration. If an incompatible version is detected, the integration will not be patched and an error message will be logged.

  This feature is currently disabled by default, but will be enabled by default in a future version.

  Currently the only supported integrations for this feature are: aiobotocore (>=1.0.0), fastapi (>=0.57.0), and elasticsearch (>=1.10).

• profiling

  • redundant locking was removed from the memory profiler. This reduces the baseline overhead of the memory profiler substantially, and resolves several subtle correctness issues.

• docs

  • Update docs and tests for uwsgi --lazy-apps config. For uWSGI<2.0.30 when --lazy-apps is set , we advise our customers to also set --skip-atexit to avoid crashes that could occur from our native extensions when worker processes are terminated.

2.21.9

Bug Fixes

• CI Visibility:
  • Resolved an issue where the CODEOWNERS file of the project being tested was looked up in the current working directory rather than the repository root.
• Dynamic Instrumentation:
  • Fixed an issue with a GC-based lookup for code origin resolution resulting in high performance overhead.
• Kafka:
  • Fixed an issue where a producer or consumer initialized with an unpacked config resulted in TypeError, causing a failed connection. confluent-kafka supports both unpacked and packed config; this change allows initialization with either.
• Profiling:
  • Fixed an issue in the SynchronizedSamplePool where pool could be null when calling into ddog_ArrayQueue_ functions, leading to segfaults in the uWSGI shutdown
• Tracing:
  • Fixed a bug in the sampling rule matcher where the pattern ?* was not being matched correctly for DD_TRACE_SAMPLING_RULES tags, due to it matching on spans with no tag matching the specified key.

3.8.1

Bug Fixes

• CI Visibility:
  • Fix resolves an issue where running from a GitHub
    action triggered on a tag push would cause the branch name to be null,
    causing errors when fetching Test Optimization settings from the
    backend.

• Dynamic Instrumentation:
  • Fix an issue where only module scopes were
    being uploaded, preventing method probes from being created.

3.8.0

New Features

• LLM Observability: add processor capability to process span inputs and outputs. See usage documentation here.
• CI Visibility: This introduces the ability to gzip the payload when using the evp proxy setup, incurring in less network bandwith consumption.
• Error Tracking: Introduces automatic reporting of handled exceptions. Enabling the feature will report handled exceptions to Error Tracking from the user code, the third party packages code, some specified modules or everything based on configuration. This feature can be controlled using two environment variables:
  • DD_ERROR_TRACKING_HANDLED_ERRORS=allthird_party
  • DD_ERROR_TRACKING_HANDLED_ERRORS_INCLUDE=module1, module2, module3.submodule
• Code Security: IAST support for langchain v0.1.0 and above.
• openai: This introduces tracing support for the OpenAI Responses endpoint.

Bug Fixes

• tracing

  • Fixes an issue where truncation of span attributes longer than 25000 characters would not consistently count the size of UTF-8 multibyte characters, leading to a unicode string is too large error.
  • Fixes a bug in distributed tracing where pickling ddtrace.trace.Context fails in coroutines. This regression was introduced in v3.7.0.

• CI Visibility

  • This fix resolves an issue where the DD_CIVISIBILITY_ITR_ENABLED was not honored properly.
  • This fix resolves an issue where pytest-xdist would not exit with the proper status code if ATR was enabled.
  • This fix resolves an issue where ddtrace pytest plugin used with xdist would report test suites as failing even when all tests pass.

• profiling

  • fixed an issue in the SynchronizedSamplePool where pool could be null when calling into ddog_ArrayQueue_ functions, leading to segfaults in the uWSGI shutdown
  • improve performance of the memory profiler for large heaps. The memory profiler previously did a linear search of tracked allocations for every free, which scaled very poorly with large heaps. Switch to a fast hash map.

• Code Security

  • IAST
    • Avoid excessive filtering of stacktrace locations when finding vulnerabilities. After this change, vulnerabilities that were previously discarded will now be reported. In particular, if they were found within code in site-packages or outside of the working directory.

• LLM Observability

  • Resolves an issue where spans and evaluation metrics were not being sent via Unix sockets.

• dynamic instrumentation

  • prevent an exception when trying to remove a probe that did not resolve to a valid source code location.

• kafka

  • This fix resolves an issue where message headers were sent to Kafka brokers that do not support them. Message headers are turned off when the Kafka server responds with UNKNOWN_SERVER_ERROR (-1).

• code origin for spans

  • fixes a performance issue with exit spans.

3.8.0rc3

New Features

• LLM Observability: add processor capability to process span inputs and outputs. See usage documentation [here](https://docs.datadoghq.com/llm_observability/setup/sdk/python/#span-processing).

Bug Fixes

• tracing: Fixes an issue where truncation of span attributes longer than 25000 characters would not consistently count the size of UTF-8 multibyte characters, leading to a unicode string is too large error.