From 1f60ac62dc1ba0b7952a8ced820f7014c94311a4 Mon Sep 17 00:00:00 2001 From: "Amit D." Date: Fri, 3 Oct 2025 10:44:04 -0700 Subject: [PATCH 1/5] Adding Inital Metrics --- .../llm_observability/monitoring/metrics.md | 80 +++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 content/en/llm_observability/monitoring/metrics.md diff --git a/content/en/llm_observability/monitoring/metrics.md b/content/en/llm_observability/monitoring/metrics.md new file mode 100644 index 0000000000000..66d2cb6020bf0 --- /dev/null +++ b/content/en/llm_observability/monitoring/metrics.md @@ -0,0 +1,80 @@ +--- +title: LLM Observability Metrics +description: 'Learn about useful metrics you can generate from LLM Observability data.' +further_reading: + - link: 'llm_observability/' + tag: "Documentation" + text: 'Learn more about LLM Observability' + - link: 'monitors/' + tag: "Documentation" + text: 'Create and manage monitors to notify your teams when it matters.' +--- + +## LLM Observability metrics + +LLM Observability metrics are collected after enabling LLM Observability and instrumenting your application. These metrics are available for dashboards and monitors. + +These metrics capture **span** counts, **error** counts, **token usage**, and **latency** measures for your LLM applications. They are calculated based on 100% of the application's traffic. + +**Note**: Other tags set on spans are not available as tags on LLM Observability metrics. + +### Span metrics + +| Metric Name | Description | Metric Type | Tags | +|-------------|-------------|-------------|------| +| `ml_obs.span` | Total amount of spans with a span kind | Count | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `span_kind`, `version` | +| `ml_obs.span.duration` | Total duration of spans in seconds | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `span_kind`, `version` | +| `ml_obs.span.error` | Number of errors that occurred in the span | Count | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `span_kind`, `version` | + +### LLM token metrics + +| Metric Name | Description | Metric Type | Tags | +|-------------|-------------|-------------|------| +| `ml_obs.span.llm.input.tokens` | Number of tokens in the input sent to the LLM | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `version` | +| `ml_obs.span.llm.output.tokens` | Number of tokens in the output | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `version` | +| `ml_obs.span.llm.prompt.tokens` | Number of tokens used in the prompt | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `version` | +| `ml_obs.span.llm.completion.tokens` | Tokens generated as a completion during the span | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `version` | +| `ml_obs.span.llm.total.tokens` | Total tokens consumed during the span (input + output + prompt) | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `version` | +| `ml_obs.span.llm.input.characters` | Number of characters in the input sent to the LLM | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `version` | +| `ml_obs.span.llm.output.characters` | Number of characters in the output | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `version` | + +### Embedding metrics + +| Metric Name | Description | Metric Type | Tags | +|-------------|-------------|-------------|------| +| `ml_obs.span.embedding.input.tokens` | Number of input tokens used for generating an embedding | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `version` | + +### Trace metrics + +| Metric Name | Description | Metric Type | Tags | +|-------------|-------------|-------------|------| +| `ml_obs.trace` | A collection of spans representing an end-to-end request or workflow | Count | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | +| `ml_obs.trace.duration` | Total duration of all traces across all spans | Distribution | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | +| `ml_obs.trace.error` | Indicator or description of an error that occurred during the trace | Count | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | + +### Estimated usage metrics + +| Metric Name | Description | Metric Type | Tags | +|-------------|-------------|-------------|------| +| `ml_obs.estimated_usage.llm.input.tokens` | Estimated number of input tokens used | Distribution | `evaluation_name`, `ml_app`, `model_name`, `model_provider`, `model_server` | + +### Deprecated metrics + +The following metrics have been deprecated: + +| Metric Name | Description | Metric Type | Tags | +|-------------|-------------|-------------|------| +| `ml_obs.estimated_usage.llm.output.tokens` | Estimated number of output tokens generated | Distribution | `evaluation_name`, `ml_app`, `model_name`, `model_provider`, `model_server` | +| `ml_obs.estimated_usage.llm.total.tokens` | Total estimated tokens (input + output) used | Distribution | `evaluation_name`, `ml_app`, `model_name`, `model_provider`, `model_server` | + +## Next steps + +{{< whatsnext desc="Use what you set up:" >}} + {{< nextlink href="dashboards/" >}}Create a Dashboard to track and correlate LLM Observability metrics{{< /nextlink >}} + {{< nextlink href="monitors/create/" >}}Create Monitors that alert and notify you when something is unexpected{{< /nextlink >}} +{{< /whatsnext >}} + + +## Further Reading + +{{< partial name="whats-next/whats-next.html" >}} \ No newline at end of file From e6ce2b994f58d824b87ab0d6aa603cca5d0ecad6 Mon Sep 17 00:00:00 2001 From: "Amit D." Date: Fri, 3 Oct 2025 11:18:23 -0700 Subject: [PATCH 2/5] Added Warning and to Sidebar --- config/_default/menus/main.en.yaml | 5 +++++ content/en/llm_observability/monitoring/metrics.md | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index c832a5db7ede8..41cc83fb0a78d 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -4760,6 +4760,11 @@ menu: parent: llm_obs_monitoring identifier: llm_obs_agent_monitoring weight: 304 + - name: Metrics + url: llm_observability/monitoring/metrics + parent: llm_obs_monitoring + identifier: llm_obs_monitoring_metrics + weight: 305 - name: Evaluations url: llm_observability/evaluations/ parent: llm_obs diff --git a/content/en/llm_observability/monitoring/metrics.md b/content/en/llm_observability/monitoring/metrics.md index 66d2cb6020bf0..e8d099b5e4406 100644 --- a/content/en/llm_observability/monitoring/metrics.md +++ b/content/en/llm_observability/monitoring/metrics.md @@ -60,6 +60,10 @@ These metrics capture **span** counts, **error** counts, **token usage**, and ** ### Deprecated metrics +
+Important: The following metrics have been deprecated and are maintained for backward compatibility only. Datadog strongly recommends using the current token metrics instead for all token usage measurement use cases. +
+ The following metrics have been deprecated: | Metric Name | Description | Metric Type | Tags | From 92ebfa002d59e51d31cdde1d1cb9ae00e636a4bc Mon Sep 17 00:00:00 2001 From: cecilia saixue watt Date: Fri, 3 Oct 2025 11:57:47 -0700 Subject: [PATCH 3/5] syntax edits --- .../llm_observability/monitoring/metrics.md | 20 +++++++------------ 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/content/en/llm_observability/monitoring/metrics.md b/content/en/llm_observability/monitoring/metrics.md index e8d099b5e4406..178335b6d1d87 100644 --- a/content/en/llm_observability/monitoring/metrics.md +++ b/content/en/llm_observability/monitoring/metrics.md @@ -10,19 +10,15 @@ further_reading: text: 'Create and manage monitors to notify your teams when it matters.' --- -## LLM Observability metrics +After you instrument your application with LLM Observability, you can access LLM Observability metrics for use in dashboards and monitors. These metrics capture span counts, error counts, token usage, and latency measures for your LLM applications. These metrics are calculated based on 100% of the application's traffic. -LLM Observability metrics are collected after enabling LLM Observability and instrumenting your application. These metrics are available for dashboards and monitors. - -These metrics capture **span** counts, **error** counts, **token usage**, and **latency** measures for your LLM applications. They are calculated based on 100% of the application's traffic. - -**Note**: Other tags set on spans are not available as tags on LLM Observability metrics. +
Other tags set on spans are not available as tags on LLM Observability metrics.
### Span metrics | Metric Name | Description | Metric Type | Tags | |-------------|-------------|-------------|------| -| `ml_obs.span` | Total amount of spans with a span kind | Count | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `span_kind`, `version` | +| `ml_obs.span` | Total number of spans with a span kind | Count | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `span_kind`, `version` | | `ml_obs.span.duration` | Total duration of spans in seconds | Distribution | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `span_kind`, `version` | | `ml_obs.span.error` | Number of errors that occurred in the span | Count | `env`, `error`, `ml_app`, `model_name`, `model_provider`, `service`, `span_kind`, `version` | @@ -61,11 +57,9 @@ These metrics capture **span** counts, **error** counts, **token usage**, and ** ### Deprecated metrics
-Important: The following metrics have been deprecated and are maintained for backward compatibility only. Datadog strongly recommends using the current token metrics instead for all token usage measurement use cases. +The following metrics are deprecated, and are maintained only for backward compatibility. Datadog strongly recommends using non-deprecated token metrics for all token usage measurement use cases.
-The following metrics have been deprecated: - | Metric Name | Description | Metric Type | Tags | |-------------|-------------|-------------|------| | `ml_obs.estimated_usage.llm.output.tokens` | Estimated number of output tokens generated | Distribution | `evaluation_name`, `ml_app`, `model_name`, `model_provider`, `model_server` | @@ -73,9 +67,9 @@ The following metrics have been deprecated: ## Next steps -{{< whatsnext desc="Use what you set up:" >}} - {{< nextlink href="dashboards/" >}}Create a Dashboard to track and correlate LLM Observability metrics{{< /nextlink >}} - {{< nextlink href="monitors/create/" >}}Create Monitors that alert and notify you when something is unexpected{{< /nextlink >}} +{{< whatsnext desc="Make use of your LLM Observability metrics:" >}} + {{< nextlink href="dashboards/" >}}Create a dashboard to track and correlate LLM Observability metrics{{< /nextlink >}} + {{< nextlink href="monitors/create/" >}}Create a monitor for alerts and notifications{{< /nextlink >}} {{< /whatsnext >}} From 05035b416a313ef4bcb451f537d31de0369ad20c Mon Sep 17 00:00:00 2001 From: "Amit D." Date: Fri, 3 Oct 2025 12:25:01 -0700 Subject: [PATCH 4/5] Update content/en/llm_observability/monitoring/metrics.md Co-authored-by: cecilia saixue watt --- content/en/llm_observability/monitoring/metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/monitoring/metrics.md b/content/en/llm_observability/monitoring/metrics.md index 178335b6d1d87..4d7e31e8776df 100644 --- a/content/en/llm_observability/monitoring/metrics.md +++ b/content/en/llm_observability/monitoring/metrics.md @@ -44,7 +44,7 @@ After you instrument your application with LLM Observability, you can access LLM | Metric Name | Description | Metric Type | Tags | |-------------|-------------|-------------|------| -| `ml_obs.trace` | A collection of spans representing an end-to-end request or workflow | Count | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | +| `ml_obs.trace` | Number of traces | Count | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | | `ml_obs.trace.duration` | Total duration of all traces across all spans | Distribution | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | | `ml_obs.trace.error` | Indicator or description of an error that occurred during the trace | Count | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | From c820a4792c39c169227f758416a5db90f52e091b Mon Sep 17 00:00:00 2001 From: "Amit D." Date: Fri, 3 Oct 2025 12:25:11 -0700 Subject: [PATCH 5/5] Update content/en/llm_observability/monitoring/metrics.md Co-authored-by: cecilia saixue watt --- content/en/llm_observability/monitoring/metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/llm_observability/monitoring/metrics.md b/content/en/llm_observability/monitoring/metrics.md index 4d7e31e8776df..cc5a79559aca0 100644 --- a/content/en/llm_observability/monitoring/metrics.md +++ b/content/en/llm_observability/monitoring/metrics.md @@ -46,7 +46,7 @@ After you instrument your application with LLM Observability, you can access LLM |-------------|-------------|-------------|------| | `ml_obs.trace` | Number of traces | Count | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | | `ml_obs.trace.duration` | Total duration of all traces across all spans | Distribution | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | -| `ml_obs.trace.error` | Indicator or description of an error that occurred during the trace | Count | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | +| `ml_obs.trace.error` | Number of errors that occurred during the trace | Count | `env`, `error`, `ml_app`, `service`, `span_kind`, `version` | ### Estimated usage metrics