From d150620a23eec87c45fe5561d46a95515c547e11 Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 7 Nov 2025 16:49:22 -0500 Subject: [PATCH 01/11] add info --- .../sources/datadog_agent.md | 5 +++- .../sources/opentelemetry.md | 23 ++++++++++++++++- .../sources/splunk_hec.md | 25 +++++++++++++++++++ 3 files changed, 51 insertions(+), 2 deletions(-) diff --git a/content/en/observability_pipelines/sources/datadog_agent.md b/content/en/observability_pipelines/sources/datadog_agent.md index d342b16628994..7b1790e0e25d2 100644 --- a/content/en/observability_pipelines/sources/datadog_agent.md +++ b/content/en/observability_pipelines/sources/datadog_agent.md @@ -5,6 +5,8 @@ disable_toc: false Use Observability Pipelines' Datadog Agent source to receive logs from the Datadog Agent. Select and set up this source when you [set up a pipeline][1]. +**Note**: If you are using the Datadog Distribution of OpenTelemetry Collector, you must [use the OpenTelemetry source to send logs to Observability Pipelines][4]. + ## Prerequisites {{% observability_pipelines/prerequisites/datadog_agent %}} @@ -38,4 +40,5 @@ Use the Agent configuration file or the Agent Helm chart values file to connect [1]: /observability_pipelines/configuration/set_up_pipelines/ [2]: /containers/docker/log/?tab=containerinstallation#linux -[3]: /containers/guide/container-discovery-management/?tab=helm#setting-environment-variables \ No newline at end of file +[3]: /containers/guide/container-discovery-management/?tab=helm#setting-environment-variables +[4]: /observability_pipelines/sources/opentelemetry/#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines \ No newline at end of file diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index ea4a37ba542b3..41323ca4f68c0 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -7,6 +7,10 @@ disable_toc: false Use Observability Pipelines' OpenTelemetry (OTel) source to collect logs from your OTel Collector through HTTP or gRPC. Select and set up this source when you set up a pipeline. The information below is configured in the pipelines UI. +**Notes**: +- If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), [use the OpenTelemetry source to send logs to Observability Pipelines](#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines). +- If you are using the Splunk HEC Distribution of the OpenTelemetry Collector, you must use the [Splunk HEC source][4] to send logs to Observability pipelines. + ### When to use this source Common scenarios when you might use this source: @@ -73,6 +77,23 @@ Based on these example configurations, these are values you enter for the follow - HTTP listener address: `worker:4317` - gRPC listener address: `worker:4318` +## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines + +To send logs from DDOT: +1. Deploy the DDOT Agent using helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions. +1. [Set up a pipeline][6] on Observabiity Pipelines using the [OpenTelemetry source](#set-up-the-source-in-the-pipeline-ui). + 1. (Optional) Datadog recommends adding an [Edit Fields processor][7] to the pipeline that appends the field `op_otel_ddot:true`. + 1. When you install the Worker, for the OpenTelemetry source environment variables: + 1. Set your HTTP listener to `0.0.0.0:4318`. + 1. Set your gRPC listener to `0.0.0.0:4317`. + +**Note**: Logs sent from DDOT might have nested objects that prevent Datadog from parsing the log into the prettified structure. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object. + [1]: https://opentelemetry.io/docs/collector/ [2]: /observability_pipelines/sources/ -[3]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/#bootstrap-options \ No newline at end of file +[3]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/#bootstrap-options +[4]: /observability_pipelines/sources/splunk_hec/#send-logs-from-the-splunk-distributor-of-the-opentelemetry-collector-to-observability-pipelines +[5]: /opentelemetry/setup/ddot_collector/install/kubernetes_daemonset/?tab=datadogoperator +[6]: /observability_pipelines/configuration/set_up_pipelines/ +[7]: /observability_pipelines/processors/edit_fields#add-field +[8]: /observability_pipelines/processors/custom_processor \ No newline at end of file diff --git a/content/en/observability_pipelines/sources/splunk_hec.md b/content/en/observability_pipelines/sources/splunk_hec.md index d14d93da17338..99c2cf2894fc2 100644 --- a/content/en/observability_pipelines/sources/splunk_hec.md +++ b/content/en/observability_pipelines/sources/splunk_hec.md @@ -5,6 +5,8 @@ disable_toc: false Use Observability Pipelines' Splunk HTTP Event Collector (HEC) source to receive logs from your Splunk HEC. Select and set up this source when you [set up a pipeline][1]. +**Note**: You must use the Splunk HEC source if you want to [send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines](#send-logs-from-the-splunk-distributor-of-the-opentelemetry-collector-to-observability-pipelines). + ## Prerequisites {{% observability_pipelines/prerequisites/splunk_hec %}} @@ -21,4 +23,27 @@ Select and set up this source when you [set up a pipeline][1]. The information b {{% observability_pipelines/log_source_configuration/splunk_hec %}} +## Send logs from the Splunk Distributor of the OpenTelemetry collector to Observability Pipelines + +To send logs from the Splunk Distributor of the OpenTelemetry Collector: + +1. Install the Splunk OpenTelemetry Collector based on the your environment: + - [Kubernetes][2] + - [Linux][3] +2. Configure the Splunk OpenTelemetry Collector: + ```bash + cp /etc/otel/collector/splunk-otel-collector.conf.example etc/otel/collector/splunk-otel-collector.conf + ``` + ```bash + # Splunk HEC endpoint URL, if forwarding to Splunk Observability Cloud + # SPLUNK_HEC_URL=https://ingest.us0.signalfx.com/v1/log + # If you're forwarding to a Splunk Enterprise instance running on example.com, with HEC at port 8088: + SPLUNK_HEC_URL=http://0.0.0.0:8088/services/collector + ``` + +3. [Set up a pipeline][4] using the [Splunk HEC source](#set-up-the-source-in-the-pipeline-ui). + [1]: /observability_pipelines/configuration/set_up_pipelines/ +[2]: https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-for-kubernetes +[3]: https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-for-linux +[4]: /observability_pipelines/configuration/set_up_pipelines From d132577ce745ad35d5b67009c8857caf6f601244 Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 7 Nov 2025 17:34:00 -0500 Subject: [PATCH 02/11] small edits --- content/en/observability_pipelines/sources/opentelemetry.md | 2 +- content/en/observability_pipelines/sources/splunk_hec.md | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index 41323ca4f68c0..9b88e375bbf52 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -9,7 +9,7 @@ Use Observability Pipelines' OpenTelemetry (OTel) source to collect logs from yo **Notes**: - If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), [use the OpenTelemetry source to send logs to Observability Pipelines](#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines). -- If you are using the Splunk HEC Distribution of the OpenTelemetry Collector, you must use the [Splunk HEC source][4] to send logs to Observability pipelines. +- If you are using the Splunk HEC Distribution of the OpenTelemetry Collector, use the [Splunk HEC source][4] to send logs to Observability pipelines. ### When to use this source diff --git a/content/en/observability_pipelines/sources/splunk_hec.md b/content/en/observability_pipelines/sources/splunk_hec.md index 99c2cf2894fc2..9d437bf936b2d 100644 --- a/content/en/observability_pipelines/sources/splunk_hec.md +++ b/content/en/observability_pipelines/sources/splunk_hec.md @@ -5,7 +5,7 @@ disable_toc: false Use Observability Pipelines' Splunk HTTP Event Collector (HEC) source to receive logs from your Splunk HEC. Select and set up this source when you [set up a pipeline][1]. -**Note**: You must use the Splunk HEC source if you want to [send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines](#send-logs-from-the-splunk-distributor-of-the-opentelemetry-collector-to-observability-pipelines). +**Note**: Use the Splunk HEC source if you want to [send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines](#send-logs-from-the-splunk-distributor-of-the-opentelemetry-collector-to-observability-pipelines). ## Prerequisites @@ -40,7 +40,6 @@ To send logs from the Splunk Distributor of the OpenTelemetry Collector: # If you're forwarding to a Splunk Enterprise instance running on example.com, with HEC at port 8088: SPLUNK_HEC_URL=http://0.0.0.0:8088/services/collector ``` - 3. [Set up a pipeline][4] using the [Splunk HEC source](#set-up-the-source-in-the-pipeline-ui). [1]: /observability_pipelines/configuration/set_up_pipelines/ From 071770e4345273a47dd51ab7aee5d39bf5f15823 Mon Sep 17 00:00:00 2001 From: May Lee Date: Mon, 10 Nov 2025 11:43:22 -0500 Subject: [PATCH 03/11] Apply suggestions from code review --- content/en/observability_pipelines/sources/datadog_agent.md | 2 +- content/en/observability_pipelines/sources/opentelemetry.md | 6 +++--- content/en/observability_pipelines/sources/splunk_hec.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/content/en/observability_pipelines/sources/datadog_agent.md b/content/en/observability_pipelines/sources/datadog_agent.md index 7b1790e0e25d2..0d7e69b57603e 100644 --- a/content/en/observability_pipelines/sources/datadog_agent.md +++ b/content/en/observability_pipelines/sources/datadog_agent.md @@ -5,7 +5,7 @@ disable_toc: false Use Observability Pipelines' Datadog Agent source to receive logs from the Datadog Agent. Select and set up this source when you [set up a pipeline][1]. -**Note**: If you are using the Datadog Distribution of OpenTelemetry Collector, you must [use the OpenTelemetry source to send logs to Observability Pipelines][4]. +**Note**: If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), you must [use the OpenTelemetry source to send logs to Observability Pipelines][4]. ## Prerequisites diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index 9b88e375bbf52..0b6b28ab2ab43 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -79,9 +79,9 @@ Based on these example configurations, these are values you enter for the follow ## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines -To send logs from DDOT: -1. Deploy the DDOT Agent using helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions. -1. [Set up a pipeline][6] on Observabiity Pipelines using the [OpenTelemetry source](#set-up-the-source-in-the-pipeline-ui). +To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT): +1. Deploy the DDOT Collector using helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions. +1. [Set up a pipeline][6] on Observability Pipelines using the [OpenTelemetry source](#set-up-the-source-in-the-pipeline-ui). 1. (Optional) Datadog recommends adding an [Edit Fields processor][7] to the pipeline that appends the field `op_otel_ddot:true`. 1. When you install the Worker, for the OpenTelemetry source environment variables: 1. Set your HTTP listener to `0.0.0.0:4318`. diff --git a/content/en/observability_pipelines/sources/splunk_hec.md b/content/en/observability_pipelines/sources/splunk_hec.md index 9d437bf936b2d..7381c6d82b088 100644 --- a/content/en/observability_pipelines/sources/splunk_hec.md +++ b/content/en/observability_pipelines/sources/splunk_hec.md @@ -23,9 +23,9 @@ Select and set up this source when you [set up a pipeline][1]. The information b {{% observability_pipelines/log_source_configuration/splunk_hec %}} -## Send logs from the Splunk Distributor of the OpenTelemetry collector to Observability Pipelines +## Send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines -To send logs from the Splunk Distributor of the OpenTelemetry Collector: +To send logs from the Splunk Distribution of the OpenTelemetry Collector: 1. Install the Splunk OpenTelemetry Collector based on the your environment: - [Kubernetes][2] From 2ce1fc0d8295b8f24ec04a399f9f0e060c426c4e Mon Sep 17 00:00:00 2001 From: May Lee Date: Tue, 11 Nov 2025 11:07:02 -0500 Subject: [PATCH 04/11] apply suggestions --- .../sources/opentelemetry.md | 23 ++++++++++++++++++- .../sources/splunk_hec.md | 13 +++++++---- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index 0b6b28ab2ab43..cc2583ea4f8f6 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -86,8 +86,29 @@ To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT) 1. When you install the Worker, for the OpenTelemetry source environment variables: 1. Set your HTTP listener to `0.0.0.0:4318`. 1. Set your gRPC listener to `0.0.0.0:4317`. + 1. After you installed the Worker and deployed the pipeline, update the OpenTelemetry Collector's `collector-config.yaml` to include an exporter that sends logs to Observability Pipelines. For example: + ``` + exporters: + datadog: + api: + site: ${env:} + key: ${env:} + otlphttp: + endpoint: http://opw-observability-pipelines-worker.default.svc.cluster.local:4318 + ``` + 1. Redeploy the Datadog Agent with the updated `collector-config.yaml`.For example, if the Agent is installed in Kubernetes: + ``` + helm upgrade --install datadog-agent datadog/datadog \ + --values ./agent.yaml \ + --set-file datadog.otelCollector.config=./collector-config.yaml + ``` -**Note**: Logs sent from DDOT might have nested objects that prevent Datadog from parsing the log into the prettified structure. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object. +**Notes**: +- These settings are used when setting up the Datadog Agent source in Observability Pipelines: + - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED:true` + - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL:"http://opw-observability-pipelines-worker.default.svc.cluster.local:4317/v1/logs"` +

These settings do **not** work when setting up DDOT with the OpenTelemetry source, because the OpenTelemetry Collector's `collector-config.yaml` must be configured with those settings using the exporter. +- Logs sent from DDOT might have nested objects that prevent Datadog from parsing the logs into the prettified structure. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object. [1]: https://opentelemetry.io/docs/collector/ [2]: /observability_pipelines/sources/ diff --git a/content/en/observability_pipelines/sources/splunk_hec.md b/content/en/observability_pipelines/sources/splunk_hec.md index 7381c6d82b088..d500840d09318 100644 --- a/content/en/observability_pipelines/sources/splunk_hec.md +++ b/content/en/observability_pipelines/sources/splunk_hec.md @@ -5,7 +5,7 @@ disable_toc: false Use Observability Pipelines' Splunk HTTP Event Collector (HEC) source to receive logs from your Splunk HEC. Select and set up this source when you [set up a pipeline][1]. -**Note**: Use the Splunk HEC source if you want to [send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines](#send-logs-from-the-splunk-distributor-of-the-opentelemetry-collector-to-observability-pipelines). +**Note**: Use the Splunk HEC source if you want to [send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines](#send-logs-from-the-splunk-distribution-of-the-opentelemetry-collector-to-observability-pipelines). ## Prerequisites @@ -30,7 +30,8 @@ To send logs from the Splunk Distribution of the OpenTelemetry Collector: 1. Install the Splunk OpenTelemetry Collector based on the your environment: - [Kubernetes][2] - [Linux][3] -2. Configure the Splunk OpenTelemetry Collector: +1. [Set up a pipeline][4] using the [Splunk HEC source](#set-up-the-source-in-the-pipeline-ui). +1. Configure the Splunk OpenTelemetry Collector: ```bash cp /etc/otel/collector/splunk-otel-collector.conf.example etc/otel/collector/splunk-otel-collector.conf ``` @@ -38,9 +39,13 @@ To send logs from the Splunk Distribution of the OpenTelemetry Collector: # Splunk HEC endpoint URL, if forwarding to Splunk Observability Cloud # SPLUNK_HEC_URL=https://ingest.us0.signalfx.com/v1/log # If you're forwarding to a Splunk Enterprise instance running on example.com, with HEC at port 8088: - SPLUNK_HEC_URL=http://0.0.0.0:8088/services/collector + SPLUNK_HEC_URL=http://:8088/services/collector ``` -3. [Set up a pipeline][4] using the [Splunk HEC source](#set-up-the-source-in-the-pipeline-ui). + - `` is the IP or URL of the host (or load balancer) associated with the Observability Pipelines Worker. + - For CloudFormation installs, the `LoadBalancerDNS` CloudFormation output has the correct URL to use. + - For Kubernetes installs, the internal DNS record of the Observability Pipelines Worker service can be used, for example `opw-observability-pipelines-worker.default.svc.cluster.local`. + +**Note**: If you are using a firewall, make sure your firewall allows traffic from the Splunk OpenTelemetry Collector to the Worker. [1]: /observability_pipelines/configuration/set_up_pipelines/ [2]: https://help.splunk.com/en/splunk-observability-cloud/manage-data/splunk-distribution-of-the-opentelemetry-collector/get-started-with-the-splunk-distribution-of-the-opentelemetry-collector/collector-for-kubernetes From f85fc2691f9239bd85a8fbae5fc7110b7dcb2022 Mon Sep 17 00:00:00 2001 From: May Lee Date: Tue, 11 Nov 2025 12:15:44 -0500 Subject: [PATCH 05/11] Update content/en/observability_pipelines/sources/opentelemetry.md --- content/en/observability_pipelines/sources/opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index cc2583ea4f8f6..33669592e14a8 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -108,7 +108,7 @@ To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT) - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED:true` - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL:"http://opw-observability-pipelines-worker.default.svc.cluster.local:4317/v1/logs"`

These settings do **not** work when setting up DDOT with the OpenTelemetry source, because the OpenTelemetry Collector's `collector-config.yaml` must be configured with those settings using the exporter. -- Logs sent from DDOT might have nested objects that prevent Datadog from parsing the logs into the prettified structure. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object. +- Logs sent from DDOT might have nested objects that prevent Datadog from parsing the logs correctly. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object. [1]: https://opentelemetry.io/docs/collector/ [2]: /observability_pipelines/sources/ From 4fc82d9ec0de32f6b5d901b460435a7317cd6ecf Mon Sep 17 00:00:00 2001 From: May Lee Date: Tue, 11 Nov 2025 13:24:45 -0500 Subject: [PATCH 06/11] apply suggestions --- .../sources/opentelemetry.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index 33669592e14a8..adbba3650c21c 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -89,12 +89,10 @@ To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT) 1. After you installed the Worker and deployed the pipeline, update the OpenTelemetry Collector's `collector-config.yaml` to include an exporter that sends logs to Observability Pipelines. For example: ``` exporters: - datadog: - api: - site: ${env:} - key: ${env:} otlphttp: endpoint: http://opw-observability-pipelines-worker.default.svc.cluster.local:4318 + ... + service: pipelines: logs: exporters: [otlphttp] ``` 1. Redeploy the Datadog Agent with the updated `collector-config.yaml`.For example, if the Agent is installed in Kubernetes: ``` @@ -104,10 +102,9 @@ To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT) ``` **Notes**: -- These settings are used when setting up the Datadog Agent source in Observability Pipelines: - - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED:true` - - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL:"http://opw-observability-pipelines-worker.default.svc.cluster.local:4317/v1/logs"` -

These settings do **not** work when setting up DDOT with the OpenTelemetry source, because the OpenTelemetry Collector's `collector-config.yaml` must be configured with those settings using the exporter. +- Since DDOT is sending logs to Observability Pipelines, and not the Datadog Agent, the following settings do not work for sending logs from DDOT to Observability Pipelines: + - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED` + - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL` - Logs sent from DDOT might have nested objects that prevent Datadog from parsing the logs correctly. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object. [1]: https://opentelemetry.io/docs/collector/ From 8c9b4f12488991b2000cf655310cda5232b26f1d Mon Sep 17 00:00:00 2001 From: May Lee Date: Tue, 11 Nov 2025 14:25:49 -0500 Subject: [PATCH 07/11] apply suggestions --- .../sources/opentelemetry.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index adbba3650c21c..624167a2cba7e 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -1,5 +1,5 @@ --- -title: OpenTelemetry +title: OpenTelemetry Source disable_toc: false --- @@ -86,19 +86,22 @@ To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT) 1. When you install the Worker, for the OpenTelemetry source environment variables: 1. Set your HTTP listener to `0.0.0.0:4318`. 1. Set your gRPC listener to `0.0.0.0:4317`. - 1. After you installed the Worker and deployed the pipeline, update the OpenTelemetry Collector's `collector-config.yaml` to include an exporter that sends logs to Observability Pipelines. For example: + 1. After you installed the Worker and deployed the pipeline, update the OpenTelemetry Collector's [`otel-config.yaml`][9] to include an exporter that sends logs to Observability Pipelines. For example: ``` exporters: otlphttp: endpoint: http://opw-observability-pipelines-worker.default.svc.cluster.local:4318 ... - service: pipelines: logs: exporters: [otlphttp] + service: + pipelines: + logs: + exporters: [otlphttp] ``` - 1. Redeploy the Datadog Agent with the updated `collector-config.yaml`.For example, if the Agent is installed in Kubernetes: + 1. Redeploy the Datadog Agent with the updated [`otel-config.yaml`][9].For example, if the Agent is installed in Kubernetes: ``` helm upgrade --install datadog-agent datadog/datadog \ --values ./agent.yaml \ - --set-file datadog.otelCollector.config=./collector-config.yaml + --set-file datadog.otelCollector.config=./otel-config.yaml ``` **Notes**: @@ -114,4 +117,5 @@ To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT) [5]: /opentelemetry/setup/ddot_collector/install/kubernetes_daemonset/?tab=datadogoperator [6]: /observability_pipelines/configuration/set_up_pipelines/ [7]: /observability_pipelines/processors/edit_fields#add-field -[8]: /observability_pipelines/processors/custom_processor \ No newline at end of file +[8]: /observability_pipelines/processors/custom_processor +[9]: https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes_daemonset/?tab=helm#configure-the-opentelemetry-collector \ No newline at end of file From 1cef01071154ae12b66ec2b8104c1c6ac80b3ef0 Mon Sep 17 00:00:00 2001 From: May Lee Date: Wed, 12 Nov 2025 09:08:18 -0500 Subject: [PATCH 08/11] Update content/en/observability_pipelines/sources/opentelemetry.md --- content/en/observability_pipelines/sources/opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index 624167a2cba7e..fbf474016c2ea 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -8,7 +8,7 @@ disable_toc: false Use Observability Pipelines' OpenTelemetry (OTel) source to collect logs from your OTel Collector through HTTP or gRPC. Select and set up this source when you set up a pipeline. The information below is configured in the pipelines UI. **Notes**: -- If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), [use the OpenTelemetry source to send logs to Observability Pipelines](#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines). +- If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), use the OpenTelemetry source to [send logs to Observability Pipelines](#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines). - If you are using the Splunk HEC Distribution of the OpenTelemetry Collector, use the [Splunk HEC source][4] to send logs to Observability pipelines. ### When to use this source From 1f034b42a5643c9d8fc1dc0df9f4b7c6ced50145 Mon Sep 17 00:00:00 2001 From: May Lee Date: Wed, 12 Nov 2025 14:16:10 -0500 Subject: [PATCH 09/11] Apply suggestions from code review Co-authored-by: cecilia saixue wat-kim --- .../en/observability_pipelines/sources/datadog_agent.md | 2 +- .../en/observability_pipelines/sources/opentelemetry.md | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/content/en/observability_pipelines/sources/datadog_agent.md b/content/en/observability_pipelines/sources/datadog_agent.md index 0d7e69b57603e..b1b49805d3264 100644 --- a/content/en/observability_pipelines/sources/datadog_agent.md +++ b/content/en/observability_pipelines/sources/datadog_agent.md @@ -5,7 +5,7 @@ disable_toc: false Use Observability Pipelines' Datadog Agent source to receive logs from the Datadog Agent. Select and set up this source when you [set up a pipeline][1]. -**Note**: If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), you must [use the OpenTelemetry source to send logs to Observability Pipelines][4]. +**Note**: If you are using the Datadog Distribution of OpenTelemetry (DDOT) Collector, you must [use the OpenTelemetry source to send logs to Observability Pipelines][4]. ## Prerequisites diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index fbf474016c2ea..38d36299ab2cf 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -8,8 +8,8 @@ disable_toc: false Use Observability Pipelines' OpenTelemetry (OTel) source to collect logs from your OTel Collector through HTTP or gRPC. Select and set up this source when you set up a pipeline. The information below is configured in the pipelines UI. **Notes**: -- If you are using the Datadog Distribution of OpenTelemetry Collector (DDOT), use the OpenTelemetry source to [send logs to Observability Pipelines](#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines). -- If you are using the Splunk HEC Distribution of the OpenTelemetry Collector, use the [Splunk HEC source][4] to send logs to Observability pipelines. +- If you are using the Datadog Distribution of OpenTelemetry (DDOT) Collector, use the OpenTelemetry source to [send logs to Observability Pipelines](#send-logs-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines). +- If you are using the Splunk HEC Distribution of the OpenTelemetry Collector, use the [Splunk HEC source][4] to send logs to Observability Pipelines. ### When to use this source @@ -79,7 +79,7 @@ Based on these example configurations, these are values you enter for the follow ## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines -To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT): +To send logs from the Datadog Distribution of the OpenTelemetry (DDOT) Collector: 1. Deploy the DDOT Collector using helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions. 1. [Set up a pipeline][6] on Observability Pipelines using the [OpenTelemetry source](#set-up-the-source-in-the-pipeline-ui). 1. (Optional) Datadog recommends adding an [Edit Fields processor][7] to the pipeline that appends the field `op_otel_ddot:true`. @@ -105,7 +105,7 @@ To send logs from the Datadog Distribution of the OpenTelemetry Collector (DDOT) ``` **Notes**: -- Since DDOT is sending logs to Observability Pipelines, and not the Datadog Agent, the following settings do not work for sending logs from DDOT to Observability Pipelines: +- Because DDOT is sending logs to Observability Pipelines, and not the Datadog Agent, the following settings do not work for sending logs from DDOT to Observability Pipelines: - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED` - `DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL` - Logs sent from DDOT might have nested objects that prevent Datadog from parsing the logs correctly. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object. From f9279b50a9399dfe25261787c4ee614a563943d6 Mon Sep 17 00:00:00 2001 From: May Lee Date: Wed, 12 Nov 2025 15:39:58 -0500 Subject: [PATCH 10/11] Apply suggestions from code review Co-authored-by: Rosa Trieu <107086888+rtrieu@users.noreply.github.com> --- content/en/observability_pipelines/sources/opentelemetry.md | 6 +++--- content/en/observability_pipelines/sources/splunk_hec.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index 38d36299ab2cf..e01b30716028d 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -80,13 +80,13 @@ Based on these example configurations, these are values you enter for the follow ## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines To send logs from the Datadog Distribution of the OpenTelemetry (DDOT) Collector: -1. Deploy the DDOT Collector using helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions. +1. Deploy the DDOT Collector using Helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions. 1. [Set up a pipeline][6] on Observability Pipelines using the [OpenTelemetry source](#set-up-the-source-in-the-pipeline-ui). 1. (Optional) Datadog recommends adding an [Edit Fields processor][7] to the pipeline that appends the field `op_otel_ddot:true`. 1. When you install the Worker, for the OpenTelemetry source environment variables: 1. Set your HTTP listener to `0.0.0.0:4318`. 1. Set your gRPC listener to `0.0.0.0:4317`. - 1. After you installed the Worker and deployed the pipeline, update the OpenTelemetry Collector's [`otel-config.yaml`][9] to include an exporter that sends logs to Observability Pipelines. For example: + 1. After you install the Worker and deployed the pipeline, update the OpenTelemetry Collector's [`otel-config.yaml`][9] to include an exporter that sends logs to Observability Pipelines. For example: ``` exporters: otlphttp: @@ -97,7 +97,7 @@ To send logs from the Datadog Distribution of the OpenTelemetry (DDOT) Collector logs: exporters: [otlphttp] ``` - 1. Redeploy the Datadog Agent with the updated [`otel-config.yaml`][9].For example, if the Agent is installed in Kubernetes: + 1. Redeploy the Datadog Agent with the updated [`otel-config.yaml`][9]. For example, if the Agent is installed in Kubernetes: ``` helm upgrade --install datadog-agent datadog/datadog \ --values ./agent.yaml \ diff --git a/content/en/observability_pipelines/sources/splunk_hec.md b/content/en/observability_pipelines/sources/splunk_hec.md index d500840d09318..9ef32ba8b937b 100644 --- a/content/en/observability_pipelines/sources/splunk_hec.md +++ b/content/en/observability_pipelines/sources/splunk_hec.md @@ -27,7 +27,7 @@ Select and set up this source when you [set up a pipeline][1]. The information b To send logs from the Splunk Distribution of the OpenTelemetry Collector: -1. Install the Splunk OpenTelemetry Collector based on the your environment: +1. Install the Splunk OpenTelemetry Collector based on your environment: - [Kubernetes][2] - [Linux][3] 1. [Set up a pipeline][4] using the [Splunk HEC source](#set-up-the-source-in-the-pipeline-ui). From 12180fabad09eb2ccca98df4efa2b8d13eb512f2 Mon Sep 17 00:00:00 2001 From: May Lee Date: Wed, 12 Nov 2025 15:42:12 -0500 Subject: [PATCH 11/11] Update content/en/observability_pipelines/sources/opentelemetry.md --- content/en/observability_pipelines/sources/opentelemetry.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index e01b30716028d..d908fd8b1edf7 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -74,8 +74,8 @@ The Worker exposes the gRPC endpoint on port 4318. This is an example of configu Based on these example configurations, these are values you enter for the following environment variables: -- HTTP listener address: `worker:4317` -- gRPC listener address: `worker:4318` +- HTTP listener address: `worker:4318` +- gRPC listener address: `worker:4317` ## Send logs from the Datadog Distribution of OpenTelemetry Collector to Observability Pipelines