Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions hugo/content/en/observability_pipelines/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,11 @@ further_reading:
text: Manage metric volume and tags in your environment with Observability Pipelines

---

## Overview

{{< img src="observability_pipelines/op_marketecture_06042025.png" alt="A graphic showing data being aggregated from a variety of sources, processed and enriched by the observability pipelines worker in your own environment, and then being routed to the security, analytics, and storage destinations of your choice" style="width:100%;" >}}

Datadog Observability Pipelines allows you to collect and process logs and metrics within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment.
Datadog Observability Pipelines allows you to collect and process logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment.

With out-of-the-box templates, you can build pipelines that redact sensitive data, enrich data, filter out noisy events, and route data to destinations like Datadog, SIEM tools, or cloud storage.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ further_reading:

{{< img src="observability_pipelines/setup/pipeline_ui.png" alt="The pipelines page with a source going to two processors groups and two destinations" style="width:100%;" >}}

Observability Pipelines lets you collect and process logs and metrics within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components:
Observability Pipelines lets you collect and process logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one does work! So maybe it's something about the .mdoc file?

Image


- [Source][1]: Receives data from a tool like the Datadog Agent.
- [Processors][2]: Transform, enrich, or filter data.
Expand Down Expand Up @@ -120,6 +120,19 @@ See [Metric Types][3] for more information.
[3]: /metrics/types/?tab=gauge#metric-types
[4]: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality

{{% /tab %}}

{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

Use the [Trace Sampling][1] template to create a trace pipeline.

See [Set Up Pipelines][2] for more information on setting up a source, processors, and destinations.

[1]: /observability_pipelines/configuration/explore_templates/?tab=traces#trace-sampling
[2]: /observability_pipelines/configuration/set_up_pipelines/

{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,15 @@ Metrics capture signals about your environment and offer insight into your syste

To help you manage the quality and volume of your metrics, use the Metric Tag Governance template to process them in Observability Pipelines before sending them to your destinations. You can use processors to keep only the metrics you need, standardize metrics tagging, and remove unwanted tags to prevent high cardinality.

{{% /tab %}}
{{% tab "Traces" %}}

### Trace Sampling

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

Ingesting and storing every trace can be costly at scale. Use the Trace Sampling template to ingest, process, and route your traces telemetry data to control costs while retaining the traces you need for troubleshooting and analysis.

{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ For RHEL and CentOS, the Observability Pipelines Worker supports versions 8.0 or

{% /if %}

The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs and metrics ({% glossary-tooltip term="preview" case="title" /%}), and then route them to different destinations.
The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs, metrics, and traces ({% tooltip contents="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." %}in Preview{% /tooltip %}), and then route them to different destinations.

@maycmlee maycmlee Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tooltip has a space before "in Preview" that I can't get rid of

Image


<!-- Kubernetes - Overview -->
{% if equals($platform, "kubernetes") %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ products:
- name: Metrics
icon: metrics
url: /observability_pipelines/configuration/?tab=metrics#pipeline-types
- name: Traces
icon: apm
url: /observability_pipelines/configuration/?tab=traces#pipeline-types
---

{{< product-availability >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,33 @@ See [Export a Pipeline Configuration to JSON or Terraform][14] if you want to pr
[6]: /observability_pipelines/configuration/pipeline_simulation/
[11]: /observability_pipelines/search_syntax/metrics/

{{% /tab %}}
{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

1. Navigate to [Observability Pipelines][1].
1. Select the [Trace Sampling][2] template, or click {{< ui >}}New Pipeline{{< /ui >}} and select {{< ui >}}Traces Pipeline{{< /ui >}}.
1. Select and set up a [trace source][3].
1. Select and set up [destinations][5] for your processed traces.
1. Click {{< ui >}}Edit{{< /ui >}} on a processor group to add, update, and validate your [processors][4] using [Pipeline Simulation][6].
- **Notes**:
- For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors.
- To copy a processor, click the copy icon for that processor and then paste it (`Cmd+V` on Mac, `Ctrl+V` on Windows or Linux).

#### Add another processor group

{{< img src="observability_pipelines/setup/another_processor_group.png" alt="The Pipelines page showing two processor groups sending logs to the same destination" style="width:100%;" >}}

{{% observability_pipelines/set_up_pipelines/add_another_processor_group %}}

[1]: https://app.datadoghq.com/observability-pipelines
[2]: /observability_pipelines/configuration/explore_templates/?tab=traces#trace-sampling
[3]: /observability_pipelines/sources/?tab=traces#sources
[4]: /observability_pipelines/processors/?tab=traces#processors
[5]: /observability_pipelines/destinations/?tab=traces#destinations
[6]: /observability_pipelines/configuration/pipeline_simulation/

{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ further_reading:

## Overview

Use the Observability Pipelines Worker to send your processed logs and metrics to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information.
Use the Observability Pipelines Worker to send your processed logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}) to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information.

Select a destination in the left navigation menu to see more information about it.

Expand Down Expand Up @@ -84,6 +84,14 @@ These are the available destinations:
[3]: /observability_pipelines/destinations/http_client/
[4]: /observability_pipelines/destinations/splunk_hec/metrics

{{% /tab %}}
{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

- Datadog APM
- OpenTelemetry Traces

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No links currently. These pages will be created in separate child PRs. I will add links there.

{{% /tab %}}
{{< /tabs >}}

Expand Down
17 changes: 16 additions & 1 deletion hugo/content/en/observability_pipelines/processors/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ further_reading:

<div class="alert alert-info">The processors outlined in this documentation are specific to on-premises logging environments. To parse, structure, and enrich cloud-based logs, see the <a href="https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics">Log Management</a> documentation.</div>

Use Observability Pipelines' processors to parse, structure, and enrich your logs and metrics. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs.
Use Observability Pipelines' processors to parse, structure, and enrich your logs, metrics, and traces ({{< tooltip text="in Preview" tooltip="Traces for Observability Pipelines is in Preview. Contact your account manager to request access." >}}). When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this is fixable, but the line is breaking right after the parenthesis

Image

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is weird. I'm seeing something similar with an extra space in the mdoc file. I'll need to ping webops about it.

Image

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Processor groups are executed from top to bottom. The order of the processors is important because events are checked by each processor, but only events that match the processor's filters are processed. To modify the order of the processors, use the drag handle on the top left corner of the processor you want to move.

Expand Down Expand Up @@ -84,6 +84,21 @@ These are the available processors:
[4]: /observability_pipelines/processors/tag_allow_block_list/
[5]: /observability_pipelines/processors/tag_cardinality_control/

{{% /tab %}}
{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

- [Custom Processor][1]
- [Filter][2]
- [Sample][3]
- [Sensitive Data Scanner][4]

[1]: /observability_pipelines/processors/custom_processor/
[2]: /observability_pipelines/processors/filter/
[3]: /observability_pipelines/processors/sample/
[4]: /observability_pipelines/processors/sensitive_data_scanner/

{{% /tab %}}
{{< /tabs >}}

Expand Down
11 changes: 11 additions & 0 deletions hugo/content/en/observability_pipelines/sources/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,17 @@ These are the available sources:
[1]: /observability_pipelines/sources/datadog_agent/
[2]: /observability_pipelines/sources/opentelemetry/

{{% /tab %}}
{{% tab "Traces" %}}

<div class="alert alert-info">Traces for Observability Pipelines is in Preview. Contact your account manager to request access.</div>

- [Datadog Agent][1]
- [OpenTelemetry][2]

[1]: /observability_pipelines/sources/datadog_agent/
[2]: /observability_pipelines/sources/opentelemetry/

{{% /tab %}}
{{< /tabs >}}

Expand Down
Loading