Skip to content
125 changes: 115 additions & 10 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5842,56 +5842,161 @@ menu:
parent: observability_pipelines_destinations
identifier: observability_pipelines_syslog
weight: 421
- name: Packs
url: observability_pipelines/packs/
parent: observability_pipelines
identifier: observability_pipelines_packs
weight: 5
- name: Akamai CDN
url: observability_pipelines/packs/akamai_cdn/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_akamai_cdn
weight: 501
- name: Amazon CloudFront
url: observability_pipelines/packs/amazon_cloudfront/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_amazon_cloudfront
weight: 502
- name: Amazon VPC Flow Logs
url: observability_pipelines/packs/amazon_vpc_flow_logs/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_amazon_vpc_flow_logs
weight: 503
- name: AWS CloudTrail
url: observability_pipelines/packs/aws_cloudtrail/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_aws_cloudtrail
weight: 504
- name: Cisco ASA
url: observability_pipelines/packs/cisco_asa/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_cisco_asa
weight: 505
- name: Cloudflare
url: observability_pipelines/packs/cloudflare/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_cloudflare
weight: 506
- name: F5
url: observability_pipelines/packs/f5/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_f5
weight: 507
- name: Fastly
url: observability_pipelines/packs/fastly/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_fastly
weight: 508
- name: Fortinet Firewall
url: observability_pipelines/packs/fortinet_firewall/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_fortinet_firewall
weight: 509
- name: HAProxy Ingress
url: observability_pipelines/packs/haproxy_ingress/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_haproxy_ingress
weight: 510
- name: Istio Proxy
url: observability_pipelines/packs/istio_proxy/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_istio_proxy
weight: 511
- name: Netskope
url: observability_pipelines/packs/netskope/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_netskope
weight: 512
- name: NGINX
url: observability_pipelines/packs/nginx/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_nginx
weight: 513
- name: Okta
url: observability_pipelines/packs/okta/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_okta
weight: 514
- name: Palo Alto Firewall
url: observability_pipelines/packs/palo_alto_firewall/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_palo_alto_firewall
weight: 515
- name: Windows XML
url: observability_pipelines/packs/windows_xml/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_windows_xml
weight: 516
- name: ZScaler ZIA DNS
url: observability_pipelines/packs/zscaler_zia_dns/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_zscaler_zia_dns
weight: 517
- name: Zscaler ZIA Firewall
url: observability_pipelines/packs/zscaler_zia_firewall/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_zscaler_zia_firewall
weight: 518
- name: Zscaler ZIA Tunnel
url: observability_pipelines/packs/zscaler_zia_tunnel/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_zscaler_zia_tunnel
weight: 519
- name: Zscaler ZIA Web Logs
url: observability_pipelines/packs/zscaler_zia_web_logs/
parent: observability_pipelines_packs
identifier: observability_pipelines_packs_zscaler_zia_web_logs
weight: 520
- name: Search Syntax
url: observability_pipelines/search_syntax/
parent: observability_pipelines
identifier: observability_pipelines_search_syntax
weight: 5
weight: 6
- name: Scaling and Performance
url: observability_pipelines/scaling_and_performance/
parent: observability_pipelines
identifier: observability_pipelines_scaling_and_performance
weight: 6
weight: 7
- name: Handling Load and Backpressure
url: observability_pipelines/scaling_and_performance/handling_load_and_backpressure/
parent: observability_pipelines_scaling_and_performance
identifier: observability_pipelines_handling_load_and_backpressure
weight: 601
weight: 701
- name: Best Practices for Scaling Observability Pipelines
url: observability_pipelines/scaling_and_performance/best_practices_for_scaling_observability_pipelines/
parent: observability_pipelines_scaling_and_performance
identifier: observability_pipelines_best_practices_for_scaling_observability_pipelines
weight: 602
weight: 702
- name: Monitoring and Troubleshooting
url: observability_pipelines/monitoring_and_troubleshooting/
parent: observability_pipelines
identifier: observability_pipelines_monitoring_and_troubleshooting
weight: 7
weight: 8
- name: Worker CLI Commands
url: observability_pipelines/monitoring_and_troubleshooting/worker_cli_commands/
parent: observability_pipelines_monitoring_and_troubleshooting
identifier: observability_pipelines_worker_cli_commands
weight: 701
weight: 801
- name: Monitoring Pipelines
url: observability_pipelines/monitoring_and_troubleshooting/monitoring_pipelines/
parent: observability_pipelines_monitoring_and_troubleshooting
identifier: observability_pipelines_monitoring_pipelines
weight: 702
weight: 802
- name: Pipeline Usage Metrics
url: observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/
parent: observability_pipelines_monitoring_and_troubleshooting
identifier: observability_pipelines_pipeline_usage_metrics
weight: 703
weight: 803
- name: Troubleshooting
url: observability_pipelines/monitoring_and_troubleshooting/troubleshooting/
identifier: observability_pipelines_troubleshooting
parent: observability_pipelines_monitoring_and_troubleshooting
weight: 704
weight: 804
- name: Guides
url: observability_pipelines/guide/
parent: observability_pipelines
identifier: observability_pipelines_guide
weight: 8
weight: 9
- name: Log Management
url: logs/
pre: log
Expand Down
39 changes: 28 additions & 11 deletions content/en/observability_pipelines/packs/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,39 @@ cascade:

## Overview

When you set up a pipeline to send logs from a specific source to Observability Pipelines, you might have questions such as:
{{< img src="observability_pipelines/packs/packs.png" alt="The packs section of Observability Pipelines" style="width:100%;" >}}

When setting up a pipeline to send logs from a specific source to Observability Pipelines, you often need to decide how to process and manage those logs.

Questions such as the following might come up:

- Which logs from this source are important?
- Which logs from this source should be dropped?
- Which logs should be retained?
- Should logs be sampled?
- Should quotas be added?
- Which logs can safely be dropped?
- Should repetitive logs be sampled?
- Which fields should be parsed or formatted for the destination?

Making these decisions typically requires coordination across multiple teams and detailed knowledge of each log source.

Observability Pipelines Packs provide predefined configurations to help you make these decisions quickly and consistently. Packs apply Datadog-recommended best practices for specific log sources such as Akamai, AWS CloudTrail, Cloudflare, Fastly, Palo Alto Firewall, and Zscaler.

### What Packs do

Each Pack includes source-specific configurations that defines:

- **Fields that can safely be removed** to reduce payload size
- **Logs that can be dropped**, such as duplicate events or health checks
- **Logs that should be retained or parsed**, such as errors or security detections
- **Formatting and normalization rules** to align logs across different destinations and environments

By using Packs, you can apply consistent parsing, filtering, and routing logic for each log source without creating configurations manually.

Often, you need to consult with different teams to answer these questions.
### Why use Packs

Use Observability Pipelines Packs to help you set up and optimize Observability Pipelines without extensive manual configuration. Packs contain predefined configurations that are specific to a source and identify:
Packs help teams:

- Log fields that can safely be removed
- Logs that can be dropped, such as duplicated logs
- Logs that need to be parsed
- Logs that need to be formatted for the destination
- **Reduce ingestion volume and costs** by filtering or sampling repetitive, low-value events
- **Maintain consistency** in parsing and field mapping across environments and destinations
- **Accelerate setup** by applying ready-to-use configurations for common sources

## Packs

Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/akamai_cdn.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Akamai CDN pack.

## Overview

{{< img src="observability_pipelines/packs/akamai_cdn.png" alt="The Akamai pack" style="width:25%;" >}}

Akamai logs show client requests and responses at the edge.

What this pack does:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the AWS CloudFront pack.

## Overview

{{< img src="observability_pipelines/packs/aws_cloudfront.png" alt="The Amazon CloudFront pack" style="width:25%;" >}}

AWS CloudFront logs show requests, cache use, and edge activity.

What this pack does:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Amazon VPC Flow Logs pack.

## Overview

{{< img src="observability_pipelines/packs/aws_vpc_flow_logs.png" alt="The Amazon VPC Flow Logs pack" style="width:25%;" >}}

Amazon VPC Flow Logs capture network traffic between VPC resources.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/aws_cloudtrail.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the AWS CloudTrail pack.

## Overview

{{< img src="observability_pipelines/packs/aws_cloudtrail.png" alt="The AWS CloudTrail pack" style="width:25%;" >}}

AWS CloudTrail records API calls and account activity across AWS services.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/cisco_asa.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Cisco ASA pack.

## Overview

{{< img src="observability_pipelines/packs/cisco_asa.png" alt="The Cisco ASA pack" style="width:25%;" >}}

Cisco ASA firewall logs capture syslog events for traffic, VPNs, and security alerts.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/cloudflare.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Cloudflare pack.

## Overview

{{< img src="observability_pipelines/packs/cloudflare.png" alt="The Cloudflare pack" style="width:25%;" >}}

Cloudflare logs show edge traffic, performance, and security.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/f5.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the F5 pack.

## Overview

{{< img src="observability_pipelines/packs/f5.png" alt="The F5 pack" style="width:25%;" >}}

F5 logs capture traffic, security policy, and intrusion events.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/fastly.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Fastly pack.

## Overview

{{< img src="observability_pipelines/packs/fastly.png" alt="The Fastly pack" style="width:25%;" >}}

Fastly CDN logs record client requests, cache states, and delivery performance.

What this pack does:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Fortinet Firewall pack.

## Overview

{{< img src="observability_pipelines/packs/fortinet_firewall.png" alt="The Fortinet Firewall pack" style="width:25%;" >}}

Fortinet firewall logs record allowed, denied, and other network traffic.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/haproxy_ingress.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the HAProxy Ingress pack.

## Overview

{{< img src="observability_pipelines/packs/haproxy_ingress.png" alt="The HAProxy Ingress pack" style="width:25%;" >}}

HAProxy Ingress logs record how Kubernetes ingress traffic is routed and served.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/istio_proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Istio Proxy pack.

## Overview

{{< img src="observability_pipelines/packs/istio_proxy.png" alt="The Istio Proxy pack" style="width:25%;" >}}

Istio Proxy logs capture inbound and outbound traffic handled by Envoy.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/netskope.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Netskope pack.

## Overview

{{< img src="observability_pipelines/packs/netskope.png" alt="The Netskope pack" style="width:25%;" >}}

Netskope logs capture cloud app use, policies, and security events.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/nginx.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the NGINX pack.

## Overview

{{< img src="observability_pipelines/packs/nginx.png" alt="The NGINX pack" style="width:25%;" >}}

NGINX logs record client requests, responses, and errors from the web server.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/okta.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Okta pack.

## Overview

{{< img src="observability_pipelines/packs/okta.png" alt="The Okta pack" style="width:25%;" >}}

Okta logs show authentication, user activity, and policy events.

What this pack does:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Palo Alto Firewall pack.

## Overview

{{< img src="observability_pipelines/packs/palo_alto_firewall.png" alt="The Palo Alto Firewall pack" style="width:25%;" >}}

Palo Alto firewall logs capture traffic, threat, and system events.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/windows_xml.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Windows XML pack.

## Overview

{{< img src="observability_pipelines/packs/windows_xml.png" alt="The Windows XML pack" style="width:25%;" >}}

Windows Event logs capture system, application, and security activity from Windows hosts.

What this pack does:
Expand Down
2 changes: 2 additions & 0 deletions content/en/observability_pipelines/packs/zscaler_zia_dns.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the ZScaler ZIA DNS pack.

## Overview

{{< img src="observability_pipelines/packs/zscaler_dns.png" alt="The ZScaler ZIA DNS pack" style="width:25%;" >}}

ZScaler Internet Access (ZIA) DNS logs capture org-wide DNS activity and policy actions.

What this pack does:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Zscaler ZIA Firewall pack.

## Overview

{{< img src="observability_pipelines/packs/zscaler_firewall.png" alt="The Zscaler ZIA Firewall pack" style="width:25%;" >}}

Zscaler Internet Access (ZIA) Firewall logs show network traffic and security events.

What this pack does:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Learn more about the Zscaler ZIA Tunnel pack.

## Overview

{{< img src="observability_pipelines/packs/zscaler_tunnel.png" alt="The Zscaler ZIA Tunnel pack" style="width:25%;" >}}

Zscaler Internet Access (ZIA) Tunnel logs show tunnel health, traffic, and key events.

What this pack does:
Expand Down
Loading
Loading