Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions content/en/conf/opentelemetry-collector/1-installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,14 @@ Obtain the `.deb` package for your platform from the [OpenTelemetry Collector Co
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.75.0/otelcol-contrib_0.75.0_linux_amd64.deb
```

{{%expand "🥷 Ninja: Build your own collector using _Open Telemetry Collector Builder (ocb)_"%}}
***
{{%expand "🥷 Ninja: Build your own collector using Open Telemetry Collector Builder (ocb)"%}}
For this part we will require the following installed on your system:

- Golang (latest version)
- ocb installed
- Most recent from the [project releases](https://github.com/open-telemetry/opentelemetry-collector/releases)
- Install using go by `go install go.opentelemetry.io/collector/cmd/builder@latest`
- Install using go by `go install go.opentelemetry.io/collector/cmd/builder@v0.75.0`
- (Optional) Docker

## Why build your own collector?
Expand Down Expand Up @@ -80,7 +81,7 @@ exporters:
- gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.75.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.75.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/signalfxexporter v0.75.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/sapmexporter v0.75.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/sapmexporter v0.75.0

processors:
- gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.75.0
Expand Down Expand Up @@ -119,6 +120,7 @@ Which leave you with the following directory structure:

1. https://opentelemetry.io/docs/collector/custom-collector/
{{% /expand%}}
***

## 2. Installing the OpenTelemetry Collector Contrib distribution

Expand Down Expand Up @@ -204,9 +206,9 @@ cat /etc/otelcol-contrib/config.yaml
extensions:
health_check:
pprof:
endpoint: 0.0.0.0:1777
endpoint: 127.0.0.1:1777
zpages:
endpoint: 0.0.0.0:55679
endpoint: 127.0.0.1:55679

receivers:
otlp:
Expand All @@ -223,7 +225,7 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']

jaeger:
protocols:
Expand Down
68 changes: 62 additions & 6 deletions content/en/conf/opentelemetry-collector/2-extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ sudo vi /etc/otelcol-contrib/config.yaml
```yaml {hl_lines=[3]}
extensions:
health_check:
endpoint: 0.0.0.0:13133
endpoint: "127.0.0.0:13133"
```

{{% /tab %}}
Expand All @@ -29,11 +29,11 @@ extensions:
```yaml {hl_lines=[3]}
extensions:
health_check:
endpoint: 0.0.0.0:13133
endpoint: 127.0.0.0:13133
pprof:
endpoint: 0.0.0.0:1777
endpoint: 127.0.0.0:1777
zpages:
endpoint: 0.0.0.0:55679
endpoint: 127.0.0.0:55679

receivers:
otlp:
Expand All @@ -50,7 +50,7 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.0:8888']

jaeger:
protocols:
Expand Down Expand Up @@ -94,6 +94,61 @@ Restart the collector:
sudo systemctl restart otelcol-contrib
```

***

{{% expand "🥷 Ninja: Improve data durability with storage extension" %}}
For this, we will need to validate our distrobution has the `file_storage` extension installed,
this can be down by running the command `otelcol-contrib components` and it should so something like:

```yaml
# ... truncated for clarity
extensions:
- file_storage
```

This extension provides exporters the ability to queue data to disk in the event that exporter is unable
to send data to the configured endpoint.

In order to configure the extension, you will need to update to include the following information:

```yaml
extensions:
file_storage:
directory: /tmp/otel-data
timeout: 10s
compaction:
directory: /tmp/otel-data
on_start: true
on_rebound: true
rebound_needed_threshold_mib: 5
rebound_trigger_threshold_mib: 3

# ... truncated for clarity

service:
extension:
# Additiona extensions here
- file_storage
```

## Why queue data to disk?

This allows the collector to queue data (and even restart) to ensure data is sent
the upstream provider.

## Considerations for queuing data to disk?

There is a potential that this could impact data throughput performance due disk performance.

#### References

1. https://community.splunk.com/t5/Community-Blog/Data-Persistence-in-the-OpenTelemetry-Collector/ba-p/624583
1. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage

{{% /expand %}}

***

## Health Check

This extension enables an HTTP url that can be probed to check the status of the OpenTelemetry Collector. This extension can be used as a liveness and/or readiness probe on Kubernetes. To learn more about the curl command, check out the [curl man page.](https://curl.se/docs/manpage.html)
Expand Down Expand Up @@ -150,4 +205,5 @@ Example URL: [http://localhost:55679/debug/extensionz](http://localhost:55679/de

![ExtensionZ](../images/extensionz.png)

Now that we have reviewed extensions, lets dive into the data pipeline portion of the workshop. The data pipeline in the OpenTelemetry Collector is made up of receivers, processors, and exporters. We will first start with receivers.
Now that we have reviewed extensions, lets dive into the data pipeline portion of the workshop. The data pipeline in the OpenTelemetry Collector is made up of receivers, processors, and exporters. We will first start with receivers.

18 changes: 9 additions & 9 deletions content/en/conf/opentelemetry-collector/3-receivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ receivers:
```yaml {hl_lines=["9-30"]}
extensions:
health_check:
endpoint: 0.0.0.0:13133
endpoint: 127.0.0.1:13133
pprof:
endpoint: 0.0.0.0:1777
endpoint: 127.0.0.1:1777
zpages:
endpoint: 0.0.0.0:55679
endpoint: 127.0.0.1:55679

receivers:
hostmetrics:
Expand Down Expand Up @@ -90,7 +90,7 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']

jaeger:
protocols:
Expand Down Expand Up @@ -142,7 +142,7 @@ prometheus/internal:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']
```

{{% /tab %}}
Expand All @@ -151,11 +151,11 @@ prometheus/internal:
```yaml {hl_lines=[39]}
extensions:
health_check:
endpoint: 0.0.0.0:13133
endpoint: 127.0.0.1:13133
pprof:
endpoint: 0.0.0.0:1777
endpoint: 127.0.0.1:1777
zpages:
endpoint: 0.0.0.0:55679
endpoint: 127.0.0.1:55679

receivers:
hostmetrics:
Expand Down Expand Up @@ -193,7 +193,7 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']

jaeger:
protocols:
Expand Down
8 changes: 4 additions & 4 deletions content/en/conf/opentelemetry-collector/4-processors.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ processors:
``` yaml
extensions:
health_check:
endpoint: 0.0.0.0:13133
endpoint: 127.0.0.1:13133
pprof:
endpoint: 0.0.0.0:1777
endpoint: 127.0.0.1:1777
zpages:
endpoint: 0.0.0.0:55679
endpoint: 127.0.0.1:55679

receivers:
hostmetrics:
Expand Down Expand Up @@ -76,7 +76,7 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']

jaeger:
protocols:
Expand Down
8 changes: 4 additions & 4 deletions content/en/conf/opentelemetry-collector/6-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,11 +96,11 @@ service:
``` yaml
extensions:
health_check:
endpoint: 0.0.0.0:13133
endpoint: 127.0.0.1:13133
pprof:
endpoint: 0.0.0.0:1777
endpoint: 127.0.0.1:1777
zpages:
endpoint: 0.0.0.0:55679
endpoint: 127.0.0.1:55679

receivers:
hostmetrics:
Expand Down Expand Up @@ -138,7 +138,7 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']

jaeger:
protocols:
Expand Down
40 changes: 20 additions & 20 deletions content/en/conf/opentelemetry-collector/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,9 @@ Vanilla/Contrib/Vendor
extensions:
health_check:
pprof:
endpoint: 0.0.0.0:1777
endpoint: 127.0.0.1:1777
zpages:
endpoint: 0.0.0.0:55679
endpoint: 127.0.0.1:55679

receivers:
otlp:
Expand All @@ -116,7 +116,7 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']

jaeger:
protocols:
Expand Down Expand Up @@ -169,10 +169,10 @@ service:

extensions:
health_check:
endpoint: 0.0.0.0:13133
endpoint: 127.0.0.1:13133
http_forwarder:
ingress:
endpoint: 0.0.0.0:6060
endpoint: 127.0.0.1:6060
egress:
endpoint: "${SPLUNK_API_URL}"
# Use instead when sending to gateway
Expand All @@ -182,7 +182,7 @@ extensions:
collectd:
configDir: "${SPLUNK_COLLECTD_DIR}"
zpages:
#endpoint: 0.0.0.0:55679
#endpoint: 127.0.0.1:55679
memory_ballast:
# In general, the ballast should be set to 1/3 of the collector's memory, the limit
# should be 90% of the collector's memory.
Expand Down Expand Up @@ -211,19 +211,19 @@ receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
endpoint: 127.0.0.1:14250
thrift_binary:
endpoint: 0.0.0.0:6832
endpoint: 127.0.0.1:6832
thrift_compact:
endpoint: 0.0.0.0:6831
endpoint: 127.0.0.1:6831
thrift_http:
endpoint: 0.0.0.0:14268
endpoint: 127.0.0.1:14268
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
endpoint: 127.0.0.1:4317
http:
endpoint: 0.0.0.0:4318
endpoint: 127.0.0.1:4318
# This section is used to collect the OpenTelemetry Collector metrics
# Even if just a Splunk APM customer, these metrics are included
prometheus/internal:
Expand All @@ -232,23 +232,23 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']
metric_relabel_configs:
- source_labels: [ __name__ ]
regex: '.*grpc_io.*'
action: drop
smartagent/signalfx-forwarder:
type: signalfx-forwarder
listenAddress: 0.0.0.0:9080
listenAddress: 127.0.0.1:9080
smartagent/processlist:
type: processlist
signalfx:
endpoint: 0.0.0.0:9943
endpoint: 127.0.0.1:9943
# Whether to preserve incoming access token and use instead of exporter token
# default = false
#access_token_passthrough: true
zipkin:
endpoint: 0.0.0.0:9411
endpoint: 127.0.0.1:9411

processors:
batch:
Expand Down Expand Up @@ -356,11 +356,11 @@ End goal for metrics only:
```yaml
extensions:
health_check:
endpoint: 0.0.0.0:13133
endpoint: 127.0.0.1:13133
pprof:
endpoint: 0.0.0.0:1777
endpoint: 127.0.0.1:1777
zpages:
endpoint: 0.0.0.0:55679
endpoint: 127.0.0.1:55679

receivers:
hostmetrics:
Expand All @@ -387,7 +387,7 @@ receivers:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
- targets: ['127.0.0.1:8888']

processors:
batch:
Expand Down