-
Notifications
You must be signed in to change notification settings - Fork 258
docs(audit-trail): add tutorial to export audit trail to datadog #4374
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
nerda-codes
merged 7 commits into
scaleway:main
from
luxifer:int-export-audit-trail-to-datadog
Feb 11, 2025
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
0478375
docs: write tutorial to export audit trail to datadog
luxifer 77f189f
fix: specify go os and arch for build
luxifer 636f305
docs(audit-trail): doc review
nerda-codes cc4b36f
docs(audit-trail): review
nerda-codes 27e7089
Update tutorials/export-audit-trail-to-datadog/index.mdx
nerda-codes 37a10ad
Update tutorials/export-audit-trail-to-datadog/index.mdx
nerda-codes 81d7aa5
Apply suggestions from code review
nerda-codes File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,174 @@ | ||
| --- | ||
| meta: | ||
| title: Exporting Audit Trail events to DataDog | ||
| description: Learn how to export Scaleway Audit Trail events to DataDog | ||
| content: | ||
| h1: Exporting Audit Trail events to DataDog | ||
| paragraph: Learn how to export Scaleway Audit Trail events to DataDog | ||
| tags: audit-trail log events | ||
| categories: | ||
| - audit-trail | ||
| - instances | ||
| dates: | ||
| validation: 2025-02-10 | ||
| posted: 2025-02-10 | ||
| --- | ||
|
|
||
| This tutorial shows you how to export your Audit Trail events to [DataDog](https://www.datadoghq.com/). For the purpose of this tutorail, we are building a [custom OpenTelemetry Collector](https://opentelemetry.io/docs/collector/custom-collector/) to collect Audit Trail events through the [Audit Trail receiver](https://github.com/scaleway/opentelemetry-collector-scaleway/tree/main/receiver/scwaudittrail) and export them with the [DataDog exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/datadogexporter). | ||
|
|
||
| <Macro id="requirements" /> | ||
|
|
||
| - A Scaleway account logged into the [console](https://console.scaleway.com) | ||
| - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization | ||
| - Created an [SSH key](/organizations-and-projects/how-to/create-ssh-key/) | ||
| - Created a Scaleway [Instance](/instances/how-to/create-an-instance/) | ||
| - Installed the [OpenTelemetry collector builder](https://opentelemetry.io/docs/collector/custom-collector/#step-1---install-the-builder) | ||
| - Created a [Datadog account](https://app.datadoghq.com/account/login) and a [Datadog API key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys) | ||
|
|
||
| ## Building the OpenTelemetry collector | ||
|
|
||
| 1. Open a terminal and check that the `ocb` binary is ready to be used. The output of the `help` command should display, meaning the `ocb` binary is ready to be used. | ||
|
|
||
| ``` | ||
| ./ocb help | ||
| ``` | ||
|
|
||
| 2. Create a manifest file named `builder-config.yaml` and paste the following content into it. This file is used to defines code generation, the compile process, and the components to include in your Collector’s distribution. | ||
|
|
||
| ```yaml | ||
| dist: | ||
| name: otelcol-audit-trail | ||
| description: OpenTelemetry Collector for Audit Trail | ||
| output_path: ./otelcol-audit-trail | ||
|
|
||
| exporters: | ||
| - gomod: | ||
| github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter v0.118.0 | ||
|
|
||
| processors: | ||
| - gomod: | ||
| go.opentelemetry.io/collector/processor/batchprocessor v0.118.0 | ||
|
|
||
| receivers: | ||
| - gomod: | ||
| github.com/scaleway/opentelemetry-collector-scaleway/receiver/scwaudittrail v0.1.0 | ||
|
|
||
| providers: | ||
| - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v1.24.0 | ||
| - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v1.24.0 | ||
| - gomod: go.opentelemetry.io/collector/confmap/provider/httpprovider v1.24.0 | ||
| - gomod: go.opentelemetry.io/collector/confmap/provider/httpsprovider v1.24.0 | ||
| - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.24.0 | ||
| ``` | ||
|
|
||
| 3. Run the following command to build the Collector. `GOOS` and `GOARCH` are needed in the command as the target deployment is Linux with an AMD64 CPU. | ||
|
|
||
| ``` | ||
| GOOS=linux GOARCH=amd64 ./ocb --config builder-config.yaml | ||
| ``` | ||
|
|
||
| You now have a new folder named `otelcol-audit-trail/` with the binary `otelcol-audit-trail` compiled inside. | ||
|
|
||
| ## Deploying the Collector | ||
|
|
||
| 1. Run the following command to upload the Collector binary to your Instance. Make sure that you replace `<INSTANCE_IP_ADDRESS>` with the IP address of your Instance. | ||
|
|
||
| ``` | ||
| scp otelcol-audit-trail/otelcol-audit-trail root@<INSTANCE_IP_ADDRESS>:/usr/local/bin/ | ||
| ``` | ||
|
|
||
| 2. Connect to your Instance via SSH: | ||
|
|
||
| ``` | ||
| ssh root@<INSTANCE_IP_ADDRESS> | ||
| ``` | ||
|
|
||
| ## Configure the Collector | ||
|
|
||
| Create a file named `/etc/opentelemetry-collector/config.yaml` and paste the following content into it. This file is the configuration our custom Collector will run. | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| scwaudittrail: | ||
| access_key: <SCW_ACCESS_KEY> | ||
| secret_key: <SCW_SECRET_KEY> | ||
| organization_id: <SCW_DEFAULT_ORGANIZATION_ID> | ||
| region: <SCW_DEFAULT_REGION> | ||
|
|
||
| processors: | ||
| batch: | ||
| send_batch_max_size: 1000 | ||
| send_batch_size: 100 | ||
| timeout: 10s | ||
|
|
||
| exporters: | ||
| datadog: | ||
| idle_conn_timeout: 10s | ||
| api: | ||
| key: <DD_API_KEY> | ||
| site: <DD_SITE> | ||
|
|
||
| service: | ||
| pipelines: | ||
| logs: | ||
| receivers: [scwaudittrail] | ||
| processors: [batch] | ||
| exporters: [datadog] | ||
| ``` | ||
|
|
||
| Make sure that you replace: | ||
|
|
||
| - `<SCW_ACCESS_KEY>` with your Scaleway API access key | ||
| - `<SCW_SECRET_KEY>` with your Scaleway API secret key | ||
| - `<SCW_DEFAULT_ORGANIZATION_ID>` with your Scaleway Organization ID | ||
| - `<SCW_DEFAULT_REGION>` with the Scaleway region to target | ||
| - `<DD_API_KEY>` with your DataDog API secret key | ||
| - `<DD_SITE>` with the [DataDog site](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) you are on | ||
|
|
||
| ## Running the Collector | ||
|
|
||
| 1. Create a file named `/etc/systemd/system/opentelemetry-collector.service` and paste the following content into it. This file will create the `systemd` service that runs the Collector. | ||
|
|
||
| ``` | ||
| [Unit] | ||
| Description=OpenTelemetry Collector | ||
| After=multi-user.target | ||
|
|
||
| [Service] | ||
| ExecStart=/usr/local/bin/otelcol-audit-trail --config /etc/opentelemetry-collector/config.yaml | ||
| Type=simple | ||
|
|
||
| [Install] | ||
| WantedBy=multi-user.target | ||
| ``` | ||
|
|
||
| 2. Run the following command to update `systemd` services: | ||
|
|
||
| ``` | ||
| systemctl daemon-reload | ||
| ``` | ||
|
|
||
| 3. Run the following commands to enable and start the service: | ||
|
|
||
| ``` | ||
| systemctl enable opentelemetry-collector.service | ||
| systemctl start opentelemetry-collector.service | ||
| ``` | ||
|
|
||
| 4. Make sure that the service is running: | ||
|
|
||
| ``` | ||
| systemctl status opentelemetry-collector.service | ||
| ``` | ||
|
|
||
| 5. Run the command below to visualize your logs: | ||
|
|
||
| ``` | ||
| journalctl -fu opentelemetry-collector.service | ||
| ``` | ||
|
|
||
| An output similar to the following should display to confirm that the Collector is polling Audit Trail events: | ||
|
|
||
| ``` | ||
| Feb 07 15:34:30 scw-beautiful-zhukovsky otelcol-audit-trail[1723]: 2025-02-07T15:34:30.687Z info scwaudittrail@v0.1.0/receiver.go:80 Polling Audit Trail logs {"kind": "receiver", "name": "scwaudittrail", "data_type": "logs"} | ||
| ``` | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.