CloudQuery

Overview

CloudQuery is an open-source, high-performance data integration framework built for developers, with support for a wide range of plugins.

CloudQuery extracts, transforms, and loads configuration from cloud APIs to a variety of supported destinations such as databases, data lakes, or streaming platforms for further analysis.

Setup

Installation

To ingest OpenTelemetry traces, metrics, and logs from CloudQuery, install the Datadog Agent version >=6.48.0 or >=7.48.0. Alternatively, you can use OpenTelemetry Collector and Datadog Exporter as described below.

Configuration

CloudQuery supports OpenTelemetry traces, metrics, and logs out of the box. There are multiple ways to configure OpenTelemetry with Datadog:

• Using an OpenTelemetry collector
• Direct OTEL Ingestion by the Datadog Agent through a configuration file
• Direct OTEL ingestion by the Datadog Agent through environment variables

For more information, see OpenTelemetry in Datadog.

OpenTelemetry collector

To configure an OpenTelemetry collector with Datadog:

1. Create a configuration file. For example, create an otel_collector_config.yaml file with the content below:

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch/datadog:
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: 10s

exporters:
  datadog:
    api:
      site: ${env:DATADOG_SITE}
      key: ${env:DATADOG_API_KEY}

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch/datadog]
      exporters: [datadog]
    traces:
      receivers: [otlp]
      processors: [batch/datadog]
      exporters: [datadog]
    logs:
      receivers: [otlp]
      processors: [batch/datadog]
      exporters: [datadog]

2. Run the collector with the following command (replace DATADOG_SITE and DATADOG_API_KEY with your own values):

docker run \
    -p 4318:4318 \
    -e DATADOG_SITE=$DATADOG_SITE \
    -e DATADOG_API_KEY=$DATADOG_API_KEY \
    --hostname $(hostname) \
    -v $(pwd)/otel_collector_config.yaml:/etc/otelcol-contrib/config.yaml \
    otel/opentelemetry-collector-contrib:0.104.0

3. After you have the collector ready, specify the endpoint in the source spec:

kind: source
spec:
  name: "aws"
  path: "cloudquery/aws"
  # Replace with the AWS source plugin version
  version: "<VERSION_SOURCE_AWS>"
  tables: ["aws_s3_buckets"]
  destinations: ["postgresql"]
  otel_endpoint: "0.0.0.0:4318"
  otel_endpoint_insecure: true
  spec:

For additional ways to run the collector, see OpenTelemetry Deployment.

Datadog Agent OTEL ingestion through a configuration file

1. Locate your datadog.yaml Agent configuration file and add the following configuration:

otlp_config:
  receiver:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
  logs:
    enabled: true
logs_enabled: true

2. Restart the Datadog agent for the change to take effect.

3. After you have the Agent ready, specify the endpoint in the source spec:

kind: source
spec:
  name: "aws"
  path: "cloudquery/aws"
  # Replace with the AWS source plugin version
  version: "<VERSION_SOURCE_AWS>"
  tables: ["aws_s3_buckets"]
  destinations: ["postgresql"]
  otel_endpoint: "0.0.0.0:4318"
  otel_endpoint_insecure: true
  spec:

Datadog Agent OTEL ingestion through environment variables

1. Pass the DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT environment variable to the Datadog Agent with a value of 0.0.0.0:4318. If you're using Docker compose, see the example below:

version: "3.0"
services:
  agent:
    image: gcr.io/datadoghq/agent:7
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    environment:
      DD_API_KEY: redacted
      DD_SITE: "datadoghq.eu"
      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318"
      DD_LOGS_ENABLED: "true"
      DD_OTLP_CONFIG_LOGS_ENABLED: "true"
    ports:
      - "4318:4318"

2. Restart the Datadog agent for the change to take effect.

3. After you have the Agent ready, specify the endpoint in the source spec:

kind: source
spec:
  name: "aws"
  path: "cloudquery/aws"
  # Replace with the AWS source plugin version
  version: "<VERSION_SOURCE_AWS>"
  tables: ["aws_s3_buckets"]
  destinations: ["postgresql"]
  otel_endpoint: "0.0.0.0:4318"
  otel_endpoint_insecure: true
  spec:

For more ways to configure the Datadog Agent, see OTLP Ingestion by the Datadog Agent.

Validation

Run cloudquery sync spec.yml. After ingestion starts, you should start seeing the traces in the Datadog APM Traces Explorer. You can also validate metrics and logs in the Metrics Summary and Log Explorer.

Data Collected

Metrics

The CloudQuery does not include any metrics.

Service Checks

The CloudQuery does not include any service checks.

Events

The CloudQuery does not include any events.

Uninstallation

If you use the OpenTelemetry collector, you can stop it by running docker stop <container_id>. If you use the Datadog Agent, remove the configuration or environment variables you added and restart the agent. Finally, delete the dashboard from your Datadog account.

Support

For support, contact CloudQuery.