Run soak tests from EC2

[anchitj]

What

Adds OTEL config which would be used by OTEL agent to export the metrics and logs of Soak tests to Datadog.

Instructions to install OTEL agent on EC2:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES=service.name=njc-py-soak-tests-normal-2-2-1,service.version=v2.2.1-RC1-dev_soaktest_k8s_on_2.2.1
export OTEL_TRACES_EXPORTER=none

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.117.0/otelcol-contrib_0.117.0_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.117.0_linux_amd64.deb
otelcol-contrib --config otel_config.yaml

This would start OTEL agent and tests would start exporting metrics and logs to it.

Other changes to run with EC2

export HOSTNAME=$(hostname)
PYTHONPATH=/home/ubuntu/confluent-kafka-python/tests opentelemetry-instrument confluent-kafka-python/tests/soak/soakclient.py -t $topic -r 80 -f confluent-kafka-python/ccloud.config -i t1
pip install psutil

Checklist

• Contains customer facing changes? Including API/behavior changes
• Did you add sufficient unit test and/or integration test coverage for this PR?
  • If not, please explain why it is not required

References

JIRA:

Test & Review

Open questions / Follow-ups

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Copilot]

PR Overview

This PR adds a configuration file for the OTEL agent to export metrics and logs from soak tests to Datadog.

• Adds a new YAML file for OTEL configuration.
• Configures receivers, processors, exporters, and service pipelines for both metrics and logs.

Reviewed Changes

File	Description
tests/soak/otel-config.yaml	Added OTEL configuration with receivers, processors, exporters, and service pipelines for Datadog integration

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

The Datadog exporter API key is set to a placeholder value ('KEY'). Replace it with the actual key or use a secure method (e.g., environment variable) to load this value in production.

Suggested change

	key: KEY
	key: ${DD_API_KEY}

[sonarqube-confluent]

Analysis Details

0 Issues

• 0 Bugs
• 0 Vulnerabilities
• 0 Code Smells

Coverage and Duplications

• No coverage information (61.70% Estimated after merge)
• No duplication information (0.90% Estimated after merge)

Project ID: confluent-kafka-python

View in SonarQube