OpenTelemetry Acceptance Tests (OATs), or OATs for short, is a test framework for OpenTelemetry.
- Declarative tests written in YAML
- Supported signals: traces, logs, metrics
- Full round-trip testing: from the application to the observability stack
- Data is stored in the LGTM stack (Loki, Grafana, Tempo, Prometheus, OpenTelemetry Collector)
- Data is queried using LogQL, PromQL, and TraceQL
- All data is sent to the observability stack via OTLP - so OATs can also be used with other observability stacks
- End-to-end testing
- Docker Compose with the docker-otel-lgtm image
- Kubernetes with the docker-otel-lgtm and k3d
- Install the
oats
binary:
go install github.com/grafana/oats@latest
- You can confirm it was installed with:
❯ ls $GOPATH/bin
oats
Tip
You can use the test cases in prom_client_java as a reference. The GitHub action uses a script to run the tests.
- Create a folder
oats-tests
for the following files - Create
Dockerfile
to build the application you want to testFROM eclipse-temurin:21-jre COPY target/example-exporter-opentelemetry.jar ./app.jar ENTRYPOINT [ "java", "-jar", "./app.jar" ]
- Create
docker-compose.yaml
to start the application and any dependenciesservices: java: build: dockerfile: Dockerfile environment: OTEL_SERVICE_NAME: "rolldice" OTEL_EXPORTER_OTLP_ENDPOINT: http://lgtm:4318 OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf OTEL_METRIC_EXPORT_INTERVAL: "5000" # so we don't have to wait 60s for metrics
- Create
oats.yaml
with the test cases# OATs is an acceptance testing framework for OpenTelemetry - https://github.com/grafana/oats docker-compose: files: - ./docker-compose.yaml expected: metrics: - promql: 'uptime_seconds_total{}' value: '>= 0'
- Run the tests:
oats /path/to/oats-tests/oats.yaml
OATs can be run directly using the command-line interface:
# Basic usage
go run main.go /path/to/oats-tests/oats.yaml
# With flags
go run main.go --timeout=1m --lgtm-version=latest --manual-debug=false /path/to/oats-tests/oats.yaml
It can run multiple tests:
oats /path/to/repo
This will search all subdirectories for test files. The tests are defined in oats*.yaml
files.
The following flags are available:
-timeout
: Set the timeout for test cases (default: 30s)-lgtm-version
: Specify the version of docker-otel-lgtm to use (default:"latest"
)-manual-debug
: Enable debug mode to keep containers running (default:false
)-lgtm-log-all
: Enable logging for all containers (default:false
)-lgtm-log-grafana
: Enable logging for Grafana (default:false
)-lgtm-log-loki
: Enable logging for Loki (default:false
)-lgtm-log-tempo
: Enable logging for Tempo (default:false
)-lgtm-log-prometheus
: Enable logging for Prometheus (default:false
)-lgtm-log-pyroscope
: Enable logging for Pyroscope (default:false
)-lgtm-log-otel-collector
: Enable logging for OpenTelemetry Collector (default:false
)-host
: Override the host used to issue requests to applications and LGTM (default:localhost
)
Here's a script that is used from GitHub Actions. It uses mise to install OATs, but you also install OATs directly.
Tip
You can use any file name that matches oats*.yaml
(e.g. oats-test.yaml
), that doesn't end in -template.yaml
.
oats-template.yaml
is reserved for template files, which are used in the include
section.
The syntax is a bit similar to Tracetest.
Here is an example:
include:
- ../oats-template.yaml
docker-compose:
file: ../docker-compose.yaml
input:
- path: /stock
status: 200 # expected status code, 200 is the default
interval: 500ms # interval between requests to the input URL
expected:
traces:
- traceql: '{ name =~ "SELECT .*product"}'
spans:
- name: 'regex:SELECT .*'
attributes:
db.system: h2
logs:
- logql: '{exporter = "OTLP"}'
contains:
- 'hello LGTM'
metrics:
- promql: 'db_client_connections_max{pool_name="HikariPool-1"}'
value: "== 10"
Here is another example with a more specific input:
include:
- ../oats-template.yaml
docker-compose:
file: ../docker-compose.yaml
input:
- path: /users
method: POST
scheme: https
host: 127.0.0.1
status: 201
headers:
Authorization: Bearer my-access-token
Content-Type: application/json
body: |-
{
"name": "Grot"
}
interval: 500ms
expected:
traces:
- traceql: '{ name =~ "SELECT .*product"}'
spans:
- name: 'regex:SELECT .*'
attributes:
db.system: h2
Each entry in the traces
array is a test case for traces.
expected:
traces:
- traceql: '{ name =~ "SELECT .*product"}'
spans:
- name: 'regex:SELECT .*' # regex match
attributes:
db.system: h2
allow-duplicates: true # allow multiple spans with the same attributes
Each entry in the logs
array is a test case for logs.
expected:
logs:
- logql: '{service_name="rolldice"} |~ `Anonymous player is rolling the dice.*`'
equals: 'Anonymous player is rolling the dice'
attributes:
service_name: rolldice
attribute-regexp:
container_id: ".*"
no-extra-attributes: true # fail if there are extra attributes
- logql: '{service_name="rolldice"} |~ `Anonymous player is rolling the dice.*`'
regexp: 'Anonymous player is .*'
expected:
metrics:
- promql: 'db_client_connections_max{pool_name="HikariPool-1"}'
value: "== 10"
Matrix tests are useful to test different configurations of the same application, e.g. with different settings of the otel collector or different flags in the application.
matrix:
- name: default
docker-compose:
files:
- ./docker-compose.oats.yml
- name: self-contained
docker-compose:
files:
- ./docker-compose.self-contained.oats.yml
- name: net8
docker-compose:
files:
- ./docker-compose.net8.oats.yml
You can then make test cases depend on the matrix name:
expected:
metrics:
- promql: 'db_client_connections_max{pool_name="HikariPool-1"}'
value: "== 10"
matrix-condition: default
matrix-condition
is a regex that is applied to the matrix name.
Describes the docker-compose file(s) to use for the test. The files typically define the instrumented application you want to test and optionally some dependencies, e.g. a database server to send requests to. You don't need (and shouldn't have) to define the observability stack (e.g. Prometheus, Grafana, etc.), because this is provided by the test framework (and may test different versions of the observability stack, e.g. OTel Collector and Grafana Alloy).
This docker-compose file is relative to the oats.yaml
file.
A local Kubernetes cluster can be used to test the application in a Kubernetes environment rather than in docker-compose. This is useful to test the application in a more realistic environment - and when you want to test Kubernetes specific features.
Describes the Kubernetes manifest(s) to use for the test.
kubernetes:
dir: k8s
app-service: dice
app-docker-file: Dockerfile
app-docker-context: ..
app-docker-tag: dice:1.1-SNAPSHOT
app-docker-port: 8080