Running Tracetest with AWS X-Ray (AWS Distro for OpenTelemetry & Pokeshop API)

:::note Check out the source code on GitHub here. :::

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

AWS X-Ray provides a complete view of requests as they travel through your application and filters visual data across payloads, functions, traces, services, APIs, and more with no-code and low-code motions.

AWS Distro for OpenTelemetry (ADOT) is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. Part of the Cloud Native Computing Foundation, OpenTelemetry provides open source APIs, libraries, and agents to collect distributed traces and metrics for application monitoring.

Pokeshop API As a testing ground, the team at Tracetest has implemented a sample instrumented API around the PokeAPI.

Pokeshop API with AWS X-Ray and Tracetest

This is a simple quick start guide on how to configure a fully instrumented API to be used with Tracetest for enhancing your E2E and integration tests with trace-based testing. The infrastructure will use AWS X-Ray as the trace data store, the ADOT as a middleware and the Pokeshop API to generate the telemetry data.

Prerequisites

You will need Docker and Docker Compose installed on your machine to run this quick start app!

And a set of AWS credentials to connect Tracetest to the cloud API.

Project Structure

The project is built with Docker Compose.

1. Tracetest

The docker-compose.yaml file, tracetest.provision.yaml, and tracetest.config.yaml in the root directory are for the setting up the Pokeshop API, Tracetest and ADOT Collector.

Docker Compose Network

All services in the docker-compose.yaml are on the same network and will be reachable by hostname from within other services. E.g. adot-collector:2000 in the src/index.js will map to the adot-collector service, where the port 2000 is the port where the X-Ray Daemon accepts telemetry data.

Pokeshop API

The Pokeshop API is a fully instrumented REST API that makes use of different services to mimic a real life scenario.

It is instrumented using the OpenTelemetry standards for Node.js, sending the data to the ADOT collector that will be pushing the telemetry information to both the AWS X-Ray service.

This is a fragment from the main tracing file from the Pokeshop API repo.

import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import * as opentelemetry from '@opentelemetry/api';
import { api, NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { Resource } from '@opentelemetry/resources';
import * as dotenv from 'dotenv';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { IORedisInstrumentation } from '@opentelemetry/instrumentation-ioredis';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';
import { AmqplibInstrumentation } from '@opentelemetry/instrumentation-amqplib';
import { SpanStatusCode } from '@opentelemetry/api';
import { B3Propagator } from '@opentelemetry/propagator-b3';

dotenv.config();
api.propagation.setGlobalPropagator(new B3Propagator());

const { COLLECTOR_ENDPOINT = '', SERVICE_NAME = 'pokeshop' } = process.env;

let globalTracer: opentelemetry.Tracer | null = null;

async function createTracer(): Promise<opentelemetry.Tracer> {
  const collectorExporter = new OTLPTraceExporter({
    url: COLLECTOR_ENDPOINT,
  });

  const spanProcessor = new BatchSpanProcessor(collectorExporter);
  const sdk = new NodeSDK({
    traceExporter: collectorExporter,
    // @ts-ignore
    instrumentations: [new IORedisInstrumentation(), new PgInstrumentation(), new AmqplibInstrumentation()],
  });

  sdk.configureTracerProvider({}, spanProcessor);
  sdk.addResource(
    new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: SERVICE_NAME,
    })
  );

  await sdk.start();
  process.on('SIGTERM', () => {
    sdk
      .shutdown()
      .then(
        () => console.log('SDK shut down successfully'),
        err => console.log('Error shutting down SDK', err)
      )
      .finally(() => process.exit(0));
  });

  const tracer = opentelemetry.trace.getTracer(SERVICE_NAME);

  globalTracer = tracer;

  return globalTracer;
}

The docker-compose.yaml file includes the definitions for all of the required services by the Pokeshop API, which includes:

Postgres - To save Pokemon information.
Redis - For in memory strage.
RabbitMQ - For async processing use cases.
API - The Pokeshop API main container.
Worker - The Pokeshop worker instance.

version: "3"

services:
  cache:
    image: redis:6
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 1s
      timeout: 3s
      retries: 60

  queue:
    image: rabbitmq:3.8-management
    restart: unless-stopped
    healthcheck:
      test: rabbitmq-diagnostics -q check_running
      interval: 1s
      timeout: 5s
      retries: 60

  demo-api:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://adot-collector:4317
      NPM_RUN_COMMAND: api
    ports:
      - "8081:8081"
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:8081"]
      interval: 1s
      timeout: 3s
      retries: 60
    depends_on:
      postgres:
        condition: service_healthy
      cache:
        condition: service_healthy
      queue:
        condition: service_healthy

  demo-worker:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://adot-collector:4317
      NPM_RUN_COMMAND: worker
    depends_on:
      postgres:
        condition: service_healthy
      cache:
        condition: service_healthy
      queue:
        condition: service_healthy

  demo-rpc:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://adot-collector:4317
      NPM_RUN_COMMAND: rpc
    ports:
      - 8082:8082
    healthcheck:
      test: ["CMD", "lsof", "-i", "8082"]
      interval: 1s
      timeout: 3s
      retries: 60
    depends_on:
      postgres:
        condition: service_healthy
      cache:
        condition: service_healthy
      queue:
        condition: service_healthy

Tracetest

The docker-compose.yaml includes three other services.

Postgres - Postgres is a prerequisite for Tracetest to work. It stores trace data when running the trace-based tests.
AWS Distro for OpenTelemetry (ADOT) - Software application that listens for traffic on UDP port 2000, gathers raw segment data and relays it to the AWS X-Ray API. The daemon works in conjunction with the AWS X-Ray SDKs and must be running so that data sent by the SDKs can reach the X-Ray service.
Tracetest - Trace-based testing that generates end-to-end tests automatically from traces.

version: "3"
services:
  tracetest:
    image: kubeshop/tracetest:${TAG:-latest}
    platform: linux/amd64
    volumes:
      - type: bind
        source: ./tracetest.config.yaml
        target: /app/tracetest.yaml
      - type: bind
        source: ./tracetest.provision.yaml
        target: /app/provisioning.yaml
    ports:
      - 11633:11633
    command: --provisioning-file /app/provisioning.yaml
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      postgres:
        condition: service_healthy
      adot-collector:
        condition: service_started
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:11633"]
      interval: 1s
      timeout: 3s
      retries: 60

  postgres:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
    healthcheck:
      test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
      interval: 1s
      timeout: 5s
      retries: 60
    ports:
      - 5432:5432

  adot-collector:
    image: amazon/aws-otel-collector:latest
    command:
      - "--config"
      - "/otel-local-config.yaml"
    volumes:
      - ./collector.config.yaml:/otel-local-config.yaml
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_SESSION_TOKEN: ${AWS_SESSION_TOKEN}
      AWS_REGION: ${AWS_REGION}
    ports:
      - 4317:4317
      - 2000:2000

Tracetest depends on Postgres and the ADOT collector. Tracetest requires config files to be loaded via a volume. The volumes are mapped from the root directory into the root directory and the respective config files.

The tracetest.config.yaml file contains the basic setup of connecting Tracetest to the Postgres instance.

postgres:
  host: postgres
  user: postgres
  password: postgres
  port: 5432
  dbname: postgres
  params: sslmode=disable

The tracetest.provision.yaml file defines the trace data store, set to AWS X-Ray, meaning the traces will be stored in X-Ray and Tracetest will fetch them from X-Ray when running tests.

But how does Tracetest fetch traces?

Tracetest uses the Golang AWS-SDK library to pull to fetch trace data.

---
type: DataStore
spec:
  name: awsxray
  type: awsxray
  awsxray:
    accessKeyId: <your-accessKeyId>
    secretAccessKey: <your-secretAccessKey>
    sessionToken: <your-session-token>
    region: "us-west-2"

How do traces reach AWS X-Ray?

The Pokeshop API code uses the native Node.js OpenTelemetry modules which sends information to the ADOT Collector to be processed and then sent to the configured AWS X-Ray SaaS.

Run Both the Pokeshop API and Tracetest

To start both the Pokeshop API and Tracetest, run this command:

docker-compose up

This will start your Tracetest instance on http://localhost:11633/. Open it and start creating tests! Make sure to use the http://demo-api:8081/ url in your test creation, because your Pokeshop API and Tracetest are in the same network.

Learn More

Please visit our examples in GitHub and join our Discord Community for more info!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

running-tracetest-with-aws-x-ray-pokeshop.md

running-tracetest-with-aws-x-ray-pokeshop.md

Running Tracetest with AWS X-Ray (AWS Distro for OpenTelemetry & Pokeshop API)

Pokeshop API with AWS X-Ray and Tracetest

Prerequisites

Project Structure

1. Tracetest

Docker Compose Network

Pokeshop API

Tracetest

Run Both the Pokeshop API and Tracetest

Learn More

Files

running-tracetest-with-aws-x-ray-pokeshop.md

Latest commit

History

running-tracetest-with-aws-x-ray-pokeshop.md

File metadata and controls

Running Tracetest with AWS X-Ray (AWS Distro for OpenTelemetry & Pokeshop API)

Pokeshop API with AWS X-Ray and Tracetest

Prerequisites

Project Structure

1. Tracetest

Docker Compose Network

Pokeshop API

Tracetest

Run Both the Pokeshop API and Tracetest

Learn More