Skip to content

Latest commit

聽

History

History
299 lines (223 loc) 路 11.8 KB

running-tracetest-with-aws-terraform.mdx

File metadata and controls

299 lines (223 loc) 路 11.8 KB
Raw
id title description hide_table_of_contents keywords image
running-tracetest-with-aws-terraform
Serverless Node.js and Jaeger with Terraform
Quick start on how to configure a Serverless Node.js app on AWS Lambda with OpenTelemetry traces and Tracetest for E2E and integration tests. Uses Jaeger as a trace data store and Terraform for infrastructure management.
false
tracetest
trace-based testing
observability
distributed tracing
testing

:::note Check out the source code on GitHub here. :::

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

Jaeger is an open-source, end-to-end distributed tracing solution. It allows you to monitor and troubleshoot transactions in complex distributed systems. It was developed and open sourced by Uber Technologies. Jaeger provides a distributed tracing solution to enable transactions across multiple heterogeneous systems or microservices to be tracked and displayed as a cascading series of spans.

Terraform is an infrastructure as code tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version, reuse, and share. You can then use a consistent workflow to provision and manage all of your infrastructure throughout its lifecycle.

AWS Fargate is a serverless, pay-as-you-go compute engine that lets you focus on building applications without managing servers. AWS Fargate is compatible with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS).

Sample Node.js Serverless API with Jaeger, OpenTelemetry, AWS Fargate and Tracetest

This is a simple quick start guide on how to deploy a Node.js Serverless API to use OpenTelemetry instrumentation with traces and Tracetest for enhancing your E2E and integration tests with trace-based testing. The infrastructure will use Jaeger as the trace data store, and OpenTelemetry Collector to receive traces from the Node.js app and Terraform to provision the required AWS services to run Tracetest in the cloud.

Services Architecture

Terraform Serverless Diagram

Prerequisites

You will need Terraform, a configured instance of the AWS CLI installed on your machine to run this quick start Serverless API!

Project Structure

The project is built with Terraform. It contains multiple .tf files where each part of the infra is divided.

1. Node.js Serverless API

The src folder is the root directory that contains the Node.js files for both the API and the OpenTelemetry instrumentation. The infrastructure required to run the serverless API can be found as part of the api.tf file.

2. Tracetest

The tracetest.tf file contains the different services and dependencies to run the Tracetest server using AWS Fargate.

3. Jaeger

Inside the jaeger.tf file you'll find the required services to run them all in one instance using AWS Fargate.

AWS Network

To connect all of the services, the example generates a VPC network which includes private and public subnets. Internal services like the Postgres RDS instance are protected behind the VPC and only accessible through a node within the internal network.

There are two Application Load Balancers, a public balancer which provides access to the Tracetest task instance through port 11633 and to the Jaeger UI from 16686.

The second is an internal load balancer that allows the instrumented Node.js Lambda functions connect to the Jaeger OTLP endpoint using a DNS instead of an IP.

Node.js Serverless API

The Node.js Serverless API is a simple Lambda function, contained in the src/hello.js file.

The OpenTelemetry tracing is contained in the src/tracing.js file. Traces will be sent to the Jaeger OpenTelemetry Collector Endpoint.

Here's the content of the tracing.js file:

const { Resource } = require("@opentelemetry/resources");
const api = require("@opentelemetry/api");
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");

const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "tracetest",
  }),
});
const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter());

provider.addSpanProcessor(spanProcessor);
provider.register();

registerInstrumentations({
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-aws-lambda": {
        disableAwsContextPropagation: true,
        eventContextExtractor: (event) => {
          event.headers.traceparent = event.headers.Traceparent;
          const eventContext = api.propagation.extract(api.context.active(), event.headers);

          return eventContext;
        },
      },
    }),
  ],
});

The hostname for the Jaeger OTLP Endpoint will be:

  • HTTP: http://<internal_load_balancer_dns_name>:4318

Enabling the tracer is done by adding an environment variable to the lambda function.

NODE_OPTIONS="--require tracing.js"

Tracetest

The tracetest.tf file contains the required services for the Tracetest server which include.

  • Postgres RDS - Postgres is a prerequisite for Tracetest to work. It stores the trace-based tests you create, information about prior test runs and other data that Tracetest needs.
  • Tracetest Task Definition - The information on how to configure and provision Tracetest using ECS.
  • ECS Service - The server provisioning metadata to run the Tracetest Task Definition.
  • Networking - Security groups, target groups and load balancer listeners required to have Tracetest connected to the rest of the AWS infrastructure.

Configuring the Tracetest Container

The Tracetest Docker image supports environment variables as the entry point for the bootstrap configuration, in this case, the task definition includes the following:

{
  "environment" : [
    // POSTGRES CONNECTION
    {
      "name" : "TRACETEST_POSTGRES_HOST",
      "value" : "${module.db.db_instance_address}"
    },
    {
      "name" : "TRACETEST_POSTGRES_PORT",
      "value" : "${tostring(module.db.db_instance_port)}"
    },
    {
      "name" : "TRACETEST_POSTGRES_DBNAME",
      "value" : "${module.db.db_instance_name}"
    },
    {
      "name" : "TRACETEST_POSTGRES_USER",
      "value" : "${module.db.db_instance_username}"
    },
    {
      "name" : "TRACETEST_POSTGRES_PASSWORD",
      "value" : "${module.db.db_instance_password}"
    },
    // DATA STORE PROVISIONING
    {
      "name" : "TRACETEST_PROVISIONING",
      "value" : base64encode(local.provisioning)
    }
  ]
}

Jaeger

Similar to the Tracetest setup, there is a file called jaeger.tf which contains a basic setup to run the all-in-one Jaeger image using AWS Fargate. In this case, it includes networking rules for the internal and external load balancers, so we can provide a way for both the Node.js Lambdas and Tracetest to have access to the API endpoints from within the VPC while providing public access to the UI.

Jaeger OTLP Endpoints

The latest version of Jaeger supports enabling OpenTelemetry endpoints by including the following:

"environment" : [{
  "name" : "COLLECTOR_OTLP_ENABLED",
  "value" : "true"
}],

Jaeger GRPC API Endpoint

As Tracetest uses GRPC to interact with the Jaeger API and load balancers only support the HTTPS protocol for this type of endpoints, we need to include a certificate to the load balancer listener to go around this requirement.

resource "aws_lb_listener" "tracetest-jaeger-api-alb-listener" {
  load_balancer_arn = aws_lb.internal_tracetest_alb.arn
  port              = "16685"
  protocol          = "HTTPS"
  certificate_arn   = aws_acm_certificate.cert.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tracetest-jaeger-api-tg.arn
  }
}

resource "tls_private_key" "tracetest_private_key" {
  algorithm = "RSA"
}

resource "tls_self_signed_cert" "tracetest_self_signed_cert" {
  private_key_pem = tls_private_key.tracetest_private_key.private_key_pem

  subject {
    common_name  = "tracetest.com"
    organization = "Tracetest"
  }

  validity_period_hours = 720

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

resource "aws_acm_certificate" "cert" {
  private_key      = tls_private_key.tracetest_private_key.private_key_pem
  certificate_body = tls_self_signed_cert.tracetest_self_signed_cert.cert_pem
}

How Does Tracetest Reach Jaeger?

The provisioning configuration for Tracetest can be found in the variables.tf file under the locals configuration which uses the internal load balancer DNS endpoint to build the data store entry.

provisioning = <<EOF
---
type: PollingProfile
spec:
  name: default
  strategy: periodic
  default: true
  periodic:
    retryDelay: 5s
    timeout: 10m

---
type: DataStore
spec:
  name: jaeger
  type: jaeger
  jaeger:
    endpoint: ${aws_lb.internal_tracetest_alb.dns_name}:16685
    tls:
      insecure_skip_verify: true
  EOF

  tags = {
    Name    = local.name
    Example = local.name
  }
}

  EOF

How Do Traces Reach Jaeger?

Similar to the Tracetest provisioning configuration, you can find the environment entry in the api.tf file that uses the internal load balancer DNS endpoint for the OTEL_EXPORTER_OTLP_ENDPOINT entry.

environment {
    variables = {
      NODE_OPTIONS                = "--require tracing.js"
      OTEL_EXPORTER_OTLP_ENDPOINT = "http://${aws_lb.internal_tracetest_alb.dns_name}:4318"
    }
  }

Running the Example

To create the AWS infrastructure using Terraform you can simply run:

terraform init \
terraform apply

After accepting the changes after running the terraform apply command and finalizing the infra creation you can find the set of output with the required endpoints to continue with some tests. The final output from the Terraform command should be a list of endpoints that is similar to the following:

Terraform Outputs

Running Trace-based Tests

Now that all of the required services and infra have been created, you can start running some Trace-based testing by doing the following:

  1. From the Terraform output you can copy the api_endpoint and replace the <your_api_endpoint> placeholder from the tests/test.yaml file.
  2. Configure the Tracetest CLI to point to the public load balancer endpoint with tracetest configure --server-url <tracetest_url>.
  3. Run the test YAML file using the CLI tracetest run test -f tests/test.yaml.
  4. Follow the link to find the results.

Learn More

Check out our examples on GitHub and join our Slack Community for more info!