Observability for Developers — Go, Microservices & Datadog

This repository is the foundation for a tutorial series focused on observability for developers.

The goal of the series is to take a fully blind distributed system and evolve it step by step into a production-grade, developer-friendly observable system, using:

• Go
• Microservices
• OpenTelemetry
• Datadog

This project is intentionally simple in domain, but realistic in architecture.

🎯 What Is This Project?

This is a small event-driven microservice system built in Go.

It simulates a basic business workflow:

1. A customer places an order
2. The payment is processed
3. A notification is sent to the customer

The important part is not what the system does, but how it is structured.

From Part 0 onward, the system is:

• Distributed
• Asynchronous
• Containerized
• Completely unobservable

That is intentional.

🧱 Architecture Overview

The system consists of three microservices:

order-service

• Exposes an HTTP API
• Endpoint: POST /orders
• Emits event: order.created

payment-service

• Subscribes to order.created
• Simulates payment processing
• Emits event: payment.completed

notification-service

• Subscribes to payment.completed
• Simulates sending a notification

Communication

• HTTP (synchronous) for ingress
• NATS (pub/sub) for async communication

There is no shared state and no direct service-to-service calls.

🧠 Philosophy of the Series

This repository intentionally starts in a bad observability state:

• Logs are unstructured
• No request IDs
• No correlation across services
• No tracing
• No metrics

Each tutorial part will introduce one observability layer at a time, clearly showing the value added by that layer.

This is not a “copy-paste observability setup”. It is a learning journey.

📂 Repository Structure

.
├── Dockerfile
├── docker-compose.yml
├── go.work
├── pkg/
│   └── nats/
└── services/
├── order-service/
│   ├── cmd/
│   └── internal/
├── payment-service/
│   ├── cmd/
│   └── internal/
└── notification-service/
├── cmd/
└── internal/

• Each service follows standard Go layout (cmd/, internal/)
• A single generic Dockerfile is used for all services
• Services are parameterized via build arguments

🐳 Containers & Build Strategy

• One Dockerfile at the repository root
• Services are built using SERVICE_NAME as a build argument
• docker-compose orchestrates the system locally

This avoids duplication and keeps the build logic explicit and teachable.

▶️ Running the Project

Prerequisites

• Docker
• Docker Compose
• Go 1.23+ (optional, only for local builds)

Start the system

From the repository root:

docker-compose up --build

This will start:

• NATS
• order-service
• payment-service
• notification-service

Create an order

Send a request to the order service:

curl -X POST http://localhost:8081/orders \
  -H "Content-Type: application/json" \
  -d '{
    "customer_email": "alice@example.com",
    "amount": 49.99
  }'

Example response:

ord_123456

Observe the logs

You will see logs from different services, but:

• There is no correlation
• There is no request flow visibility
• There is no traceability

This is the baseline.

🚧 Current State (Part 0)

At this stage, the system is:

• ✅ Functional
• ✅ Distributed
• ❌ Observable

This is the starting point for the tutorial series.

➡️ What Comes Next

The next parts of the series will introduce:

1. Intentional logging
2. Structured logs
3. Request ID propagation
4. Distributed tracing (OpenTelemetry)
5. Log ↔ trace correlation
6. Domain-level metrics
7. Developer-oriented dashboards

Each step builds on top of this exact codebase.

📌 Who Is This For?

This project is for developers who:

• Work with Go and microservices
• Want to understand observability from first principles
• Prefer incremental learning over copy-paste solutions
• Care about debuggability and developer experience

📄 License

MIT