DevOps Pipeline Project

Project Overview

This repository is a DevOps coursework project that demonstrates modern cloud-native development practices through a fully functional microservices data pipeline.

What this project showcases:

• 🔄 Microservices Architecture — A data processing pipeline with 4 interconnected services (Maker → Hasher → Trimmer → Store) communicating via Redis pub/sub and queues
• 🚀 GitLab CI/CD — Change-based pipeline automation that builds, tests, and deploys only the modified components
• ☸️ Kubernetes Deployment — Helm charts for each service with Helmfile for orchestrated full-stack deployment
• 📊 Full Observability — Distributed tracing (Jaeger), metrics collection (Prometheus), and dashboards (Grafana)

The pipeline processes data through a series of transformations: random data generation → SHA256 hashing → hash trimming → PostgreSQL persistence, with complete end-to-end tracing and metrics at each step.

Table of Contents

• Architecture Overview
• CI/CD Pipeline Architecture
• Components
• Observability
• Quick Start
• Documentation

---

Architecture Overview

Data Pipeline Flow

The system implements a data processing pipeline with the following flow:

graph LR
    A[Maker CronJob] -->|POST /hash| B[Hasher Service]
    B -->|Pub/Sub: hasher_output| C[Trimmer Service]
    C -->|Queue: trimmer_output| D[Store Service]
    D -->|Persist| E[(PostgreSQL)]
    B -.->|Uses| F[(Redis)]
    C -.->|Uses| F
    D -.->|Uses| F

Loading

Components:

1. Maker - CronJob that generates random data and sends it to Hasher
2. Hasher - Calculates SHA256 hashes and publishes to Redis Pub/Sub
3. Trimmer - Subscribes to hash messages, trims data, and pushes to Redis queue
4. Store - Consumes from Redis queue and persists to PostgreSQL
5. Redis - Message broker for Pub/Sub and queue operations
6. PostgreSQL - Persistent storage for processed data

---

CI/CD Pipeline Architecture

Overview

The CI/CD pipeline uses GitLab CI/CD with a change-based trigger system that enables individual component deployment and testing. Only the components that have changed are built, tested, and deployed.

Main Pipeline Structure

The root .gitlab-ci.yml defines triggers for each component:

stages:
  - triggers

trigger-hasher:
  stage: triggers
  trigger:
    include: /services/hasher/.gitlab-ci.yml
  rules:
    - changes:
      - services/hasher/*
      - charts/hasher/*

Change Detection: Each trigger only activates when files in the specific component directory change.

---

Component Pipeline Workflow

Each microservice follows a 5-stage pipeline:

graph LR
    A[Preparation] --> B[Test]
    B --> C[Build]
    C --> D[Deliver]
    D --> E[Deploy]

Loading

Stage 1: Preparation

• Generates unique BUILD_ID based on timestamp
• Creates image naming conventions
• Exports variables for subsequent stages via artifacts

Artifacts: context.env file with build metadata

Stage 2: Test

Two parallel test jobs:

Unit Tests:

• Runs in Docker-in-Docker environment
• Executes component-specific unit tests
• Uses test Dockerfile for isolated testing

Functional Tests:

• Validates component functionality
• Runs integration scenarios

Stage 3: Build

• Uses Kaniko for secure, rootless image building
• Builds Docker image from component Dockerfile
• Pushes to GitLab Container Registry
• Naming: <image-name>-<environment>-<commit-sha>:<build-id>

Stage 4: Deliver

• Pulls built image from GitLab Registry
• Re-tags and pushes to ttl.sh (temporary registry)
• TTL: Images expire after 2 hours
• Enables deployment without long-term registry storage

Stage 5: Deploy

• Uses Helm to deploy to Kubernetes
• Resource Lock: production-lock ensures sequential deployments
• Strategy: Each service is deployed as its own Helm release
• Updates the specific service chart with new image tag

Deployment Command:

helm upgrade --install <service-name> ./charts/<service-name> \
  --namespace default \
  --set image.repository="<image-repo>" \
  --set image.tag="<image-tag>"

---

Individual Component Testing

Each component can be tested independently:

Services (Hasher, Maker, Store, Trimmer)

Template: services/.gitlab-ci-template.yml

Configuration Example:

include:
  - local: 'services/.gitlab-ci-template.yml'
    inputs:
      image-name: 'hasher'
      environment: 'prod'
      directory-name: 'services/hasher'

Individual Stages:

1. Unit tests via Docker container
2. Functional tests
3. Image build and registry push
4. Deployment to Kubernetes cluster

Functional Tests Overview:

Each service has a dedicated functional test suite that runs in a Docker environment using testcontainers. These tests verify the service's integration with its dependencies (Redis, PostgreSQL, other services).

• Maker Service:

  • Goal: Verify message generation and delivery.
  • Flow: Starts Hasher and Redis containers -> Starts Maker -> Maker generates a message -> Sends POST to Hasher -> Hasher publishes to Redis.
  • Verification: The test subscribes to Redis to ensure the message was successfully received and processed by the Hasher.

• Hasher Service:

  • Goal: Verify hashing logic and pub/sub communication.
  • Flow: Starts Redis container -> Starts Hasher -> Test sends a POST request with data -> Hasher calculates SHA256 hash -> Publishes to hasher_output channel.
  • Verification: The test subscribes to the Redis channel and asserts that the received message contains the correct hash and metadata.

• Trimmer Service:

  • Goal: Verify hash trimming and queue operations.
  • Flow: Starts Redis container -> Starts Trimmer -> Test publishes a message to hasher_output channel (simulating Hasher) -> Trimmer receives, trims hash to 10 chars, and pushes to trimmer_output queue.
  • Verification: The test pops from the Redis queue and asserts that the hash is correctly trimmed.

• Store Service:

  • Goal: Verify data persistence.
  • Flow: Starts Redis and PostgreSQL containers -> Starts Store -> Test pushes a message to trimmer_output queue (simulating Trimmer) -> Store consumes and inserts into Postgres.
  • Verification: The test queries the PostgreSQL database to confirm the record exists with the correct data.

Charts (Redis, PostgreSQL)

Template: charts/.gitlab-ci-template.yml

Configuration Example:

include:
  - local: 'charts/.gitlab-ci-template.yml'
    inputs:
      chart-name: 'redis'
      directory-name: 'charts/redis'

Workflow:

• Detects chart changes (values.yaml, templates, etc.)
• Deploys updated chart to Kubernetes
• Uses Helm upgrade with pipeline chart

---

Deployment Scenarios

Scenario 1: Service Code Change

What happens:

1. Developer updates services/hasher/src/index.ts
2. Pipeline detects change in services/hasher/*
3. Triggers trigger-hasher job
4. Runs: Preparation → Tests → Build → Deliver → Deploy
5. Deploys with: helm upgrade --install hasher ./charts/hasher
6. Only the hasher release is updated in Kubernetes

Other services: Remain unchanged

Scenario 2: Chart Configuration Change

What happens:

1. Developer updates charts/hasher/values.yaml
2. Pipeline detects change in charts/hasher/*
3. Triggers trigger-hasher job
4. Runs full pipeline including image rebuild
5. Deploys with: helm upgrade --install hasher ./charts/hasher
6. Updates the hasher release with new configuration

Image: Rebuilt and deployed with updated chart configuration

Scenario 3: Infrastructure Chart Change

What happens:

1. Developer updates charts/redis/values.yaml
2. Pipeline detects change in charts/redis/*
3. Triggers trigger-redis job
4. Deploys with: helm upgrade --install redis ./charts/redis
5. Updates the Redis release with new configuration

Services: Unaffected

Scenario 4: Multiple Components Changed

What happens:

1. Developer updates both hasher and trimmer services
2. Both trigger-hasher and trigger-trimmer activate
3. Pipelines run in parallel
4. Deployment stage uses resource_group: production-lock
5. Deployments execute sequentially to avoid conflicts

---

Key Features

Change-Based Activation

Only components with file changes trigger their pipelines

Parallel Testing

Unit and functional tests run concurrently

Isolated Builds

Each component builds independently using Kaniko

Safe Deployments

• Resource locks prevent race conditions
• Each service deployed as independent release
• Helm rollback capability per service

Ephemeral Images

ttl.sh provides temporary image storage without registry bloat

---

Components

Services

• Hasher - SHA256 hash calculation service
• Maker - Data generation CronJob
• Trimmer - Hash trimming service
• Store - Data persistence service

Infrastructure

• Redis - Message broker and cache
• PostgreSQL - Relational database

Helm Charts

• Charts Directory - Individual Helm charts for each service and infrastructure component

---

Observability

The pipeline is fully instrumented with distributed tracing and metrics collection using industry-standard tools.

Distributed Tracing

• Technology: OpenTelemetry SDK with Jaeger exporter
• Protocol: Jaeger Thrift HTTP (port 14268)
• Trace Propagation: W3C Trace Context via message payloads

Each message carries a traceContext field enabling end-to-end tracing from maker → hasher → trimmer → store. See each service README for detailed message formats including the traceContext structure.

graph LR
    A[Maker] -->|trace-id| B[Hasher]
    B -->|trace-id via Redis| C[Trimmer]
    C -->|trace-id via Redis| D[Store]
    B & C & D -->|spans| E[Jaeger]

Loading

Prometheus Metrics

Each service exposes a /metrics endpoint with:

Metric	Description
*_processing_time_seconds	Processing time per service
*_messages_processed_total	Successful message count
*_messages_errors_total	Error count
pipeline_end_to_end_time_seconds	Full pipeline latency

Required Metrics (from Assignment)

1. Tiempo promedio de procesamiento end-to-end - Pipeline latency from maker to store
2. Delay promedio en cada cola de Redis - Redis pub/sub and queue latencies
3. Cantidad de mensajes procesados - Success/error counts per service

Access Dashboards

# Grafana (metrics visualization)
kubectl port-forward svc/kube-prometheus-stack-grafana -n monitoring 3000:80

# Prometheus (query interface)
kubectl port-forward svc/kube-prometheus-stack-prometheus -n monitoring 9090:9090

# Jaeger (distributed traces)
kubectl port-forward svc/jaeger-query -n monitoring 16686:16686

For detailed PromQL queries and configuration, see Observability Guide.

---

Quick Start

Deploy Everything with Observability (Recommended)

The easiest way to deploy the entire stack including the observability suite (Prometheus, Grafana, Jaeger) is using helmfile:

# Install helmfile if not already installed
# https://github.com/helmfile/helmfile#installation

# Deploy all components + observability
helmfile sync

This single command deploys:

• Observability: kube-prometheus-stack (Prometheus + Grafana) + jaeger
• Infrastructure: redis + postgres
• Services: maker, hasher, trimmer, store

See helmfile.yaml for the complete configuration.

Deploy Services Individually (Without Observability Stack)

If you already have Prometheus/Jaeger or want to deploy services manually:

# Deploy infrastructure
helm install redis ./charts/redis
helm install postgres ./charts/postgres

# Deploy services
helm install hasher ./charts/hasher
helm install trimmer ./charts/trimmer
helm install store ./charts/store
helm install maker ./charts/maker

Deploy Individual Component

# Deploy only hasher service
helm install hasher ./charts/hasher

# Deploy only Redis
helm install redis ./charts/redis

Verify Deployment

# Check all pods
kubectl get pods

# Check pipeline data flow
kubectl port-forward svc/store 3002:3002
curl http://localhost:3002/stats

---

Documentation

• Service READMEs - Detailed service documentation with API endpoints
• Helm Chart READMEs - Deployment and upgrade instructions
• Observability Guide - Tracing, metrics, and PromQL queries
• GitLab CI Templates - Pipeline configuration reference

---

Environment Variables

Each component can be configured via Helm values or environment variables. See individual component READMEs for specific configuration options.

Testing Locally

You can run tests locally using the provided helper scripts. These scripts handle building the Docker images and running the tests in the correct environment.

Run Unit Tests

To run unit tests for a specific service:

# Example for Hasher service
./services/hasher/.scripts/unit/test.sh

# Example for Maker service
./services/maker/.scripts/unit/test.sh

Run Functional Tests

To run functional (integration) tests for a specific service:

# Example for Hasher service
./services/hasher/.scripts/functional/test.sh

# Example for Store service
./services/store/.scripts/functional/test.sh

Local Development

See individual service READMEs for local development instructions.

---

License

This project is for educational purposes as part of a DevOps coursework.