AdPulse

AdPulse is a lightweight Kafka-based ad event streaming pipeline written in Go. It simulates how an ads platform backend can process impressions, clicks, and conversions in real time for campaign analytics and basic traffic-quality detection.

This repository is designed as a realistic backend/infra learning project. It is intentionally small: no frontend, no database, no Kubernetes, and no stream-processing framework beyond Kafka consumers.

Project Overview

AdPulse generates simulated ad events, publishes them to Kafka, and processes the same event stream through independent consumers:

• event-generator produces ad_impression, ad_click, and ad_conversion events.
• Kafka stores events in the ad-events topic, with campaign_id used as the message key.
• analytics-consumer computes campaign metrics such as impressions, clicks, conversions, CTR, and CVR.
• fraud-detector detects simple suspicious click patterns using in-memory sliding windows.
• Invalid or unprocessable events can be written to ad-events-dlq.
• api-service exposes summary data over HTTP.
• Prometheus scrapes service metrics from /metrics endpoints.

Why Kafka?

Kafka fits this use case because ad events are high-throughput, append-only event streams. A click or impression should be published once, then consumed independently by systems that need it.

Kafka helps AdPulse demonstrate:

• High-throughput event streams for impressions, clicks, and conversions.
• Asynchronous processing so producers do not call every downstream service directly.
• Decoupled services, where analytics and fraud detection evolve independently.
• Scalability through partitions, using campaign_id as the partition key.
• Real-time analytics and fraud detection from the same event stream.

System Architecture

flowchart LR
  U[User click or impression] --> G[event-generator]
  G -->|produces ad event| K[(Kafka topic: ad-events)]
  K -->|partition key = campaign_id| P{Kafka partitions}
  P --> A[analytics-consumer<br/>campaign metrics]
  P --> F[fraud-detector<br/>suspicious click signals]
  A -->|invalid or unprocessable events| D[(Kafka topic: ad-events-dlq)]
  F -->|invalid or unprocessable events| D
  A --> AM[Campaign metrics endpoint]
  F --> FA[Fraud alerts endpoint]
  API[api-service] -->|HTTP| AM
  API -->|HTTP| FA
  API --> S[GET /v1/summary]
  PR[Prometheus] -->|scrapes /metrics| G
  PR -->|scrapes /metrics| A
  PR -->|scrapes /metrics| F

Loading

More detail: docs/architecture.md

Event Flow

1. event-generator creates a simulated ad event.
2. The event is published to Kafka topic ad-events.
3. The Kafka message key is campaign_id, which keeps campaign-level events routed consistently by Kafka partitioning.
4. analytics-consumer and fraud-detector consume the same topic using separate consumer groups.
5. Valid events update in-memory campaign metrics or fraud alerts.
6. Invalid or unprocessable events can be written to ad-events-dlq.
7. api-service reads summary data from the analytics and fraud services over HTTP.

Fraud Detection

The fraud detector uses simple in-memory rules. The most recent rule detects rapid repeated clicks from the same user on the same campaign:

If the same user_id clicks the same campaign_id more than 5 times within a 10-second sliding window, AdPulse marks it as suspicious.

This matters in ad systems because repeated clicks from the same user can be a signal of invalid traffic, bot-like behavior, or low-quality engagement. AdPulse treats these as signals, not final enforcement decisions.

Current fraud signals:

• Same IP clicks too many times within a short window.
• Same user clicks the same campaign more than 5 times within 10 seconds.
• Click without a recent impression for the same user and campaign.
• Campaign receives an unusually high click burst.

Observability

Each service exposes Prometheus metrics on /metrics. Prometheus is included in docker-compose.yml and scrapes:

• event-generator
• analytics-consumer
• fraud-detector

Important metrics include:

• adpulse_events_produced_total
• adpulse_events_consumed_total
• adpulse_campaign_impressions_total
• adpulse_campaign_clicks_total
• adpulse_campaign_conversions_total
• adpulse_fraud_alerts_total
• adpulse_suspicious_events_total
• adpulse_consumer_errors_total
• adpulse_fraud_consumer_errors_total

Observability matters because distributed event systems can fail in several places: producers, Kafka, consumers, validation, and downstream APIs. Metrics help show whether events are being produced, consumed, rejected, or marked suspicious. The DLQ also gives invalid events a known place to inspect instead of silently dropping them.

Running Locally

Start the full stack:

make up

Useful endpoints:

• API summary: http://localhost:8080/v1/summary
• Event generator metrics: http://localhost:8081/metrics
• Campaign metrics: http://localhost:8082/v1/campaigns/metrics
• Fraud alerts: http://localhost:8083/v1/fraud/alerts
• Prometheus: http://localhost:9090

Run the smoke test:

make smoke

Inspect DLQ messages:

make dlq

The DLQ script reads from Kafka topic ad-events-dlq and prints message keys, headers, and raw payloads. You can customize it:

MAX_MESSAGES=5 TIMEOUT_MS=10000 make dlq

Run unit tests:

make test

Stop all containers:

make down

Example Metrics

These are example Prometheus metric names exposed by the services:

adpulse_events_produced_total{event_type="ad_click"}
adpulse_events_consumed_total{event_type="ad_impression"}
adpulse_campaign_clicks_total{campaign_id="campaign-01"}
adpulse_fraud_alerts_total{rule="rapid_user_campaign_clicks"}
adpulse_suspicious_events_total{rule="rapid_user_campaign_clicks"}

The exact values depend on the simulated events generated while the stack is running.

Sample API Responses

GET /v1/summary

{
  "service": "AdPulse API",
  "description": "lightweight Kafka ad clickstream demo",
  "analytics": {
    "events_processed": 136,
    "campaigns": {
      "campaign-01": {
        "impressions": 16,
        "clicks": 11,
        "conversions": 0,
        "ctr": 0.6875,
        "cvr": 0
      }
    }
  },
  "fraud": {
    "count": 2,
    "returned": 2,
    "alerts": [
      {
        "rule": "rapid_user_campaign_clicks",
        "campaign_id": "campaign-01",
        "user_id": "user-1234",
        "ip": "203.0.113.42",
        "message": "6 clicks from same user on same campaign in 10s",
        "timestamp": "2026-05-21T20:15:00Z"
      }
    ]
  }
}

GET /v1/campaigns/metrics

{
  "events_processed": 136,
  "campaigns": {
    "campaign-01": {
      "impressions": 16,
      "clicks": 11,
      "conversions": 0,
      "ctr": 0.6875,
      "cvr": 0
    }
  }
}

GET /v1/fraud/alerts

{
  "count": 1,
  "returned": 1,
  "alerts": [
    {
      "rule": "rapid_user_campaign_clicks",
      "campaign_id": "campaign-01",
      "user_id": "user-1234",
      "ip": "203.0.113.42",
      "message": "6 clicks from same user on same campaign in 10s",
      "timestamp": "2026-05-21T20:15:00Z"
    }
  ]
}

Fraud-related Prometheus output uses text format:

adpulse_fraud_alerts_total{rule="rapid_user_campaign_clicks"} 1
adpulse_suspicious_events_total{rule="rapid_user_campaign_clicks"} 1
adpulse_fraud_consumer_errors_total 0

DLQ Inspection

AdPulse includes a lightweight DLQ inspection script:

make dlq

The script consumes from ad-events-dlq using Kafka's console consumer inside the existing Kafka container. It does not add a database or any new service.

Example output:

Inspecting Kafka DLQ topic: ad-events-dlq
Showing up to 10 message(s). Set MAX_MESSAGES or TIMEOUT_MS to customize.

campaign-03 | dlq-reason:event_id is required | {"event_type":"ad_click","campaign_id":"campaign-03"}

The DLQ matters operationally because malformed events should not stop the consumer loop. Keeping failed messages in a separate Kafka topic makes them visible for debugging.

Kafka Replay Demo

Kafka tracks consumer progress with offsets. A consumer group offset records the last position read by that group in each topic partition.

Replay means moving a consumer group's offsets backward so it can read older events again. This is useful when testing new processing logic or recovering from a consumer bug.

Safe local demo workflow:

make up
docker compose stop analytics-consumer
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --group adpulse-analytics-consumer \
  --topic ad-events \
  --reset-offsets \
  --to-earliest \
  --dry-run

If the dry run looks correct, execute the reset and restart the consumer:

docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --group adpulse-analytics-consumer \
  --topic ad-events \
  --reset-offsets \
  --to-earliest \
  --execute

docker compose start analytics-consumer

This is a local learning workflow, not enterprise replay infrastructure. It demonstrates the core Kafka idea: events can remain in Kafka, and consumer groups decide where to resume.

Interview Talking Points

• Kafka is used instead of direct DB writes because multiple downstream systems can consume the same event stream independently.
• campaign_id is the partition key because campaign-level ordering is useful for campaign metrics.
• Separate consumer groups allow analytics and fraud detection to process the full stream independently.
• The DLQ exists so invalid events are preserved for inspection instead of blocking the consumer loop.
• Asynchronous event systems scale better because the producer does not need to synchronously call every downstream service.
• Fraud detection fits ad platforms because clicks and impressions affect reporting quality, billing accuracy, and advertiser trust.
• Replayability matters because a consumer group can reprocess earlier Kafka events after a bug fix or logic change.
• Observability helps debug whether events are being produced, consumed, rejected, or marked suspicious.
• The MVP uses in-memory state to keep the project focused on Kafka concepts, not storage infrastructure.

Future Improvements

• Schema registry.
• Persistent storage.
• Real-time dashboard.
• Attribution pipeline.
• Stronger DLQ inspection and alerting.
• ML-based fraud scoring.