Skip to content
BranchesTags

Repository files navigation

Tally

A high-throughput event counting & analytics service — built to stay fast and lose nothing under heavy load.

Tally swallows a firehose of "this happened" events (clicks, views, purchases), counts them all without dropping any, and answers "how many happened?" instantly. It's a small, from-scratch version of the engine behind tools like Google Analytics or Mixpanel.

CI Go License

What is this? (in plain English)

Tally is a counting machine for things that happen on an app or website.

Picture an online store with a "Buy" button:

  • Every time someone clicks Buy, the store sends Tally a tiny message: "someone clicked Buy."
  • Tally's one job is to keep count.
  • Later, the store owner asks: "how many people clicked Buy today?" — and Tally instantly answers: "5,000."

Messages pour in, totals come out. The reason this is a whole project — and not a 10-minute task — is scale: real apps generate events tens of thousands of times per second, non-stop. Counting that fast without slowing down, losing events, or double-counting is the hard, interesting part.

Features

  • High-volume ingest — accepts events fast because it never makes a caller wait on the database (queue in the middle, batched writes behind).
  • Loses nothing — graceful shutdown drains every accepted event; in durable mode, even a SIGKILLed worker loses zero events (there's a script that proves it).
  • Never double-counts — every write is idempotent; retries, crashes, and duplicate sends can't inflate a number.
  • Instant answers — per-minute rollups make "how many today?" a summation over a few hundred rows, not a scan over millions.
  • Unique users, at scale — "how many different people did X today?" answered by a from-scratch HyperLogLog (~16 KB per event/day, ~1% error, verified live at 0.85% against exact counts).
  • Protects itself — per-client rate limits (429) and queue-full backpressure (503 + Retry-After) instead of falling over.
  • Observable — Prometheus metrics, pprof profiling, a provisioned Grafana dashboard, and a built-in live dashboard at /.
  • Honest numbers — BENCHMARKS.md publishes measured results only, with the methodology to reproduce them.

Unique-count accuracy — the from-scratch HyperLogLog run against exact truth from 100 to 1,000,000 distinct users. Error stays near the ~0.8% theoretical bound the whole way (chart generated from internal/hll; the same accuracy is asserted by its test suite):

HyperLogLog estimate vs exact counts, 100 to 1M distinct users, error under ~1%

How it works (the journey of one event)

flowchart LR
    A["Your apps<br/>(send events)"] -->|"click / view / purchase"| B["Ingest API"]
    B -->|"drop onto the conveyor belt"| C[("Queue<br/>memory or Redpanda")]
    C -->|"pulled in batches"| D["Workers"]
    D -->|"one atomic statement:<br/>insert + rollup"| E[("Postgres")]
    Q["Dashboard / Query API"] -->|"how many today?"| E
    B -.->|"rate limit"| R[("Redis")]
Loading
  1. An app sends an event → the Ingest API validates it and drops it on a queue, replying 202 immediately. Accept fast, process later.
  2. Workers pull events off in batches (1,000 at a time or every 200ms) — one database round-trip per batch instead of per event.
  3. Each batch lands in one atomic SQL statement that inserts the raw events, skips duplicates, and increments per-minute rollup counters — counting only rows that were actually inserted, so a replayed batch can't double-count.
  4. The dashboard and query API read the rollups for instant answers.

The same journey as a sequence — notice the 202 returns before anything is written:

sequenceDiagram
    autonumber
    participant App as Your app
    participant API as Ingest API
    participant Q as Queue
    participant W as Worker pool
    participant PG as Postgres
    App->>API: POST /v1/events (or gRPC Publish)
    API->>Q: enqueue
    API-->>App: 202 Accepted — immediately
    Note over Q,W: asynchronously, in batches<br/>(1,000 events or every 200ms)
    W->>Q: pull a batch
    W->>PG: one transaction — insert + dedupe + rollup + HLL
    PG-->>W: committed → counted exactly once
Loading

Two queue backends

QUEUE=memory (default) QUEUE=kafka
What it is Bounded in-process channel Redpanda (Kafka API) broker
Speed Fastest An extra hop (the durability tax)
Survives a crash/restart Queued events die with the process Yes — offsets commit only after a batch is stored, so killed workers' work is redelivered and deduped
Scale shape One process MODE=ingest / MODE=worker run and scale as separate processes

That "commit only after storing, dedupe on replay" pair is the classic at-least-once delivery + idempotent consumer pattern — ADR 0003 explains it and its honest limits.

Durable mode, drawn out — ingest and workers become separate processes that scale independently around the broker:

flowchart TB
    C["App / SDK"]
    subgraph ingest ["Ingest processes — scale with traffic"]
        I1["tally MODE=ingest"]
        I2["tally MODE=ingest"]
    end
    subgraph broker ["Redpanda (Kafka API)"]
        T[("topic: tally.events")]
    end
    subgraph workers ["Worker processes — scale with write load"]
        W1["tally MODE=worker"]
        W2["tally MODE=worker"]
    end
    PG[("Postgres")]
    C -->|HTTP / gRPC| I1
    C -->|HTTP / gRPC| I2
    I1 -->|produce| T
    I2 -->|produce| T
    T -->|consume in batches| W1
    T -->|consume in batches| W2
    W1 -->|insert + rollup + HLL| PG
    W2 -->|insert + rollup + HLL| PG
Loading

Demo

Screen recordings go in docs/media/. To capture them (~5 min): run the app, record with Kap (free), drop the files in docs/media/, and uncomment the images below.

1. Live dashboard — totals and the per-minute chart moving in real time as events arrive.

2. Load test — thousands of events/sec via k6, with latency percentiles printed live.

3. Zero-loss chaos — a worker is SIGKILLed mid-batch; the final count is still exact.

Quick start

Prerequisites: Go 1.22+ and Docker.

make up        # start Postgres + Redis
make migrate   # create tables
make run       # start Tally on http://localhost:8080

Open http://localhost:8080 — that's the live dashboard. Then:

# Send one event
curl -X POST http://localhost:8080/v1/events \
  -H 'Content-Type: application/json' \
  -d '{"event_id":"evt-1","name":"buy_click","distinct_id":"user_42"}'

# Ask how many happened today
curl "http://localhost:8080/v1/counts?event=buy_click"
# => {"event":"buy_click","count_today":1}

# Send the SAME event again — the count stays 1 (idempotency)
curl -X POST http://localhost:8080/v1/events \
  -H 'Content-Type: application/json' \
  -d '{"event_id":"evt-1","name":"buy_click","distinct_id":"user_42"}'

Fire fake traffic and watch the dashboard tick:

make loadtest                                  # ~2,000 events/sec for 10s
go run ./cmd/loadgen -rate 5000 -duration 30s  # heavier, with p50/p95/p99 report
go run ./cmd/loadgen -rate 1000 -dupes 20      # 20% duplicate sends — counts stay exact

Durable mode + the chaos demo

make kafka-up                  # adds Redpanda
make build
make chaos                     # kills a worker mid-stream, proves 0 events lost

Metrics stack

make obs-up                    # Prometheus :9090 + Grafana :3000 (dashboard pre-provisioned)

API

Method Path Purpose
POST /v1/events Ingest one event: {event_id, name, distinct_id, properties?} → 202, 429 (over your rate limit), or 503 (backpressure)
gRPC tally.v1.TallyService/Publish (:9091) Batch ingest for backend SDKs — many events per call, same queue/limits; contract in proto/tally/v1
GET /v1/counts?event=NAME Today's count for one event name
GET /v1/uniques?event=NAME Today's unique-user estimate for one event (HyperLogLog, ~1% error)
GET /v1/stats Today's totals + unique-user estimates per name + last-15-min series
GET / Built-in live dashboard
GET /metrics Prometheus metrics
GET /healthz Liveness
GET /debug/pprof/ Profiling

Configuration (env vars)

Variable Default Meaning
ADDR :8080 HTTP listen address
GRPC_ADDR :9091 gRPC listen address ("" disables; only on instances that accept events)
DATABASE_URL local dev DSN Postgres connection string
QUEUE memory memory or kafka
MODE all all, ingest, or worker (kafka only)
QUEUE_SIZE 100000 Queue capacity (memory) / max buffered records (kafka)
BATCH_SIZE 1000 Events per database write
FLUSH_INTERVAL 200ms Max wait before a partial batch is written
WORKERS 4 Worker goroutines (memory mode)
KAFKA_BROKERS localhost:9092 Comma-separated brokers
KAFKA_TOPIC / KAFKA_GROUP tally.events / tally-workers Topic and consumer group
RATE_LIMIT_RPS 0 (off) Per-client events/sec (X-API-Key or IP)
RATE_LIMIT_BURST 2×RPS Burst allowance (in-memory limiter)
REDIS_ADDR "" Set to enforce the rate limit globally across instances

Architecture notes (technical)

  • Ingest (internal/ingest) — validates, rate-limits, enqueues, returns. Never blocks on Postgres.
  • Queue (internal/queue) — bounded channel, or Kafka producer/consumer (franz-go) behind the same Enqueue contract. Full queue → ErrFull → 503 Retry-After.
  • Workers (internal/worker) — size-or-time batching, bounded retries with backoff, partial-batch flush on shutdown. In kafka mode the consumer commits offsets only post-insert.
  • Store (internal/store) — one CTE does insert + dedupe + rollup atomically; counts derive from actually-inserted rows only. Unique-user sketches update in the same transaction (insert-then-lock upsert, sorted lock order — a shape chosen after a load test surfaced a real deadlock).
  • HLL (internal/hll) — HyperLogLog implemented from scratch (~100 lines), accuracy-tested from 10² to 10⁶ distinct values; see ADR 0005.
  • Shutdown ordering — stop HTTP → drain queue → flush workers → close pool. Accepted events always land.
  • Design decisions — written up as ADRs in docs/adr/: the queue-in-the-middle, delivery semantics ("exactly-once is a lie"), and the rate-limit/backpressure split.

Benchmarks

Methodology, profiling instructions, and result tables live in BENCHMARKS.md. Numbers get published only after they're measured on real hardware — accepted-vs-stored must reconcile to zero loss for a run to count.

Deploying

  • Docker: make docker (multi-stage build, distroless, ~20 MB).
  • Kubernetes: manifests + walkthrough in deploy/k8s.

Roadmap

  • Phase 0 — walking skeleton: receive → store → query
  • Phase 1 — queue, batching workers, atomic rollups, graceful drain, backpressure
  • Phase 2 — durable queue (Redpanda), split ingest/worker, chaos script
  • Phase 3 — publish measured benchmarks + flame graphs + chaos results (tooling ready)
  • Phase 4 — rate limiting, metrics + Grafana, live dashboard, Docker/k8s/CI
  • Phase 5 — unique-user counting via from-scratch HyperLogLog (verified at 0.85% error vs exact)
  • Phase 6 — gRPC batch-ingest endpoint alongside HTTP (same queue, limits, and idempotency)
  • Later — ClickHouse for heavy aggregation, weekly/monthly unique rollups (sketches already merge)

License

MIT © Shreyas Chaudhary

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages