Agency Analytics Kit | Developmi

Eliminate manual reporting across 10 marketing platforms - one pipeline, one dashboard, zero spreadsheets.

Multi-tenant marketing data pipeline - extract data from 10 ad and social platforms, transform it into clean metrics, and visualize it in Metabase. Built for digital agencies managing multiple clients.

mindmap
  ((Agency Analytics Kit))
    Ads
      Meta Ads
      TikTok Ads
      Google Ads
    Organic & Social
      Facebook
      Instagram
      TikTok Organic
      YouTube
      Pinterest
    Analytics & Tagging
      GA4
      GTM
    Stack
      dlt
      PostgreSQL 16
      dbt
      Metabase

Loading

---

Table of contents

• Features
• Connector status
• Quick start
• Architecture
• Docker deployment
• Configuration
• Tests
• Security
• Changelog
• Contributing
• License
• Contact & support

---

Features

• 10 data connectors: Meta Ads, TikTok Ads, Google Ads, Facebook Pages, Instagram Business, TikTok Organic, YouTube Data, Pinterest, GA4, GTM
• Honest status tracking: See Connector status for which integrations are tested live vs. code-complete only
• Multi-tenant by design: Per-client schemas, YAML-based client config, dbt macros for isolation
• No-code connector config: Add/remove platforms per client via YAML, no code changes
• Pipeline orchestration: Health checks, per-client loops, Telegram alerts
• dbt transformations: 24 staging models → intermediate → marts
• Quality gate: 93 tests, ruff linting, mypy type checking
• Docker native: Isolated services, two Docker networks, no vendor lock-in

---

Connector status

Transparency over hype. All 10 connectors are implemented with unit tests and dbt staging models. 3 have been tested against live APIs - the remaining 7 have code complete but need live API validation before production use.

Connector	Type	Tested live	Production ready	Notes
Meta Ads	Ads	✅ Yes	✅ Yes	v25.0, 250+ rows verified
Facebook Page	Organic	✅ Yes	✅ Yes	v25.0, requires Page Access Token (not User Token)
Instagram Business	Organic	✅ Yes	✅ Yes	v25.0, some metrics require metric_type=total_value
TikTok Ads	Ads	❌ No	⚠️ Code + tests	API v1.3, needs live test
Google Ads	Ads	❌ No	⚠️ Code + tests	API v25, needs OAuth setup + live test
TikTok Organic	Organic	❌ No	⚠️ Code + tests	API may have migrated to developers.tiktok.com
YouTube	Organic	❌ No	⚠️ Code + tests	API v3, quota 10k units/day
Pinterest	Organic	❌ No	⚠️ Code + tests	API v5
GA4	Analytics	❌ No	⚠️ Code + tests	Service account auth required
GTM	Analytics	❌ No	⚠️ Code + tests	API v2

---

Quick Start

Prerequisites

• Docker + Docker Compose
• uv (Python package manager - recommended) or Python 3.12

1. Clone and configure

git clone https://github.com/Developmi/agency-analytics-kit.git
cd agency-analytics-kit
cp .env.example .env
# Edit .env with your API tokens, database passwords, and Telegram credentials

2. Start infrastructure

make setup-networks   # Create Docker networks
make docker-all        # Start Postgres, pipeline, Metabase

3. Run the pipeline

make setup-env        # Ensure .env exists
make test             # 93 tests should pass
./scripts/pipeline.sh  # Full E2E pipeline (or use cron)

4. Access Metabase

Open http://localhost:3000 and connect the metabase_reader Postgres user.

---

Architecture

• Ingestion: Each platform has a standalone dlt script with exponential backoff, rate-limit handling, and token-expiry detection
• Storage: PostgreSQL 16 - raw data per schema (raw_meta, raw_facebook, etc.), transformed in staging/intermediate/marts
• Transformation: dbt with multi-tenant macros - generate_schema_name routes to client_<id> schemas
• Visualization: Metabase connected as metabase_reader (read-only Postgres user)
• Orchestration: ./scripts/pipeline.sh runs nightly via cron, validates Docker health, loops over active clients, sends Telegram summary

See ARCHITECTURE.md for full design.

Project structure

├── main.py                         # CLI entry point
├── Makefile                        # Quality & Docker targets
│
├── services/                       # Docker Compose stacks
│   ├── db/                         #   PostgreSQL 16
│   ├── pipeline/                   #   dlt + dbt worker
│   └── metabase/                   #   Metabase dashboards
│
├── src/                            # Application source
│   ├── connectors/                 #   10 dlt connector scripts
│   ├── dbt_project/                #   27 dbt models + macros
│   └── agency_analytics/           #   CLI module
│
├── clients/                        # Multi-tenant YAML config
│   ├── _template.yml
│   ├── acme.yml
│   └── nike.yml
│
├── scripts/                        # Shell orchestration
│   ├── pipeline.sh
│   └── setup-networks.sh
│
├── tests/                          # 93 mock-based tests
├── .env.example
├── pyproject.toml
└── .gitignore

---

Docker deployment

Build the pipeline image

make docker-build

This uses a multi-stage build:

1. Builder stage: installs Python dependencies via uv sync - caches on pyproject.toml + uv.lock
2. Runtime stage: Python 3.12-slim, non-root user (app, UID 1000), only the virtual env and source code

Start all services

make docker-up-all     # Postgres 16 → Pipeline worker → Metabase

Service isolation

Each service runs in its own Compose stack with separate .env files:

• services/db/.env - database credentials
• services/pipeline/.env - API tokens, dlt/dbt config
• services/metabase/.env - Metabase credentials

Docker networks (agency_analytics_net, agency_internal_net) isolate traffic: Metabase cannot reach the pipeline container.

---

Configuration

See .env.example for the full list of environment variables.

Category	Key variables
Database	POSTGRES_PASSWORD, DESTINATION__POSTGRES__CREDENTIALS__*
Meta Ads	META_ACCESS_TOKEN_{CLIENT}
Google Ads	GOOGLE_ADS_DEVELOPER_TOKEN, GOOGLE_ADS_ACCESS_TOKEN_{CLIENT}
Facebook	FACEBOOK_ACCESS_TOKEN_{CLIENT}
Instagram	INSTAGRAM_ACCESS_TOKEN_{CLIENT}
Pipeline	CLIENTS_DIR, TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID

Security note: Never commit .env files. Use .env.example as a template and keep credentials local.

---

Tests

make test               # Run all tests (93 tests across 10 connectors)
make lint               # ruff check
make typecheck          # mypy
make quality            # Full gate: lint + typecheck + test

All tests are mock-based - no live API calls. Each connector has tests for success paths, rate limits, token expiry, pagination, and error handling.

---

Security

This project follows a coordinated disclosure policy. If you discover a vulnerability, do not open a public issue. See SECURITY.md for reporting instructions and response timelines.

---

Changelog

See CHANGELOG.md for the full version history. The project follows Keep a Changelog and Semantic Versioning.

---

Roadmap

See ROADMAP.md for completed features and upcoming work.

---

Contributing

Contributions are welcome. Please read CONTRIBUTING.md before opening a pull request. This project follows Conventional Commits and the Developmi engineering standard.

---

License

Copyright © 2026 Miguel Lozano | Developmi. All rights reserved. Licensed under the MIT License.

---

Contact & support

Maintained by: Miguel Lozano | Developmi

• Role: Cloud & Infrastructure Engineer | FinOps & Bare Metal Specialist
• Philosophy: Security is not a feature; it is the baseline.
• Website: developmi.com
• GitHub: Miguel Lozano
• Organization GitHub: Developmi
• LinkedIn: Miguel Lozano

---

© 2026 Miguel Lozano | Developmi. All rights reserved.