Botnet Tracker and IP Intelligence Platform

Production-oriented botnet tracking stack with separated ingestion and dashboard services, Redis-based sliding windows, PostgreSQL persistence, pluggable IP intelligence, real-time visualization, alerting, and mitigation exports.

Implemented Architecture

Reverse Proxy / Access Logs
   -> Ingestion Service (Fastify)
      - Validation, normalization, replay protection
      - Cloudflare Logpush endpoint (`/v1/logs/cloudflare`)
      - PostgreSQL write (request_logs)
      - Queue publish (BullMQ/Redis)
   -> Detection Worker
      - Sliding windows + heuristics + weighted scoring
      - IP enrichment cache/provider chain (Geo + external threat intel)
      - PostgreSQL updates + malicious IP upserts
      - Redis pub/sub events
   -> Detection API (Fastify + JWT + SSE)
      - Dashboard summary endpoints
      - Country drilldown
      - Manual block endpoint
      - Mitigation export endpoints
      - Real-time SSE (`/api/stream`) and WebSocket (`/api/ws`)
   -> Alerts Service
      - Threshold/country/ASN alerts
      - Webhook + Slack + Discord integrations
   -> Mitigation Service
      - Cloudflare/Nginx/JSON exports
      - Manual + auto-block hooks

Next.js Dashboard (Tailwind + Leaflet + Recharts)
   - Real-time SSE updates (<1s propagation)
   - World map, leaderboard, ASN clusters, timeline

Project Layout

/services
  /ingestion
  /detection
  /enrichment
  /alerts
  /mitigation
/frontend
  /dashboard
/lib
  /ip
  /scoring
  /redis
/db
/scripts
/tests

Core Detection Coverage

Implemented behaviors:

High-frequency IP behavior (requests/min and burst windows)
Repeated 401/403/404 scan behavior
Distributed low-and-slow heuristics
- many IPs from same ASN (5m unique set)
- same User-Agent across many countries (1h unique set)
Sequential endpoint scanning
Bot fingerprinting
- suspicious UA
- headless UA
- missing headers
- identical request intervals
Country anomalies
- spike detection against recent baseline
- rare-country threshold alerts
- geo-velocity (country changes too quickly per IP)

Scoring Model (0-100)

lib/scoring/config.ts and lib/scoring/model.ts implement modular weighted scoring.

Default weights include:

High RPS: +25
403/401/404 scanning: +20
ASN cluster: +15
Suspicious UA: +20
Geo anomaly: +20
External threat intel (AbuseIPDB/GreyNoise/Shodan/ThreatFox): +25

Thresholds:

Score >= 70: Bot
Score >= 85: High-confidence botnet node

All thresholds are environment-configurable.

Security Controls

Implemented safeguards:

Strict input validation (Zod) on ingestion and control endpoints
Rate limiting on ingestion and dashboard APIs
JWT admin auth for dashboard and mitigation routes
Ingestion token gate (x-ingestion-token)
Replay protection using Redis SET NX TTL keys
Parameterized SQL queries only
Log poisoning mitigation (control-char stripping + length caps)
SSRF mitigation by fixed provider endpoints and no user-driven outbound URLs
No trust in X-Forwarded-For by default (TRUST_PROXY_HEADER=false)
Service separation: ingestion isolated from dashboard API

IP Intelligence Layer

Pluggable providers in lib/ip/providers/:

MaxMind (GeoLite2 city/asn DB)
IP2Location BIN lookup
Cloudflare metadata API (optional)
AbuseIPDB reputation API
GreyNoise community API
Shodan host API (or Shodan InternetDB fallback)
ThreatFox IOC API

Results are cached in Redis (enrich:<ip>).

Automatic sources:

Cloudflare Logpush can push HTTP request logs directly to /v1/logs/cloudflare.
External intel APIs are queried per new IP and cached to avoid repeated lookups.

Real-Time Dashboard

frontend/dashboard provides:

Real-time world traffic map (Leaflet)
Country click -> IP breakdown table
Top attacking IP leaderboard with block button
ASN cluster coordination table
Timeline graph (requests + 403 spikes)
Country comparison chart
Live event feed via SSE

Mitigation Hooks

Implemented outputs/endpoints:

Cloudflare rule export (/api/mitigation/cloudflare, /v1/exports/cloudflare)
Nginx deny export (/api/mitigation/nginx, /v1/exports/nginx)
JSON malicious IP export (/api/mitigation/json, /v1/exports/json)
Manual block endpoint (/api/block, /v1/block)
Optional auto-block threshold stream (AUTO_BLOCK_ENABLED=true)

Alerting

services/alerts supports:

Threshold alerts (score >= configured bot threshold)
Country spike alerts
ASN coordination alerts
Webhook, Slack webhook, Discord webhook delivery
Alert dedup keys in Redis to reduce noise

Local Development

1) Configure environment

cp .env.example .env
cp frontend/dashboard/.env.example frontend/dashboard/.env.local

Set at minimum:

INGESTION_TOKEN
CLOUDFLARE_LOGPUSH_TOKEN (if using Cloudflare Logpush)
JWT_SECRET
ADMIN_PASSWORD

Optional (to enable auto-enrichment from public intel APIs):

ABUSEIPDB_API_KEY
GREYNOISE_API_KEY
SHODAN_API_KEY (or keep SHODAN_USE_INTERNETDB=true)
THREATFOX_ENABLED=true

2) Start dependencies

docker compose up -d postgres redis

3) Install dependencies

npm install
cd frontend/dashboard && npm install && cd ../..

4) Run backend services

npm run dev:backend

5) Run dashboard

cd frontend/dashboard
npm run dev

Open: http://localhost:3000

Full Docker Stack

cp .env.example .env
docker compose up --build

Services:

Ingestion: http://localhost:4001
Detection API: http://localhost:4002
Enrichment: http://localhost:4003
Alerts: http://localhost:4004
Mitigation: http://localhost:4005
Dashboard: http://localhost:3000

Ingestion API Example

curl -X POST http://localhost:4001/v1/logs \
  -H 'content-type: application/json' \
  -H 'x-ingestion-token: change-me-ingestion-token' \
  -d '{
    "sourceService":"edge-proxy",
    "events":[
      {
        "ip":"203.0.113.10",
        "timestamp":"2026-02-19T10:12:33.000Z",
        "userAgent":"python-requests/2.32.3",
        "endpoint":"/admin",
        "method":"GET",
        "statusCode":403,
        "headers":{"accept":"*/*","user-agent":"python-requests/2.32.3"},
        "countryCode":"US",
        "asn":"AS13335"
      }
    ]
  }'

Cloudflare Logpush Ingestion Example

curl -X POST http://localhost:4001/v1/logs/cloudflare \
  -H 'content-type: application/x-ndjson' \
  -H 'x-cloudflare-logpush-token: change-me-cloudflare-token' \
  --data-binary $'{"ClientIP":"203.0.113.12","EdgeStartTimestamp":1739947788123000000,"ClientRequestMethod":"GET","ClientRequestURI":"/admin","EdgeResponseStatus":403,"ClientRequestUserAgent":"curl/8.5.0","ClientCountry":"US","ClientASN":13335}\n'

Expected Cloudflare fields used:

ClientIP
EdgeStartTimestamp (or equivalent timestamp)
ClientRequestUserAgent
ClientRequestURI / ClientRequestPath
ClientRequestMethod
EdgeResponseStatus
ClientCountry
ClientASN

Optional: Create Cloudflare Logpush Job via Script

CLOUDFLARE_API_TOKEN=... \
CLOUDFLARE_ZONE_ID=... \
CLOUDFLARE_LOGPUSH_DESTINATION='https://your-ingestion.example.com/v1/logs/cloudflare?token=change-me-cloudflare-token' \
npm run setup:cloudflare-logpush

Testing and Simulation

Unit tests:

npm test

Simulate 10k bot requests:

npm run simulate:botnet

Simulate distributed botnet across 20 countries:

npm run simulate:distributed

Load test ingestion endpoint:

npm run loadtest:ingestion

Test external intel enrichment on one IP:

npm run enrich:ip -- 1.1.1.1

Performance and Scaling Notes

For 5k req/s ingestion target:

Run ingestion replicas behind L4/L7 load balancer
Increase BullMQ worker concurrency and worker replicas
Use Redis cluster or dedicated high-memory nodes
Partition PostgreSQL by time (request_logs daily partitions)
Add COPY-based bulk ingest path for high-volume sources
Offload long-horizon analytics to ClickHouse (optional)
Introduce Kafka between ingestion and detection for durable buffering (optional)

Production Hardening Guidance

Put ingestion behind mTLS/private network only
Rotate ingestion tokens and JWT secret regularly
Apply WAF controls on dashboard APIs
Add audit logging for all block operations
Enable row-level retention + archival jobs
Use signed webhook requests and retry queues
Add SIEM integration for high-confidence alerts
Add canary detectors for model drift and anomaly hooks (events:anomaly-hooks)

Notes

MaxMind/IP2Location providers are optional and activate only when DB paths are configured.

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
db		db
docker		docker
frontend/dashboard		frontend/dashboard
lib		lib
scripts		scripts
services		services
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
docker-compose.yml		docker-compose.yml
package.json		package.json
tsconfig.base.json		tsconfig.base.json
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Botnet Tracker and IP Intelligence Platform

Implemented Architecture

Project Layout

Core Detection Coverage

Scoring Model (0-100)

Security Controls

IP Intelligence Layer

Real-Time Dashboard

Mitigation Hooks

Alerting

Local Development

1) Configure environment

2) Start dependencies

3) Install dependencies

4) Run backend services

5) Run dashboard

Full Docker Stack

Ingestion API Example

Cloudflare Logpush Ingestion Example

Optional: Create Cloudflare Logpush Job via Script

Testing and Simulation

Performance and Scaling Notes

Production Hardening Guidance

Notes

About

Uh oh!

Releases

Packages

Languages

Legendarylibr/botnet-agg

Folders and files

Latest commit

History

Repository files navigation

Botnet Tracker and IP Intelligence Platform

Implemented Architecture

Project Layout

Core Detection Coverage

Scoring Model (0-100)

Security Controls

IP Intelligence Layer

Real-Time Dashboard

Mitigation Hooks

Alerting

Local Development

1) Configure environment

2) Start dependencies

3) Install dependencies

4) Run backend services

5) Run dashboard

Full Docker Stack

Ingestion API Example

Cloudflare Logpush Ingestion Example

Optional: Create Cloudflare Logpush Job via Script

Testing and Simulation

Performance and Scaling Notes

Production Hardening Guidance

Notes

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages