StreamForge v3.0 — Distributed Mode

StreamForge v3.0 — Distributed Mode

What's New

• NodeManager — node identity, uptime tracking, status reporting
• LeaderElection — one node owns SNS alerts, no duplicate emails
• GET /nodes — view all active cluster nodes, leader status, events per node
• Cluster mode — SINGLE NODE (default) or DISTRIBUTED (multi-EC2)
• Node ID configurable via config.json
• Terraform scripts — infrastructure as code for multi-node AWS setup
• ALB (Application Load Balancer) — distributes ingest traffic across nodes
• 3 x EC2 t2.micro nodes — horizontal scaling under load

New config.json fields

{
"node_id": "node-1",
"cluster_mode": 0
}

How to check cluster status

curl http://localhost:9090/nodes

Terraform deployment

cd terraform
terraform init
terraform apply

StreamForge v2.1 — Custom Alert Thresholds

StreamForge v2.1 — Custom Alert Thresholds

What's New

• Per-metric Z-score threshold — default 3.0 sigma, now fully configurable per metric
• Alert cooldown — minimum time between alerts per metric (prevents alert spam)
• POST /config — set custom threshold and cooldown for any metric via API
• GET /config — view current thresholds for all configured metrics
• DELETE /config/{metric} — reset any metric back to default threshold (3.0)

How to use

Set strict threshold for payment failures (alert at 1.5 sigma, max 1 alert/min)

curl -X POST "http://localhost:9090/config?metric=payment_failures&threshold=1.5&cooldown=60"

Set relaxed threshold for CPU (alert at 4.0 sigma, max 1 alert/5min)

curl -X POST "http://localhost:9090/config?metric=cpu_usage&threshold=4.0&cooldown=300"

View all current thresholds

curl http://localhost:9090/config

Reset metric to default

curl -X DELETE http://localhost:9090/config/cpu_usage

Why this matters

Different metrics need different sensitivity.
A payment failure rate of 0.1% is an emergency.
A CPU spike to 80% might be normal during batch jobs.
v2.1 lets you tune every metric without recompiling.

StreamForge v2.0 — Live Web Dashboard

StreamForge v2.0 — Live Web Dashboard

What's New

• Live web dashboard on port 8090 — no external tools needed
• Real-time throughput chart — events processed vs anomalies over time
• Anomaly log table — last 50 anomalies with metric, source, value, score, votes, timestamp
• Active metrics panel — shows all live metric streams and their sources
• Cyberpunk UI — neon cyan/purple/pink theme, animated scan line, glowing stats
• Red flashing alert banner — fires automatically when AI detects an anomaly
• Auto-refreshes every 5 seconds via JavaScript fetch
• Works in both local mode and AWS mode equally

How to access

Start StreamForge

./streamforge --local

Open in browser

http://localhost:8090

Dashboard Endpoints

GET / → Main dashboard
GET /dashboard → Same as /
GET /api/health → Live pipeline stats (used by dashboard)
GET /api/anomalies → Last 50 anomalies (used by dashboard)
GET /api/metrics-list → Active metric streams (used by dashboard)

StreamForge v1.2 — Multi-Metric Support

StreamForge v1.2 — Multi-Metric Support

Added

• Track unlimited metrics simultaneously — latency, error_rate, cpu_usage etc.
• Per-metric isolation — spike on metric A never affects metric B
• Source grouping — metrics tracked per source field
• GET /metrics/list — returns all active metric names with their sources
• GET /query?metric=X&source=Y — filter by both metric and source

StreamForge v1.1 — Config & Polish

StreamForge v1.1 — Config & Polish

Added

• config.json — all AWS settings now configurable without recompiling (region, bucket, SNS ARN, ports, batch size, workers, zscore threshold)
• GET /version endpoint — returns build version, language, and mode
• Graceful shutdown — flushes all S3 buffers cleanly before exit

Changed

• S3_BUCKET, SNS_ARN, REGION moved from hardcoded main.cpp to config.json
• Version banner now shows v1.1 with config summary on startup

How to upgrade

git pull origin main

Edit config.json with your AWS values

cd build && make streamforge -j$(nproc)

StreamForge v1.0

StreamForge v1.0 — Core Pipeline

What's included

• Lock-free pipeline — 18,274 req/sec sustained
• AI anomaly detection — ZScore, EWMA, IsolationForest (2-of-3 voting)
• Full AWS integration — S3 archiving, DynamoDB logging, SNS alerts
• REST Query API — /health /metrics /anomalies /query
• Local mode — runs on any machine without AWS (--local flag)
• Docker support — docker compose up
• GitHub Actions CI — passing
• Benchmark — 24.6x faster than Python FastAPI on identical hardware

How to run

# Local mode
./streamforge --local

# AWS mode
./streamforge