Add monitoring stack: log-viewer, metrics-collector, Python client, CI/CD, and deployment tooling by mattmezza · Pull Request #1 · mattmezza/monlight

mattmezza · 2026-02-12T13:39:42Z

Summary

This PR implements the complete monlightstack monitoring platform, adding three major components and supporting infrastructure:

Log Viewer service (log-viewer/): Zig-based service for Docker log ingestion with JSON parsing, multiline reassembly, cursor tracking, FTS5 full-text search, SSE live tail, and an embedded web UI
Metrics Collector service (metrics-collector/): Zig-based service for metrics ingestion, minute/hour aggregation with percentile computation, data retention, query API with auto-resolution, dashboard endpoint, and an embedded web UI with uPlot charts
Python client (clients/python/): monlightstack package with ErrorClient, MetricsClient, and FastAPI integration (MonlightMiddleware, MonlightExceptionHandler, setup_monlight) — includes comprehensive test suites
CI/CD workflows: GitHub Actions for Python client (test matrix + PyPI publishing) and Zig services (test, Docker build, GHCR push)
Deployment tooling: SQLite backup script with retention and optional S3 upload, upgrade script with rolling restarts and health verification, end-to-end smoke test suite, and deployment config fixes (API key mapping, LOG_LEVEL, memory limits)
Error Tracker enhancements: Web UI for error listing/detail, data retention cleanup for resolved errors, additional test coverage (fingerprinting, auth, rate limiting, retention)

Changes

57 files changed, ~12,200 lines added
25 commits covering iterative development from shared infrastructure through services, client, tests, CI/CD, and deployment

How to Review

This is a large PR covering the full platform build-out. Suggested review order:

shared/ — reusable Zig modules (HTTP router, SQLite, auth, rate limiting, logging)
error-tracker/ — enhancements (web UI, retention, tests)
log-viewer/ — new service
metrics-collector/ — new service
clients/python/ — Python client and tests
.github/workflows/ — CI/CD pipelines
deploy/ — operational scripts and config

- Add plan.md with full implementation roadmap: shared Zig modules, per-service build order, Python client with FastAPI integration, CI/CD - Expand specs.md with resolved design gaps: - Error occurrences table (last 5 per error group) - Log Viewer API key auth and multiline log reassembly - SSE lifecycle (30min max, 15s heartbeat, 5 max concurrent) - Metrics Collector additional indexes and label filtering strategy - Cross-cutting: request limits, concurrency model, graceful shutdown, structured logging, migration strategy, CORS, backup, deployment

Create top-level directories (error-tracker, log-viewer, metrics-collector, clients/python, shared, deploy) with placeholder files. Add comprehensive .gitignore for Zig, SQLite, Python, and Node artifacts. Set up docker-compose.monitoring.yml with correct port mappings, volume mounts, health checks, and network config. Add secrets.env.example template.

Implement the shared SQLite module (shared/sqlite.zig) with connection wrapper, prepared statements, migration runner, and comprehensive tests. Add error-tracker Zig project with build config, Dockerfile, HTTP server skeleton, and database schema (errors + error_occurrences tables with indexes). Fix Zig 0.13 compatibility issues and Docker build context.

Implement shared/config.zig providing reusable env var parsing (string, int, bool, required/optional) and LOG_LEVEL initialization. Add error-tracker/src/config.zig that loads all service-specific env vars (DATABASE_PATH, API_KEY, POSTMARK_*, ALERT_EMAILS, RETENTION_DAYS, BASE_URL, LOG_LEVEL) with proper defaults and required validation. Integrate config into main.zig replacing inline env var reads.

…n tests

…d integration tests

…ests

…ication, and email alerting Implement POST /api/errors with full request lifecycle: parse/validate JSON body (required: project, exception_type, message, traceback; optional: environment, request_url, request_method, request_headers, user_id, extra), compute MD5 fingerprint, and upsert error records with create/increment/reopen semantics. Each ingestion creates an occurrence record with per-request context, trimmed to max 5 per error group. Email alerts via Postmark API fire on new fingerprints only (skipped silently if unconfigured). All 173 tests pass.

… response

… and occurrences

…otent resolution and tests

…project names

… and tests

…n.md

…og level extraction, HTTP skeleton

…sor tracking, rotation detection, ring buffer cleanup, and background polling thread

…route wiring Implements GET /api/logs (container, level, search, since, until, limit, offset), GET /api/containers, GET /api/stats, and enhanced GET /health with log count and last ingest timestamp. All 9 query tests pass.

…ing, heartbeat, and filters Implements Server-Sent Events streaming with database polling for new log entries, optional container/level filters, 30-minute max duration, 15-second heartbeat, and max 5 concurrent SSE connections (503 when exceeded). Each SSE connection runs in its own thread with its own SQLite connection.

@embedfile

…-expand Embeds HTML page via @embedfile with Tailwind CSS styling, container/level filter dropdowns, full-text search with debounce, time range selector, SSE-based live tail mode, and expandable log entries showing full details. Served at GET / before auth middleware. Marks all log-viewer tasks complete.

…board, web UI Implements the full metrics-collector service: - Bootstrap: build.zig, Dockerfile, HTTP server on port 8000 - SQLite database with metrics_raw and metrics_aggregated tables - POST /api/metrics batch ingestion with validation - Minute/hour aggregation engine with p50/p95/p99 percentiles - Data retention cleanup (raw, minute, hourly tiers) - GET /api/metrics query with period/resolution/label filters - GET /api/metrics/names, GET /api/dashboard, GET /health - Web UI dashboard with uPlot charts (metric explorer, latency percentiles) - Auth middleware, rate limiting (200 req/min), 512KB body limit - Background aggregation thread with own SQLite connection - All tests passing (known config test exclusion)

…session 14 learnings

…, and FastAPI integration Create the monlightstack Python package with full implementations of: - ErrorClient (async/sync error reporting with PII filtering) - MetricsClient (buffered metrics with periodic flush) - FastAPI integration (MonlightMiddleware, MonlightExceptionHandler, setup_monlight) - 7 scaffolding tests verifying imports and basic behavior Also mark shared infrastructure tasks as complete in plan.md (implemented inline in each service rather than as separate shared modules).

…g, and fire-and-forget behavior

…er, shutdown, and error handling

…ented in session 15)

…oint normalization, and edge cases

…ng and metrics

… push - Create .github/workflows/zig-services.yml with matrix strategy for all 3 services - Run zig build test for each service with env vars (API_KEY, CONTAINERS) - Build Docker images, verify size < 20MB, push to ghcr.io on main only - Fix config tests to be environment-aware (work with or without API_KEY set) - Fix docker-compose build context for log-viewer and metrics-collector (need repo root context to access shared/ directory)

…hing

…e-test.sh

- Map prefixed API keys from secrets.env (e.g., LOG_VIEWER_API_KEY) to the API_KEY env var each service reads - Add LOG_LEVEL environment variable with info default to all services - Add deploy.resources.limits.memory: 30M to all three services - Remove obsolete docker-compose version field - Document LOG_LEVEL in secrets.env.example

…id type mismatch

…erence, and ops documentation

Copilot

Pull request overview

This PR introduces the full MonlightStack monitoring platform: three Zig microservices (error-tracker, log-viewer, metrics-collector), a Python client with FastAPI integration and tests, CI/CD workflows, and deployment/ops tooling (compose configs, backup/upgrade scripts).

Changes:

Added shared Zig modules for configuration, API-key auth, and request rate/body-size limiting.
Implemented/extended services with embedded web UIs, SQLite schemas/migrations, retention cleanup, and test suites.
Added Python client package (monlightstack) + CI workflows and deployment scripts/configs for running/upgrading the stack.

Reviewed changes

Copilot reviewed 72 out of 78 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`shared/rate_limit.zig`	Shared in-memory rate limiter + Content-Length body-size guard + tests (note: retry-after and invalid Content-Length handling need adjustment).
`shared/config.zig`	Shared env-var configuration helpers and LOG_LEVEL parsing.
`shared/auth.zig`	Shared X-API-Key authentication helper with constant-time compare + tests.
`shared/.gitkeep`	Keeps `shared/` directory in git.
`prompt.md`	Agent runbook / iteration workflow instructions.
`metrics-collector/src/web_ui.zig`	Serves embedded metrics dashboard HTML.
`metrics-collector/src/retention.zig`	Retention cleanup for raw/minute/hour aggregates + tests.
`metrics-collector/src/database.zig`	SQLite schema/migrations + startup/migration tests.
`metrics-collector/src/config.zig`	Env-driven service configuration (retention is hours/days; docs should match).
`metrics-collector/build.zig.zon`	Zig package metadata for metrics-collector.
`metrics-collector/build.zig`	Builds executable, wires shared modules, and defines unit test targets.
`metrics-collector/Dockerfile`	Multi-stage Alpine build/runtime image for metrics-collector.
`looop.sh`	Wrapper script to run an external “opencode” loop (has a shell quoting issue in `--help` usage output).
`log-viewer/src/web_ui.zig`	Serves embedded log viewer HTML.
`log-viewer/src/main.zig`	HTTP server routing + auth/rate-limit/body-size enforcement + SSE tail wiring.
`log-viewer/src/log_level.zig`	Log level extraction heuristics + tests.
`log-viewer/src/database.zig`	SQLite schema with FTS5 + cursor tracking + tests.
`log-viewer/src/config.zig`	Env-driven log-viewer configuration + basic load test.
`log-viewer/build.zig.zon`	Zig package metadata for log-viewer.
`log-viewer/build.zig`	Builds executable, wires shared modules, and defines unit test targets.
`log-viewer/Dockerfile`	Multi-stage Alpine build/runtime image for log-viewer.
`error-tracker/src/web_ui.zig`	Serves embedded error listing/detail pages and path matcher tests.
`error-tracker/src/static/index.html`	Error list UI (date parsing logic should not append an extra `Z`).
`error-tracker/src/static/error_detail.html`	Error detail UI + resolve button (date parsing logic should not append an extra `Z`).
`error-tracker/src/retention.zig`	Deletes old resolved errors + background thread + tests.
`error-tracker/src/rate_limit_test.zig`	Integration tests for rate limiting and body size enforcement.
`error-tracker/src/projects_listing.zig`	`/api/projects` JSON formatter + tests (ensure ArrayList is deinit’d on error paths).
`error-tracker/src/fingerprint.zig`	MD5 fingerprinting from traceback location + extensive tests.
`error-tracker/src/error_resolve.zig`	Resolve endpoint parsing + idempotent resolve logic + tests.
`error-tracker/src/config.zig`	Env-driven error-tracker configuration + basic load test.
`error-tracker/src/auth_test.zig`	Integration tests for X-API-Key behavior and health endpoint exclusions.
`error-tracker/src/.gitkeep`	Keeps `error-tracker/src/` directory in git.
`error-tracker/build.zig.zon`	Zig package metadata for error-tracker.
`error-tracker/build.zig`	Builds executable, wires shared modules, and defines unit/integration test targets.
`error-tracker/Dockerfile`	Multi-stage Alpine build/runtime image for error-tracker.
`deploy/upgrade.sh`	Rolling upgrade script with backups, rebuilds, restarts, and health checks.
`deploy/secrets.env.example`	Example env file for per-service API keys and shared Postmark config.
`deploy/docker-compose.test.yml`	E2E/smoke-test compose stack for all services.
`deploy/docker-compose.monitoring.yml`	Production-ish compose config with ports, env wiring, healthchecks, and memory limits.
`deploy/data/metrics/.gitkeep`	Keeps metrics data dir in git.
`deploy/data/logs/.gitkeep`	Keeps logs data dir in git.
`deploy/data/errors/.gitkeep`	Keeps errors data dir in git.
`deploy/backup.sh`	SQLite `.backup` snapshot script with retention policy and optional S3 upload stub.
`clients/python/tests/test_setup_monlight.py`	FastAPI integration tests for `setup_monlight` wiring and behavior.
`clients/python/tests/test_scaffolding.py`	Basic import/scaffolding tests for the Python package.
`clients/python/pyproject.toml`	Python package metadata, dependencies, and pytest config.
`clients/python/monlightstack/metrics_client.py`	Buffered metrics client with periodic background flush.
`clients/python/monlightstack/integrations/fastapi.py`	Middleware + exception handler + `setup_monlight` convenience wiring.
`clients/python/monlightstack/integrations/__init__.py`	Integrations package marker.
`clients/python/monlightstack/error_client.py`	Async/sync error reporting with sensitive-header filtering.
`clients/python/monlightstack/__init__.py`	Top-level exports and package version.
`clients/python/AGENTS.md`	Developer notes/patterns for the Python client module.
`README.md`	Project documentation, setup guides, API reference, and env var tables (metrics retention units/defaults need to match code).
`.gitignore`	Ignores Zig build artifacts, SQLite files, env files, and Python artifacts.
`.github/workflows/zig-services.yml`	CI for Zig services: tests + docker build + size check + GHCR push on main.
`.github/workflows/python-client.yml`	CI for Python client tests + intended PyPI publish job (publish trigger needs tag events to run).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-12T15:28:23Z

+    import monlightstack
+
+    assert monlightstack.__version__ == "0.1.0"


Module 'monlightstack' is imported with both 'import' and 'import from'.

Suggested change

import monlightstack

assert monlightstack.__version__ == "0.1.0"

from monlightstack import __version__

assert __version__ == "0.1.0"

Copilot · 2026-02-12T15:28:24Z

+from __future__ import annotations
+
+import logging
+from unittest.mock import patch


Import of 'patch' is not used.

Suggested change

from unittest.mock import patch

Copilot · 2026-02-12T15:28:24Z

+import json
+import logging
+import time
+from unittest.mock import patch


Import of 'patch' is not used.

Suggested change

from unittest.mock import patch

Copilot · 2026-02-12T15:28:25Z

+from __future__ import annotations
+
+import time
+from unittest.mock import MagicMock, patch


Import of 'patch' is not used.

Suggested change

from unittest.mock import MagicMock, patch

from unittest.mock import MagicMock

Copilot · 2026-02-12T15:28:25Z

+
+from __future__ import annotations
+
+import asyncio


Import of 'asyncio' is not used.

Suggested change

import asyncio

Copilot · 2026-02-12T15:28:25Z

+from __future__ import annotations
+
+import asyncio
+from unittest.mock import MagicMock, patch


Import of 'MagicMock' is not used.

Suggested change

from unittest.mock import MagicMock, patch

from unittest.mock import patch

Copilot · 2026-02-12T15:28:25Z

+import asyncio
+from unittest.mock import MagicMock, patch
+
+import httpx


Import of 'httpx' is not used.

Suggested change

import httpx

Copilot · 2026-02-12T15:28:26Z

+
+import httpx
+import pytest
+from fastapi import FastAPI, HTTPException


Import of 'HTTPException' is not used.

Suggested change

from fastapi import FastAPI, HTTPException

from fastapi import FastAPI

Copilot · 2026-02-12T15:28:26Z

+        # Extract headers as a plain dict
+        try:
+            request_context["request_headers"] = dict(request.headers)
+        except Exception:
+            pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

# Extract headers as a plain dict

try:

request_context["request_headers"] = dict(request.headers)

except Exception:

pass

# Extract headers as a plain dict (best-effort; failures are non-fatal)

try:

request_context["request_headers"] = dict(request.headers)

except Exception as e:

logger.debug(

"Failed to capture request headers for Monlight error context: %s",

e,

)

Rename the Python package directory from monlightstack/ to monlight/, update all references across README, deploy scripts, test files, plan, progress notes, AGENTS.md, and the metrics dashboard HTML. All 121 Python tests pass after the rename.

mattmezza added 30 commits February 11, 2026 13:21

added loop prompt

fc2f309

added looop

72f721e

removed some lines

c85ffba

add API key authentication with shared auth middleware and integratio…

fce4ed3

…n tests

add rate limiting and body size enforcement with shared middleware an…

484f1b8

…d integration tests

add error fingerprinting algorithm with MD5-based deduplication and t…

f53a432

…ests

add error listing endpoint with query filtering, pagination, and JSON…

702fcd1

… response

add error detail endpoint (GET /api/errors/{id}) with full group data…

dc3d214

… and occurrences

add error resolve endpoint (POST /api/errors/{id}/resolve) with idemp…

9cc6d91

…otent resolution and tests

add projects listing endpoint (GET /api/projects) returning distinct …

f985fa4

…project names

better prompt

c8538de

better branching strategy

6318329

add data retention cleanup for resolved errors with background thread…

5326066

… and tests

mark fingerprinting, auth, and rate limiting tests as complete in pla…

0f4bd14

…n.md

add web UI with error listing and detail pages embedded via @embedfile

b32e9cf

add log-viewer bootstrap: build system, database with FTS5, config, l…

e0a25dd

…og level extraction, HTTP skeleton

add Docker log ingestion with JSON parsing, multiline reassembly, cur…

9198c1e

…sor tracking, rotation detection, ring buffer cleanup, and background polling thread

mark metrics-collector complete in plan.md, update progress.txt with …

2f8a1b5

…session 14 learnings

better prompt

da8bfe0

bette rprompt

3665097

better prompt

3e440c4

mattmezza added 15 commits February 12, 2026 13:14

add ErrorClient unit tests verifying payload formatting, PII filterin…

2660ee4

…g, and fire-and-forget behavior

add MetricsClient unit tests verifying buffering, flush, periodic tim…

7b6ca06

…er, shutdown, and error handling

verify MonlightExceptionHandler meets all acceptance criteria (implem…

557d467

…ented in session 15)

verify MonlightMiddleware with 25 tests covering timing, labels, endp…

7d5b6ff

…oint normalization, and edge cases

add setup_monlight tests verifying single-call wiring of error tracki…

5ab97db

…ng and metrics

add CI/CD workflow for Python client with test matrix and PyPI publis…

751801a

…hing

add end-to-end smoke test suite with docker-compose.test.yml and smok…

f9b79c8

…e-test.sh

add SQLite backup script with retention and optional S3 upload

30e6ad5

update plan.md and progress.txt for backup script completion

a73507d

add upgrade script with rolling restarts and health verification

e91a21a

fix sse_tail.zig: add continue in catch else branch to resolve i64/vo…

eba636f

…id type mismatch

add README with architecture overview, getting started guide, API ref…

f88c689

…erence, and ops documentation

mattmezza requested a review from Copilot February 12, 2026 15:19

Copilot started reviewing on behalf of mattmezza February 12, 2026 15:20 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

mattmezza merged commit adbdd5f into main Feb 12, 2026
10 checks passed

mattmezza mentioned this pull request Apr 9, 2026

fix: SMTP STARTTLS support and non-blocking email alerts #25

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add monitoring stack: log-viewer, metrics-collector, Python client, CI/CD, and deployment tooling#1

Add monitoring stack: log-viewer, metrics-collector, Python client, CI/CD, and deployment tooling#1
mattmezza merged 46 commits into
mainfrom
looop

mattmezza commented Feb 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		import monlightstack

		assert monlightstack.__version__ == "0.1.0"

	from unittest.mock import MagicMock, patch
	from unittest.mock import MagicMock

	from unittest.mock import MagicMock, patch
	from unittest.mock import patch

	from fastapi import FastAPI, HTTPException
	from fastapi import FastAPI

Conversation

mattmezza commented Feb 12, 2026

Summary

Changes

How to Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants