feat(observability): ship API + UI logs dashboards in Grafana - #46
Merged
Conversation
The "general logs view" in Grafana's Explore is great for one-off greps but terrible as a dashboard — no overview, no level stats, no error aggregation. Two focused Loki-sourced dashboards now provision under the boringstack folder: BoringStack — API logs - Stats (errors / warnings / info / total) using the level_name Loki label that Promtail's Pino pipeline already extracts. - Stacked-bar log volume per minute, colour-coded by level (red for error, orange for warn, blue for info, grey for debug). - "Top error / warn events" table aggregating the Pino `event` field — recurring failures rise to the top, long-tail collapses. - "Routes with the most error/warn logs" table grouped by the `route` field for fast "which endpoint is hurting?" triage. - Live API logs panel (descending, time visible, click-to-expand for trace_id / span_id / requestId structured metadata). - $level template variable for one-click per-level filtering. BoringStack — UI logs - Vite + nginx output isn't structured, so the panels are heuristic: substring-matches on /error|failed|exception|panic|EADDR|ECONN/ for the error stat + per-minute series, and /hmr update|page reload/ for the HMR-activity stat. - Per-container log-volume timeseries makes container restarts (flat- line gaps) visually obvious. - $filter template with presets: All / Errors only / HMR-only / Drop HMR noise — applied as a LogQL fragment to the live stream panel. Both auto-load via the existing dashboards provider (`updateInterval: 30s`), tagged `boringstack,api,logs` and `boringstack,ui,logs`. The provisioner comment in dashboards.yml is updated to list all five baseline dashboards (was three). Verification: both files pass `python -m json.tool`; both reference the pinned `loki` datasource uid; LogQL queries target labels and selectors that Promtail's existing config actually emits (level_name, compose_service, the api-* and ui-* container set). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
agjs
enabled auto-merge (squash)
May 29, 2026 08:36
Three fixes in one commit, all touching the same observability surface. 1. apps-api-openapi-drift workflow: always run (no path gate on the trigger), short-circuit the heavy steps via dorny/paths-filter when no API or schema files changed. The job always reports a status, so it can be required on main without permanently blocking PRs that only touch unrelated paths (which was blocking PR #46 — dashboards touched no API code, so the workflow never ran, and a required check that never reports is a forever-block). Mirrors the pattern already used by infra-compose-validate-compose. 2. Promtail Pino pipeline: the `level` template was comparing Pino's `level` field to numeric strings ("10" → trace, "30" → info, etc.) but apps/api's Pino logger uses a `formatters.level` callback that emits the level as an uppercase string ("INFO", "ERROR"). Every record was matching the fallthrough "unknown", so Grafana couldn't auto-colour by level — a wall of grey text. Lowercase the string and promote as the `level` Loki label (renamed from `level_name`, since `level` is the conventional label Grafana looks for). 3. API logs dashboard: drop the unused $level template variable; rename every level_name reference to level; split the single "Live API logs" panel into "Application logs" (the actually-useful stream — filters out Drizzle's per-query SQL via `!~ "^Query: "`) and a separate "Database queries" panel below it (`|~ "^Query: "`). Drizzle's SQL logging was drowning out everything else; you can still see queries when you want to debug them, but they no longer crowd the application-event stream. 4. UI logs dashboard: the $filter template variable was interpolating raw LogQL fragments (including `|~` operators and quoted regexes), which Grafana's panel editor flagged as a syntax error. Replace with $pattern — a pure regex string interpolated inside `|~ "$pattern"`, default `.+` (matches every non-empty line). Same UX, no syntax error. Verification: all 5 dashboards pass `python -m json.tool`; promtail config validated against grafana/promtail:3.2.1 -check-syntax; workflow YAML parses; full-stack-smoke runs locally via pre-push. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
agjs
added a commit
that referenced
this pull request
May 29, 2026
Catches the operator-facing docs site up with all the infra and correlation work that landed across PRs #45, #46, #47, and #48 — defaults flip + dashboards + structured logging + user/trace correlation + env-driven alert receivers. topics/observability.mdx - "What ships" — five dashboards (was one); Pino pipeline detail; Alertmanager wired with env-driven receivers (was "you wire it yourself"). - "Default dashboards" — full per-dashboard breakdown (API, API logs, UI logs, Postgres, Host) with what each is best at and a pointer to the click-through data links on trace_id / userId / requestId. - "Design choices" — added "on by default", trace_id flow, and env-driven Alertmanager entries; dropped the now-misleading "Prometheus + Loki, not OpenTelemetry" claim (the Sentry tracing side does carry traces). - "Querying" — LogQL examples now use the `level` label (was the broken `level=ERROR` substring match) and include trace_id / userId pivot queries. - "Adding an alert" — points at the single bundled rules.yml and the new Alerts topic. topics/error-tracking.mdx - New "Correlation: GlitchTip ↔ Loki ↔ requestId" section with a mermaid diagram of the trace_id flow (browser → API → Loki + GlitchTip), a description of what flows automatically (trace_id / span_id / userId / requestId), and the three click-through paths (Grafana → GlitchTip via dashboard data links, Grafana → Grafana via the requestId link, GlitchTip → Grafana via the operator-side one-time tag link template). - Design-choices block updated with on-by-default GlitchTip and the user.id / trace_id tagging. topics/alerts.mdx (new) - New topic page promoting the alerts walkthrough to the user-facing docs site. Covers what the 14 default rules fire on, the three receiver formats (Slack, Discord-via-/slack-suffix, generic webhook), an end-to-end curl one-liner to ping a fake alert, how to add rules, and the cost-of-pager-fatigue note. reference/env-vars.mdx - New entries for ALERTMANAGER_SLACK_WEBHOOK_URL, ALERTMANAGER_SLACK_CHANNEL, ALERTMANAGER_WEBHOOK_URL. reference/commands.mdx - "Hot-reload Prometheus rules" — `curl POST /-/reload`. - "Ping a fake alert (verify receiver wiring)" — the verification one-liner from the new Alerts topic. astro.config.mjs - Wire the new Alerts topic into the Topics sidebar between Observability and Provisioning with OpenTofu. Verification: `bun run build` clean, 66 pages built (was 65, now +1 for Alerts), pagefind index built without errors. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
agjs
added a commit
that referenced
this pull request
May 29, 2026
) Two small follow-ups to the recent observability work — both pure hygiene, no behavioural changes. Tempo image pinned by digest ---------------------------- PR #50 introduced `grafana/tempo:2.6.1` as a tag-only reference because OrbStack wasn't running when the commit was prepared. Every other image in the observability stack (prometheus, grafana, loki, promtail, alertmanager, postgres-exporter, node-exporter) is pinned to a specific `@sha256:` digest for supply-chain hygiene. Tempo now matches: grafana/tempo:2.6.1@sha256:ef4384fce6e8ad22b95b243d8fc165628cda655376fd50e7850536ad89d71d50 size-diff workflow github_token ------------------------------- The `bundle-diff` workflow on every PR has been printing a red ✗ with `Parameter token or opts.auth is required` since #46 introduced the dashboard work. The action (`andresz1/size-limit-action@v1.8.0`) initialises Octokit *before* reading the env var GITHUB_TOKEN, so the env-based pass we had wasn't reaching it. Fix: pass the token explicitly as a `with:` input too. The env line stays as belt-and-braces in case a future action version flips back to env. Result: no more spurious red ✗ on PRs that touch apps/ui. Verification: docker compose config dry-run exits 0 with the pinned Tempo image; workflow YAML still parses; pre-push gate green. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "general logs view" in Grafana's Explore is great for one-off greps
but terrible as a dashboard — no overview, no level stats, no error
aggregation. Two focused Loki-sourced dashboards now provision under
the boringstack folder:
BoringStack — API logs
label that Promtail's Pino pipeline already extracts.
error, orange for warn, blue for info, grey for debug).
eventfield —recurring failures rise to the top, long-tail collapses.
routefield for fast "which endpoint is hurting?" triage.
trace_id / span_id / requestId structured metadata).
BoringStack — UI logs
substring-matches on /error|failed|exception|panic|EADDR|ECONN/ for
the error stat + per-minute series, and /hmr update|page reload/
for the HMR-activity stat.
line gaps) visually obvious.
HMR noise — applied as a LogQL fragment to the live stream panel.
Both auto-load via the existing dashboards provider (
updateInterval: 30s), taggedboringstack,api,logsandboringstack,ui,logs. Theprovisioner comment in dashboards.yml is updated to list all five
baseline dashboards (was three).
Verification: both files pass
python -m json.tool; both referencethe pinned
lokidatasource uid; LogQL queries target labels andselectors that Promtail's existing config actually emits
(level_name, compose_service, the api-* and ui-* container set).
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com