diff --git a/assets/img/alert-create-dark.png b/assets/img/alert-create-dark.png new file mode 100644 index 0000000..fc5551d Binary files /dev/null and b/assets/img/alert-create-dark.png differ diff --git a/assets/img/alert-create.png b/assets/img/alert-create.png new file mode 100644 index 0000000..d6c71f4 Binary files /dev/null and b/assets/img/alert-create.png differ diff --git a/assets/img/alert-detail-dark.png b/assets/img/alert-detail-dark.png new file mode 100644 index 0000000..671a63e Binary files /dev/null and b/assets/img/alert-detail-dark.png differ diff --git a/assets/img/alert-detail.png b/assets/img/alert-detail.png new file mode 100644 index 0000000..21adf0a Binary files /dev/null and b/assets/img/alert-detail.png differ diff --git a/assets/img/alert-list-dark.png b/assets/img/alert-list-dark.png new file mode 100644 index 0000000..1610140 Binary files /dev/null and b/assets/img/alert-list-dark.png differ diff --git a/assets/img/alert-list.png b/assets/img/alert-list.png new file mode 100644 index 0000000..ec51a0f Binary files /dev/null and b/assets/img/alert-list.png differ diff --git a/content/automation/alerts.md b/content/automation/alerts.md index 9b5ea77..b8bdb0b 100644 --- a/content/automation/alerts.md +++ b/content/automation/alerts.md @@ -23,6 +23,8 @@ lead: 'An alert rule watches one metric on one deployment and fires when it stay ## Create a rule +{{< shot src="/img/alert-list.png" url="console.deploys.app/alert?project=acme" alt="The alert rules list with ok, firing, nodata, and disabled statuses" caption="Each rule shows its target, condition, status, and last evaluated value." >}} + From the console, open **Alerts** and click **Create rule**. Or use the CLI: ```bash @@ -57,6 +59,8 @@ minute is tolerated — see [When a rule fires](#when-a-rule-fires). | **Renotify** | Re-send `alert.trigger` every N minutes while still firing. `0` disables it (notify only on transitions) — see [Renotify](#renotify). | | **Disabled** | A disabled rule keeps its config but stops evaluating. Saving any edit — including disabling — resets the rule's status to `ok`, so it starts fresh when re-enabled. | +{{< shot src="/img/alert-create.png" url="console.deploys.app/alert/create?project=acme" alt="The create-alert-rule form" caption="Pick a deployment, a metric, a threshold, and how long it must hold. Delivery is configured on notification channels, not on the rule." >}} + ## Metric vocabulary | `Metric` | Meaning | Threshold unit | Bucket aggregation | @@ -65,6 +69,8 @@ minute is tolerated — see [When a rule fires](#when-a-rule-fires). | `memory` | Memory usage as a share of the deployment's limit, averaged across pods | percent (same headroom as `cpu`) | avg per minute | | `requests` | Request rate, summed across pods | requests/min | sum per minute | | `egress` | Egress traffic, summed across pods | bytes/min | sum per minute | +| `value` | A [custom metric](/deployments/custom-metrics/) gauge (kind=custom) | the series unit | avg per minute | +| `rate` | A custom-metric counter's per-minute increase (kind=custom) | series unit / min | max−min per minute | `cpu` and `memory` are computed the same way the Metric tab's chart lines are — `avg(usage) / avg(limit)` per one-minute bucket — so the threshold you set lines @@ -112,7 +118,11 @@ sample doesn't flap it straight back to firing. | `nodata` | Not enough recent data to evaluate — the deployment is stopped or deleted, or (for `cpu`/`memory`) has no limit set. Does not notify, and does not resolve an active `firing` alert. | The console list and detail pages show a rule's current status, last evaluated -value, and — while firing — how long it's been firing. From the CLI: +value, and — while firing — how long it's been firing: + +{{< shot src="/img/alert-detail.png" url="console.deploys.app/alert/detail?project=acme&name=api-memory-high" alt="An alert rule detail page while firing, with evaluator state and transition history" caption="The detail page shows the current value, how long the rule has been firing, and the 30-day transition history." >}} + +From the CLI: ```bash deploys alert list --project acme diff --git a/content/automation/notification-channels.md b/content/automation/notification-channels.md index 631dd16..08bc69b 100644 --- a/content/automation/notification-channels.md +++ b/content/automation/notification-channels.md @@ -258,6 +258,38 @@ auto-detected failure. See [Monitoring → react to failures without polling](/deployments/monitoring/) for the end-to-end agent loop this enables. +## Metric alerts: `alert.trigger` and `alert.resolve` + +A [metric alert rule](/automation/alerts/) has no delivery config of its own — +it reuses these channels. When the rule's rolling window becomes a breach, or +clears, the platform emits a notification (not an audit row), the same +notify-only path as `deployment.health`: + +| Event | Fires when | Outcome | +|---|---|---| +| `alert.trigger` | the rule's window is a breach, or a renotify while it is still firing | `failure` (red in Discord) | +| `alert.resolve` | the metric is back inside the threshold | `success` (green in Discord) | + +Subscribe a channel to those two names, or to `alert.*` (which also matches +audited rule create / update / delete): + +```bash +deploys notification create --project acme --name alerts-discord \ + --type discord \ + --url https://discord.com/api/webhooks/123/abc \ + --event alert.trigger --event alert.resolve +``` + +`alert.trigger` / `alert.resolve` are written by the **system**, not a user, so +they carry an **empty actor** (`actorType` `user`, no email). The `message` is +a one-line summary of the condition and the value that crossed it, e.g. +`web: cpu >= 90% for 10m (current 94.2%)`. A project with no matching channel +still evaluates the rule and still shows `firing` in the console — it just has +nowhere to send the page. + +See [Alerts](/automation/alerts/) for the metric vocabulary, window semantics, +and how to create a rule. + ## Test and the delivery log Use **Send test** (console) or `notification test` to deliver a synthetic change diff --git a/content/deployments/custom-metrics.md b/content/deployments/custom-metrics.md new file mode 100644 index 0000000..1587544 --- /dev/null +++ b/content/deployments/custom-metrics.md @@ -0,0 +1,149 @@ +--- +title: 'Custom metrics' +linkTitle: 'Custom metrics' +weight: 9 +description: 'Scrape your own Prometheus /metrics endpoint, chart the series, and alert on them.' +lead: 'A metric source tells the platform to scrape a Prometheus text endpoint on one of your deployments once a minute. The series are stored for 30 days, charted in the console, and can be the target of an alert rule. Sources are project-scoped and run on Deploys.app — there is nothing extra to host.' +--- + +## What you get + +- **Own-deployment scrape** — the collector inside the location GETs + `http://:` on your deployment. You pick the deployment, + port, and path (`/metrics` by default). There is no URL field; the platform + builds the in-cluster address so nothing outside your project can be scraped. +- **Gauges and counters** — Prometheus gauges, counters, and untyped series are + stored. Histogram and summary families (including `_bucket`) are dropped. +- **Hard caps** — 4 sources per project, 100 series per source. Hitting the + series cap marks the source **truncated** instead of silently dropping extras. +- **Charts** — the same line-chart shape as platform metrics, over 1h / 6h / + 12h / 1d / 7d / 30d. +- **Alerts** — a rule can target `kind=custom` with a source + exact series key + and metric `value` (gauge) or `rate` (counter, per-minute increase). The + [alert window](/automation/alerts/#when-a-rule-fires) is the same as for + CPU / memory / requests / egress. + +## Create a source + +From the console, open **Metric sources** and create a source. Or use the CLI: + +```bash +deploys metricsource set \ + --project acme \ + --name web \ + --location gke.cluster-rcf2 \ + --deployment web \ + --port 9090 \ + --path /metrics +``` + +`set` is an upsert: the first call creates the source, later calls replace the +config. Path defaults to `/metrics`. + +### Fields + +| Field | Description | +|---|---| +| **Name** | A project-unique name (lowercase, e.g. `web`). | +| **Location** | The location the target deployment runs in. | +| **Deployment** | Your own deployment in that location. External hosts are not allowed. | +| **Port** | 1–65535. Must be reachable on the deployment's in-cluster Service. | +| **Path** | Path only, leading `/`, no host, no `://`. Default `/metrics`. | +| **Disabled** | Keep the config but skip scraping. | + +{{< callout type="note" >}} +The scrape target is always in-cluster DNS derived from the deployment — never +a free-form URL. That is what keeps this from becoming an SSRF trampoline. +{{< /callout >}} + +## Series identity + +Each sample is stored under `name{sortedLabels}`, for example +`queue_depth{queue="email"}`. Labels are sorted by name. A series with no +labels is just `name`. + +On each scrape the collector keeps gauges, counters, and untyped values, and +drops histogram/summary families. Counters are stored **raw** (monotonic); +rate is computed at query / alert time as a per-minute increase, with resets +clamped at zero. + +If a scrape would add an 101st series, the extra series are not stored and the +source is marked **truncated**. The console shows a banner; nothing is dropped +quietly. + +## Charts + +Query a source over the same short windows as cache/WAF activity (1h, 6h, 12h, +1d, 7d, 30d): + +```bash +deploys metricsource query \ + --project acme \ + --name web \ + --series 'queue_depth{queue="email"}' \ + --timerange 1h +``` + +Empty `--series` lets the server pick the most recently seen series (capped at +100). Gauges average inside each bucket; counters use `max − min` per bucket +(never negative). + +A deployment's Metric tab also grows a **Custom** section when a source targets +that deployment. Scrape errors surface as `lastError` on the source. + +## Alert on a custom series + +Create an alert rule with `kind=custom` instead of a deployment metric: + +```bash +deploys alert create \ + --project acme \ + --name email-queue-depth \ + --kind custom \ + --source web \ + --series 'queue_depth{queue="email"}' \ + --metric value \ + --op ">=" \ + --threshold 1000 \ + --for 5 +``` + +`value` compares the gauge (or untyped) sample. `rate` compares the +per-minute increase of a counter. Delivery is still +[`alert.trigger` / `alert.resolve`](/automation/notification-channels/#metric-alerts-alerttrigger-and-alertresolve) +on your notification channels. Window semantics — 80% present to fire, a data +gap does not resolve a firing rule — are the same as [platform-metric +alerts](/automation/alerts/#when-a-rule-fires). + +A series that has not been scraped yet evaluates as `nodata`. You can create +the rule before the first scrape. + +## Limits + +| Limit | Value | +|---|---| +| Sources per project | 4 | +| Series per source | 100 (further series mark the source truncated) | +| Path length | 256 | +| Scrape timeout | 5 seconds | +| Scrape body | 1 MiB | +| Retention | 30 days | + +Custom metrics at these caps are included; there is no per-series SKU in v1. + +## Using the API directly + +`metricSource.set`, `metricSource.get`, `metricSource.list`, +`metricSource.delete`, `metricSource.series`, and `metricSource.query`. Also +on the [MCP server](/automation/mcp/). + +## Permissions + +| Action | Permission | +|---|---| +| Create / edit | `metricSource.set` | +| View / list / series / query | `metricSource.get` / `metricSource.list` | +| Delete | `metricSource.delete` | + +Nothing in the payload is secret, so `metricSource.get` / `metricSource.list` +are grantable to public principals like most read permissions. diff --git a/content/deployments/monitoring.md b/content/deployments/monitoring.md index 2ef1bc3..e0cd73a 100644 --- a/content/deployments/monitoring.md +++ b/content/deployments/monitoring.md @@ -14,7 +14,8 @@ gap between them tells you whether you're under- or over-provisioned. The Replicas chart is available replica count (HPA scale, crash-loop drop); it is hidden for Static and CronJob deployments, which have no k8s Deployment. The time-range selector spans **1 hour aggregate**, **1 day**, **7 days**, and -**30 days**. +**30 days**. To chart series from your own Prometheus `/metrics` endpoint, add +a [custom metric source](/deployments/custom-metrics/). {{< shot src="/img/deployment-metrics.png" url="console.deploys.app/deployment/metrics?project=acme&location=gke.cluster-rcf2&name=web" alt="Live CPU, memory, request rate, and egress charts for the web deployment" caption="Solid lines are real usage; dashed lines are the request you've allocated." >}} diff --git a/scripts/screenshots/README.md b/scripts/screenshots/README.md index 19ac20c..0bd4af7 100644 --- a/scripts/screenshots/README.md +++ b/scripts/screenshots/README.md @@ -33,9 +33,12 @@ Prereqs: - The console repo is cloned alongside this one: `~/Projects/deploys-app/console` (or wherever your deploys-app workspace lives — the script resolves it as - `../console`). -- The console repo's working tree on `src/lib/server/mock.js` is clean (the - script refuses to run otherwise so it never clobbers your own edits). + `../console`). From a docs git worktree set `CONSOLE_DIR` to the console + checkout that has `node_modules` (e.g. `.worktrees/console-custom-metrics`). +- The console repo's working tree on `src/lib/server/mock.ts` (or `mock.js`) + is clean (the script refuses to run otherwise so it never clobbers your own + edits). If `mock-enrichment.patch` does not apply, capture still runs against + the stock mock fixtures. - `bun` is on PATH for `bun dev:mock`. - Playwright Chromium is installed in the console repo (`bunx playwright install` if not — the console already ships `@playwright/test`). diff --git a/scripts/screenshots/capture.mjs b/scripts/screenshots/capture.mjs index 9f619e4..c2ff8c2 100644 --- a/scripts/screenshots/capture.mjs +++ b/scripts/screenshots/capture.mjs @@ -52,7 +52,10 @@ const screens = [ ['cache-manage', `/cache/manage?${P}&${LOC}`, { h: 1150 }], ['cache-metrics', `/cache/metrics?${P}&${LOC}`, { h: 1150 }], ['scheduler-list', `/scheduler?${P}`], - ['scheduler-detail', `/scheduler/detail?${P}&name=daily-health-check`, { h: 1150 }] + ['scheduler-detail', `/scheduler/detail?${P}&name=daily-health-check`, { h: 1150 }], + ['alert-list', `/alert?${P}`], + ['alert-create', `/alert/create?${P}`, { h: 1150 }], + ['alert-detail', `/alert/detail?${P}&name=api-memory-high`, { h: 1150 }] ] const browser = await chromium.launch() diff --git a/scripts/screenshots/refresh.sh b/scripts/screenshots/refresh.sh index 3e0b0e1..c5893af 100755 --- a/scripts/screenshots/refresh.sh +++ b/scripts/screenshots/refresh.sh @@ -14,17 +14,31 @@ set -euo pipefail here=$(cd "$(dirname "$0")" && pwd) docs=$(cd "$here/../.." && pwd) -console=$(cd "$docs/../console" && pwd) +# CONSOLE_DIR overrides the sibling-checkout default so a docs worktree can +# point at a console worktree (../../console from .worktrees/docs-* is wrong). +if [[ -n "${CONSOLE_DIR:-}" ]]; then + console=$(cd "$CONSOLE_DIR" && pwd) +else + console=$(cd "$docs/../console" && pwd) +fi patch="$here/mock-enrichment.patch" +mock_file="" +for candidate in src/lib/server/mock.ts src/lib/server/mock.js; do + if [[ -f "$console/$candidate" ]]; then + mock_file=$candidate + break + fi +done [[ -d "$console" ]] || { echo "console repo not found at $console" >&2; exit 1; } +[[ -n "$mock_file" ]] || { echo "console mock fixture not found (tried mock.ts / mock.js)" >&2; exit 1; } [[ -f "$patch" ]] || { echo "patch not found: $patch" >&2; exit 1; } cd "$console" -# refuse to run if mock.js has uncommitted changes — we'd revert them -if ! git diff --quiet -- src/lib/server/mock.js; then - echo "src/lib/server/mock.js has uncommitted changes — commit or stash first." >&2 +# refuse to run if the mock fixture has uncommitted changes — we'd revert them +if ! git diff --quiet -- "$mock_file"; then + echo "$mock_file has uncommitted changes — commit or stash first." >&2 exit 1 fi @@ -32,12 +46,14 @@ runner="" cleanup () { [[ -n "${mock_pid:-}" ]] && kill "$mock_pid" 2>/dev/null || true [[ -n "$runner" ]] && rm -f "$runner" || true - git checkout -- src/lib/server/mock.js 2>/dev/null || true + git checkout -- "$mock_file" 2>/dev/null || true } trap cleanup EXIT -echo "==> applying mock-enrichment.patch" -git apply "$patch" +echo "==> applying mock-enrichment.patch (against $mock_file)" +if ! git apply "$patch"; then + echo "warning: mock-enrichment.patch did not apply; capturing against stock mock fixtures" >&2 +fi echo "==> starting bun dev:mock" MOCK_API=1 bun run dev --port 5173 > /tmp/console-mock.log 2>&1 &