Skip to content

Site Status Page

Jump to bottom
mdeguzis edited this page Jul 21, 2026 · 3 revisions

Site Status Page

The Site status page at /status.html shows the live health of every Supabase edge function this site uses. It also carries an announcement feed pulled from GitHub issues labeled incident, and a small colored dot in the topbar summarizes the overall state so a reader knows something is off before they even click through.

Filed as issue #254. The check moved off a GitHub Actions cron to a Cloudflare Worker in #275. This page documents the moving parts so future changes can be made safely.

How it runs now (Cloudflare Worker, #275)

The health check runs on a Cloudflare Worker called pp-edge-status, not GitHub Actions. GitHub's schedule: cron was best-effort: runs drifted 15 to 45+ minutes under load, got skipped, and got auto-disabled after 60 days of repo inactivity. It also only wrote edge-status.json to the prod gh-pages branch, so the staging status page had no live data. A Cloudflare Cron Trigger fires on time, and one CORS endpoint feeds both prod and staging.

The old .github/workflows/edge-fn-health.yml (running scripts/check-edge-fn-health.sh) stays as a manual workflow_dispatch backstop that still writes the static edge-status.json. Its schedule: block is removed once the worker heartbeat is confirmed in prod. The bash script's FNS=() array is the source of truth the worker's FNS list is kept in sync with (a test enforces parity).

Where everything lives

Thing Path
Page shell status.html
Frontend logic js/status/main.js
Frontend styles css/status/status.css
Worker (probe + serve) workers/edge-status/index.js
Worker config (cron, KV, vars) workers/edge-status/wrangler.toml
Worker deploy target make deploy-worker -> scripts/deploy-worker.sh
Worker tests tests/edgeStatusWorker.test.js
Live endpoint constant EDGE_STATUS_ENDPOINT in js/status/main.js
Backstop health script scripts/check-edge-fn-health.sh
Backstop cron workflow .github/workflows/edge-fn-health.yml (dispatch-only)
Discord incident notifier .github/workflows/discord-incidents.yml
Static fallback data edge-status.json (on gh-pages)
Topbar dot markup + fetch js/lib/topbar.js (wireStatusDot)
Topbar dot styles css/shared/topbar.css

How the health check works

  1. The worker's scheduled() handler fires on a real */15 * * * * Cron Trigger ([triggers] crons in wrangler.toml).
  2. It pings every tracked Supabase edge function with an OPTIONS request (the CORS preflight, which every deployed function answers quickly regardless of auth state).
  3. For each function it records: response HTTP status, latency in ms, an ISO timestamp, and a derived status (operational / degraded / down).
  4. It writes the aggregated payload (same shape as the old edge-status.json) to Workers KV under the edge-status key, and appends each function's latency onto a rolling 7-day history under edge-status-history.
  5. The worker's fetch() handler serves that payload as JSON with CORS for the site's real origins: https://www.proton-pulse.com (prod), https://staging.proton-pulse.com (CF Pages staging), and https://mdeguzis.github.io (retired GH Pages staging kept for rollback). If a request comes in from an origin not on the list the worker falls back to the first entry (prod), which is what caused the "JSON.parse: unexpected character" error on staging after the CF Pages migration until #362 added the new origin. Regression test in tests/edgeStatusWorker.test.js pins the full list.

The worker needs one plain var and one secret (set in wrangler.toml / wrangler secret put):

  • SUPABASE_URL (plain [vars] entry)
  • SUPABASE_ANON_KEY (secret)

It also binds a KV namespace EDGE_STATUS_KV (id in wrangler.toml). If moving to a different Supabase project, update the var/secret on the worker, not this file.

The worker endpoint

pp-edge-status is deployed at https://pp-edge-status.mdeguzis.workers.dev. Routes:

Request Behavior
GET / Returns the latest snapshot from KV. Cold KV (fresh deploy) runs one live probe so the page is never blank.
GET /?history=<fn> Returns that function's [[epochSec, ms], ...] latency series for the sparkline. ?history=1 or ?history=all returns every series.
POST / Admin-only "Check now" (see below).
OPTIONS / CORS preflight.

EDGE_STATUS_ENDPOINT in js/status/main.js holds the worker URL. If it is '', the page falls back to the static edge-status.json, so the page keeps working before the worker is wired in or if the worker is unreachable.

HTTP code mapping

HTTP code Derived state Rationale
200, 204 operational Function answered the preflight normally.
401, 403 operational Function is up and responded fast, it just rejected the anonymous probe based on its auth policy. Real signed-in traffic goes through. This is intentional for steam-callback and any function that requires a session.
5xx down Server-side error, function crashed or Supabase misbehaving.
000 (curl connection failure) down Never got a response, DNS or network.
everything else degraded Not clearly good or bad, worth eyeballing.

overall is the worst of the individual states: any down makes overall down; else any degraded makes overall degraded; else operational.

Tracked edge functions

The list is hard-coded as FNS in workers/edge-status/index.js (and mirrored in scripts/check-edge-fn-health.sh). When adding a new edge function under supabase/functions/, add its name to both (see "Adding a new service to track"). The order in the array is the order the cards render in.

Current set:

  • image-refetch
  • plugin-link-complete
  • plugin-link-remove
  • plugin-link-start
  • plugin-link-status
  • plugin-link-unlink
  • plugin-links-list
  • protondb-summary
  • steam-appdetails
  • steam-callback
  • steam-depot-info
  • steam-explore
  • steam-library-lookup
  • steam-news
  • sync-steam-library
  • user-system-upload

Each function's source: supabase/functions/<name>/index.ts.

Frontend behavior

js/status/main.js boots on page load and does two things in parallel:

  1. Service list: fetchStatusPayload() tries EDGE_STATUS_ENDPOINT (the worker) first and falls back to the static edge-status.json via dataUrl() if the worker is unset or unreachable. Renders one clickable card per service, auto-refreshing every 60 seconds.
  2. Announcements: fetches the public GitHub issues API for the repo, filtered to label=incident&state=all, unauthenticated. Renders open incidents first (yellow left border), then resolved ones (muted). Also auto-refreshes every 5 minutes.

Clicking a service card opens a modal (see openServiceModal in main.js). The modal has:

  • An explainer at the top that varies by HTTP code. explainerFor(svc) is the switch that picks the right copy for the code.
  • A definition list of the raw fields.
  • An admin-only Check now button (super admins only, see below).
  • A <pre> block with the full raw JSON blob for the service.
  • A latency sparkline under the JSON.

The modal closes on the X button, an outside click, or Escape.

Latency sparkline

When a tile modal opens, fetchHistory(fn) pulls that function's [[epochSec, ms], ...] series from the worker (?history=<fn>) and renderSparkline() draws a small time-scaled inline SVG under the raw JSON, with min / avg / max labels. No chart library. The series starts empty on a fresh worker deploy and fills as crons run (a super admin can seed points instantly with Check now). Shows a "still collecting" message until there are at least two points.

Admin "Check now"

Super admins get a Check now button in each service tile that re-probes that one function on demand instead of waiting for the next cron. detectSuperAdmin() gates whether the button renders (reads the caller's admins row filtered to role=eq.super_admin). The button is not the security boundary: checkServiceNow() POSTs the user's Supabase access token to the worker, which re-verifies super_admin server-side (verifySuperAdmin: resolve the token via /auth/v1/user, then read the caller's admins row) before running any probe. Unauthenticated or non-admin callers get 401/403 and no probe runs. A successful check merges the fresh result into KV, appends a history point, and returns the full payload so the page and modal re-render.

Deploying / updating the worker

make deploy-worker          # deploys workers/edge-status (default)
make deploy-worker WORKER=<name>   # any other folder-based worker under workers/

make deploy-worker calls scripts/deploy-worker.sh, which validates the worker dir + wrangler.toml and runs npx wrangler deploy. It relies on a prior npx wrangler login. One-time setup for a fresh worker (KV namespace, SUPABASE_ANON_KEY secret) is documented in the header of workers/edge-status/index.js.

Topbar dot

Every page loads topbar.js, which injects the Status nav link and calls wireStatusDot. That helper:

  1. Reads a sessionStorage cache under pp-edge-status. If less than 60 seconds old, applies the cached color and skips the network.
  2. Otherwise fetch('edge-status.json', { cache: 'no-store' }), reads overall, and applies it to every .topbar-status-dot on the page.
  3. Writes the new state to sessionStorage with a fresh timestamp.

The dot uses data-state="operational|degraded|down|unknown" to pick its color. See the .topbar-status-dot[data-state=...] rules in css/shared/topbar.css.

Discord incident notifications

.github/workflows/discord-incidents.yml fires on the issues event when an issue labeled incident is opened, closed, reopened, labeled, or unlabeled. It filters in the if: clause so non-incident issues exit early without spinning up a runner.

Posts to the DISCORD_WEBHOOK_ANNOUNCE webhook with a short plain-text message + URL. If the secret is not set the workflow logs a message and exits 0 without posting.

Message format:

[INCIDENT] New announcement: #123
The title of the issue
https://github.com/mdeguzis/proton-pulse-web/issues/123
_by mdeguzis_

Actions covered: opened, labeled (added incident label), closed, reopened, unlabeled (incident label removed).

Announcements: how to file one

To raise an incident from the site's status page, click the Report an issue button in the Announcements block. It deep-links to a new issue with the incident label already applied.

Directly:

gh issue create --repo mdeguzis/proton-pulse-web \
  --label incident \
  --title "Incident: <what>" \
  --body "<summary and impact>"

When the incident is resolved, close the issue. The status page and Discord announcement will both reflect the resolved state on their next refresh.

Adding a new service to track

The function list lives in two places that a test (tests/edgeStatusWorker.test.js) keeps in parity:

  1. Add the function name to the FNS array in workers/edge-status/index.js.
  2. Add the same name to the FNS=() array in scripts/check-edge-fn-health.sh (the backstop).
  3. Run make deploy-worker so the live worker picks it up on the next cron.
  4. Commit + push both files. The parity test fails if the two lists diverge.

Troubleshooting

Status page shows "Could not load status data":

  • The worker is unreachable AND the static edge-status.json fallback is missing. Hit https://pp-edge-status.mdeguzis.workers.dev/ directly: a 200 with JSON means the worker is fine and the problem is the page's fetch. A worker error usually means a bad deploy, a missing KV binding, or an unset SUPABASE_ANON_KEY secret. Re-run make deploy-worker and confirm the KV id + secret.

Sparkline says "still collecting" forever:

  • History only accumulates from probes run under the current worker deploy. Confirm GET /?history=<fn> returns a non-empty array. If it's empty, the cron may not have fired yet (wait 15 min) or a super admin can seed points with Check now.

One function is always "degraded":

  • Check the HTTP code in the detail modal. If it's 401 or 403, it should be operational, not degraded. Confirm the case statement in the script covers that code.

Topbar dot never turns green:

  • SessionStorage cache may be sticky. Open devtools, delete the pp-edge-status key, reload. If that fixes it, look for a stale write pattern in wireStatusDot.

Discord notification did not fire on an incident:

  • Confirm the issue actually carries the incident label at the moment the event fired. Label additions to existing issues fire with action: labeled, which the workflow handles.
  • Confirm DISCORD_WEBHOOK_ANNOUNCE is set in the repo secrets.

Related docs

Clone this wiki locally