Skip to content

Restore Runbook

Jump to bottom
mdeguzis edited this page Jul 21, 2026 · 3 revisions

Restore Runbook

If the current maintainer walked away tomorrow, this is the page a stranger reads to bring Proton Pulse back up on their own account, keys, and infrastructure. Filed as #265.

Read this top to bottom before touching anything. The steps are ordered because Phase 3 (data import) depends on Phase 2 (fresh Supabase project) which depends on Phase 1 (inventory + accounts).

Phase 0 - what will you need

Accounts:

  • GitHub account with gh CLI installed and authenticated.
  • Supabase account (free tier is enough to start).
  • Cloudflare account with the proton-pulse.com domain transferred to it (or a new domain if the old one is gone).
  • Discord account with admin access to a server for webhooks + announcements.
  • Optional: Steam API key, SteamGridDB API key, OpenAI API key, VirusTotal API key. All can be obtained fresh for free.

Tools:

  • gh, git, jq, curl, psql, python3, uv, pnpm, supabase CLI.

Phase 1 - what exists

Git repos

Repo Contains Public
mdeguzis/proton-pulse-web Main app source, edge fn source, pipeline scripts, CI workflows Yes
mdeguzis/proton-pulse-web-staging Staging preview target for gh-pages Yes
mdeguzis/proton-pulse-web.wiki Public wiki (this page lives here) Yes
mdeguzis/proton-pulse-data-backup Weekly + build-triggered tarball backups of Supabase + gh-pages Private
mdeguzis/decky-proton-pulse Companion Decky plugin source Yes
mdeguzis/decky-proton-pulse.wiki Plugin wiki Yes
mdeguzis/condenser-proton-pulse Condenser port of the plugin Yes
mdeguzis/claude-configs Private repo of Claude memory + rule backups Private

Live infrastructure

  • Domain: proton-pulse.com (through Cloudflare).
  • Site: https://www.proton-pulse.com served directly from Cloudflare Pages project proton-pulse-web (post-#362).
  • Staging: https://staging.proton-pulse.com served from Cloudflare Pages project proton-pulse-web-staging.
  • Per-game data: R2 bucket proton-pulse-data bound to https://data.proton-pulse.com (public GET, CORS open).
  • Edge worker: pp-edge-status on Cloudflare Workers, powers the status page probe.
  • Supabase project: ilsgdshkaocrmibwdezk (URL https://ilsgdshkaocrmibwdezk.supabase.co).
  • Discord server: check pinned invite https://discord.gg/UdPaEsMtd. See Site Status Page for the incident + announcement webhooks.
  • GH Pages plumbing (gh-pages branch + mdeguzis/proton-pulse-web-staging repo) is preserved as the rollback path only. Nothing points to it in the live config.

Security scanner + moderation infrastructure

Non-obvious pieces that live in .github/workflows/ and need to keep working after a restore:

File Purpose Where it fails visibly
.github/workflows/semgrep.yml Static analysis (TypeScript, OWASP, security-audit rulepacks) on every push + PR PR checks show semgrep/ci red
.github/workflows/sbom.yml Syft SBOM + Grype vulnerability scan, uploads CycloneDX artifact PR checks show sbom + grype red
.grype.yaml Ignore rules: drop go-module + github-action findings (vendored inside the supabase npm CLI, not our runtime) Removing this file re-floods Grype with 300+ false positives
.github/workflows/restore-drill-reminder.yml Quarterly cron opens the drill checklist issue No drill issue on 1st of Jan/Apr/Jul/Oct
.github/workflows/security-monitor.yml + .github/scripts/security-monitor.mjs Hourly Supabase log check: auth spikes, 5xx rates, RLS denials, aggregate + per-IP 429s New issues stop appearing when a real event lands

Content moderation lives in a sibling repo:

File Purpose
proton-pulse-data/.github/workflows/content-moderation.yml Every 4h + daily deep scan, calls the script below
proton-pulse-data/.github/scripts/moderate-content.mjs Two-layer scan (wordlist + OpenAI) on user_configs, plus aux scans on user_proton_configs.app_name, user_systems.label, user_systems.sysinfo_text, flagged_reports.reason_text. Aux hits open GH issues labeled content-moderation-review + security (rate-limited to 10 issues per run)

The content-moderation-review label must exist in the target repo (mdeguzis/proton-pulse-web) or issue creation will 422. Recreate with:

gh label create content-moderation-review --repo mdeguzis/proton-pulse-web \
  --color d73a4a --description "Aux moderation scan flagged a row -- admin triage needed"

Edge functions (all under supabase/functions/<name>/index.ts)

  • image-refetch
  • plugin-link-complete, plugin-link-remove, plugin-link-start, plugin-link-status, plugin-link-unlink, plugin-links-list
  • protondb-summary
  • public-steam-profile (anonymous, powers /lookup -- reads Steam profile URL / SteamID + owned games via Steam Web API using STEAM_API_KEY)
  • steam-appdetails, steam-callback, steam-depot-info, steam-explore, steam-library-lookup, steam-news
  • sync-steam-library
  • sync-steam-wishlist
  • user-system-upload

GitHub Actions secrets (in mdeguzis/proton-pulse-web)

Secret Purpose Where to get a new value
SUPABASE_URL REST base URL for the project Supabase console -> Project Settings -> API
SUPABASE_ANON_KEY Public anon key Supabase console -> Project Settings -> API
SUPABASE_SERVICE_ROLE_KEY Server-only key, RLS bypass Supabase console -> Project Settings -> API
SUPABASE_TOKEN Personal Access Token for supabase CLI + Management API Supabase console -> Account -> Access Tokens
BACKUP_HMAC_SECRET Signs the backup tarballs Generate: openssl rand -hex 32
BACKUP_REPO_TOKEN Push access to proton-pulse-data-backup GitHub fine-grained PAT, Contents: R/W on the backup repo
STAGING_DEPLOY_TOKEN Push access to proton-pulse-web-staging (rollback path only, GH Pages) GitHub fine-grained PAT, Contents: R/W on the staging repo
CLOUDFLARE_API_TOKEN CF Pages Edit + Workers Scripts Write + R2 Bucket Item Write on proton-pulse-data Cloudflare dashboard -> My Profile -> API Tokens
CLOUDFLARE_ACCOUNT_ID Cloudflare account ID (not the token) Cloudflare dashboard -> right sidebar on any Pages or Workers page
R2_ACCESS_KEY_ID R2 access key for the pipeline data upload Cloudflare dashboard -> R2 -> Manage R2 API Tokens
R2_SECRET_ACCESS_KEY Paired secret for the above Same
DISCORD_WEBHOOK_ANNOUNCE #announcements channel Discord -> Server -> Integrations -> Webhooks
DISCORD_WEBHOOK_BACKUPS Backup channel Same
DISCORD_WEBHOOK_BUILDS CI failures channel Same
DISCORD_WEBHOOK_ISSUES Issue events channel Same
DISCORD_WEBHOOK_PULL_REQUESTS PR events channel Same
DISCORD_WEBHOOK_RELEASES Releases channel Same
STEAM_API_KEY Steamworks Web API https://steamcommunity.com/dev/apikey
SGDB_API_KEY SteamGridDB fallback for missing cover art https://www.steamgriddb.com/profile/preferences/api
OPENAI_API_KEY Content moderation classifier (lives in mdeguzis/proton-pulse-data too -- same key, both repos) https://platform.openai.com/api-keys
VT_API_KEY VirusTotal attachment scanner https://www.virustotal.com/gui/my-apikey
CODECOV_TOKEN Coverage uploads codecov.io repo settings
GITHUB_TOKEN Injected by Actions itself Never manually set

Phase 2 - rebuild from zero

Assume you have the code repo cloned and the backup repo cloned side by side.

2.1 Fresh Supabase project

  1. In the Supabase console, create a new project. Region should match latency needs (US East is what we used).
  2. Note the project ref (from the URL). It replaces ilsgdshkaocrmibwdezk everywhere below.
  3. Copy the anon key, service role key, project URL. These become the new secrets.
  4. Run every migration under supabase/migrations/ in order via the Management API:
source ~/.supabase
export SUPABASE_PROJECT_REF=<new-ref>
for f in supabase/migrations/*.sql; do
  echo "applying $f"
  SQL=$(cat "$f")
  curl -s -X POST "https://api.supabase.com/v1/projects/${SUPABASE_PROJECT_REF}/database/query" \
    -H "Authorization: Bearer $SUPABASE_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"query\": $(echo "$SQL" | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')}"
done
  1. Deploy every edge function:
export SUPABASE_ACCESS_TOKEN=$SUPABASE_TOKEN
for fn in supabase/functions/*/; do
  slug=$(basename "$fn")
  [ "$slug" = "_shared" ] && continue
  supabase functions deploy "$slug" --project-ref "$SUPABASE_PROJECT_REF"
done
  1. In Supabase console -> Edge Functions -> Secrets, add: STEAM_API_KEY, SGDB_API_KEY, OPENAI_API_KEY. See Secrets and Environment for the current names each fn expects.

2.2 Fresh GitHub secrets

Populate every secret from the Phase 1 table into mdeguzis/proton-pulse-web -> Settings -> Secrets and variables -> Actions. Also copy SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, and OPENAI_API_KEY into mdeguzis/proton-pulse-data -> Settings -> Secrets so the content moderation workflow can run.

Recreate the content-moderation-review label in mdeguzis/proton-pulse-web (see the Security scanner + moderation infrastructure section above for the gh label create command). Without the label, the aux moderation scanner cannot open issues.

2.3 Domain + DNS + Pages (Cloudflare Pages, post-#362)

  1. Ensure proton-pulse.com is on Cloudflare.
  2. Create two Cloudflare Pages projects in the CF dashboard:
    • proton-pulse-web-staging (custom domain: staging.proton-pulse.com)
    • proton-pulse-web (custom domain: www.proton-pulse.com)
  3. Create the R2 bucket proton-pulse-data and bind the custom domain data.proton-pulse.com to it (CORS: allow GET from any origin).
  4. In the Cloudflare API Tokens dashboard, create a token with these permissions:
    • Cloudflare Pages: Edit (both accounts)
    • Workers Scripts: Write + Read (entire account)
    • R2 Storage Bucket Item: Write (scoped to proton-pulse-data)
  5. Add the token as GitHub Actions secret CLOUDFLARE_API_TOKEN. Also add CLOUDFLARE_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY (R2 credentials come from the CF R2 dashboard, NOT the API token).
  6. Add the Supabase edge fn secret STAGING_URL=https://staging.proton-pulse.com so Steam OAuth callbacks from staging land back on staging instead of prod. SITE_URL stays as https://www.proton-pulse.com (see the project_supabase_site_url.md memory).
  7. Update the CF Worker pp-edge-status's ALLOWED_ORIGINS array in workers/edge-status/index.js if any origin has changed; deploy via make deploy-worker.
  8. First push to main triggers publish-shell.yml, which deploys the shell to CF Pages.

2.3.1 Rollback path (GitHub Pages)

The GitHub Pages plumbing is preserved as rollback. To activate:

  1. Repoint www DNS to the GitHub Pages IPs (185.199.108-111.153).
  2. In mdeguzis/proton-pulse-web -> Settings -> Pages -> Custom domain: set www.proton-pulse.com.
  3. Wait for GitHub's cert to be issued, then flip Cloudflare proxy back on with SSL mode = Full (non-strict).
  4. Dispatch update-data.yml with deploy_target=ghpages to populate the gh-pages branch.

2.4 Rebuild pipeline state

The initial CF Pages deploy needs data. Dispatch the full pipeline (deploys to CF Pages by default post-#362):

gh workflow run update-data.yml --repo mdeguzis/proton-pulse-web
gh run watch --repo mdeguzis/proton-pulse-web

For the R2 bucket, the first pipeline run syncs all ~187k per-game data files. Wall time ~30 min for the sync alone. If the pipeline succeeds but R2 is empty at the end, check the CLOUDFLARE_API_TOKEN has R2 Bucket Item: Write on proton-pulse-data specifically.

Phase 3 - data import

The backup repo has three artifact families under latest/:

File Contents Restore path
latest-schema.tar.gz pg_dump of the schema (tables, RLS, functions, triggers) Not needed if Phase 2.1 migrations were applied cleanly. Use for drift check only
latest-user_configs.tar.gz JSON dump of user_configs (Pulse reports) psql \copy back into Supabase
latest-author_avatars.tar.gz Storage bucket dump Upload via supabase storage upload
latest-site.tar.gz Full gh-pages snapshot (data JSONs, search-index, etc.) git checkout --orphan gh-pages && tar xf latest-site.tar.gz && git add -A && git commit && git push --force

3.1 user_configs import

tar xzf latest-user_configs.tar.gz
# yields user_configs.jsonl
export PGPASSWORD=<db-password>
psql -h db.<PROJECT_REF>.supabase.co -U postgres -d postgres \
  -c "\copy user_configs FROM 'user_configs.jsonl' WITH (FORMAT csv)"

Adjust FORMAT + column list based on what the dump actually contains -- inspect the first row before running.

3.2 author_avatars import

tar xzf latest-author_avatars.tar.gz
supabase storage cp --project-ref "$SUPABASE_PROJECT_REF" \
  --recursive ./author_avatars/ supabase://author-avatars/

3.3 Data restore (R2 + gh-pages backup)

The latest-site.tar.gz tarball is a snapshot of the pre-#362 gh-pages layout. In the CF Pages era it is not what the live site reads (that's the R2 bucket + the CF Pages shell). Use it for:

  • Rollback: repush the tarball to gh-pages and follow section 2.3.1 to activate GitHub Pages.
  • Cold-start R2: dispatch update-data.yml (see 2.4). The pipeline probes fresh and uploads to R2, so the tarball is not strictly required. Use it only if you need the historical exact snapshot.

Rollback restore:

git clone --depth 1 git@github.com:<yourorg>/proton-pulse-web.git ghp
cd ghp
git checkout --orphan gh-pages
git rm -rf .
tar xzf ../latest-site.tar.gz
git add -A
git commit -m "Restore from backup"
git push origin gh-pages --force

Phase 4 - wiki restore

Each wiki repo is its own git remote. To restore:

git clone git@github.com:<yourorg>/proton-pulse-web.wiki.git
# copy .md files from an old backup or from a fork
cd proton-pulse-web.wiki
git add -A
git commit -m "Restore wiki"
git push

Same pattern for decky-proton-pulse.wiki.git.

Phase 5 - verification checklist

Before declaring recovery done, verify each of these:

  • /about.html shows the version and short SHA you expected.
  • /data/570/index.json returns a real ProtonDB report list.
  • Steam sign-in from /profile.html completes and lands on the profile page.
  • /admin.html recognises the seed admin uuid you inserted into the admins table.
  • File a test issue with the incident label -> Discord #announcements receives a message.
  • /status.html loads and shows every edge function green.
  • Submit a dummy report through the plugin or /submit.html, approve it in admin, confirm it shows up on the game page.
  • Open any PR -- CI shows semgrep/ci and sbom + grype green.
  • SBOM artifact appears on the PR run: Actions -> the sbom+grype job -> Artifacts panel has sbom-cyclonedx + grype-sarif.
  • Content moderation workflow (proton-pulse-data) runs on schedule and its summary shows all four aux tables were scanned (0 hits acceptable, the count row is what matters).
  • content-moderation-review label exists in mdeguzis/proton-pulse-web (gh label list --repo mdeguzis/proton-pulse-web | grep content-moderation-review).

Phase 6 - ownership handoff

If the domain, Supabase project, Cloudflare zone, and Discord server are being transferred (not recreated), do all of these:

  1. Discord: Server Settings -> Members -> transfer ownership. Requires 2FA on the new account.
  2. Supabase: console -> Organization -> add new owner -> demote current owner. Free tier only allows one owner per org, so may require moving the project to a new org.
  3. Cloudflare: DNS zone -> Members -> add new member as Administrator, verify DNS still resolves, then remove the old member.
  4. Domain: registrar transfer (auth code + 5-day window).
  5. GitHub: repo settings -> Manage access -> add new maintainer, wait for accept, then transfer ownership under Settings -> Danger Zone.
  6. Cycle every PAT that the previous maintainer's account issued: SUPABASE_TOKEN, BACKUP_REPO_TOKEN, STAGING_DEPLOY_TOKEN, GITHUB_TOKEN shows up as GITHUB_TOKEN in Actions so that's free.

Zero-downtime secret rotation

Rotating any long-lived secret should be a two-write:

  1. Add the new value under a temporary secret name (e.g. SUPABASE_TOKEN_NEW).
  2. Deploy code that reads both SUPABASE_TOKEN and SUPABASE_TOKEN_NEW, preferring the new one.
  3. Trigger every workflow that uses the secret and confirm.
  4. Rename SUPABASE_TOKEN_NEW -> SUPABASE_TOKEN, remove the compat code, redeploy.

Related pages

Clone this wiki locally