Skip to content

Data Pipeline

Jump to bottom
mdeguzis edited this page Jul 5, 2026 · 11 revisions

Data Pipeline

CI/CD architecture for the proton-pulse-data GitHub Pages mirror. This pipeline runs daily via GitHub Actions and produces the per-game report files the plugin consumes.

Pipeline Stages

The update-data.yml workflow has three main stages:

Stage 1: build          Stage 2: probe-chunks        Stage 3: finalize
-----------------       ---------------------        ------------------
Process official     -> Probe ProtonDB summaries  -> Backfill probe
  ProtonDB dump           in parallel chunks           discoveries
Manifest backfill       (resumable, cache-backed)    Merge Pulse Reports
Compute probe plan                                   Build indexes
                                                     Coverage report
                                                     Deploy to gh-pages

Dispatch modes

The same three jobs run in different subsets depending on which mode fired the dispatch. This is what lets staging_with_finalize do a schema-shape verification in ~5 min instead of ~30.

flowchart LR
    subgraph B[Stage 1: build]
        B1[dump processing +<br/>backfill + chunk plan]
    end
    subgraph P[Stage 2: probe-chunks]
        P1[ProtonDB probe<br/>parallel chunks<br/>persist to gh-pages]
    end
    subgraph F[Stage 3: finalize]
        F1[restore chunk state<br/>from gh-pages] --> F2[probe backfill]
        F2 --> F3[finalize.py<br/>build indexes]
        F3 --> F4[stats + hardware]
        F4 --> F5{deploy target}
        F5 -- default --> F6[push to origin/gh-pages<br/>prod]
        F5 -- staging_with_pipeline<br/>or staging_with_finalize --> F7[push to staging repo<br/>gh-pages]
    end
    Full[gh-run<br/>full pipeline] --> B --> P --> F
    Resume[gh-resume<br/>rescue partial] --> B
    Finalize[gh-finalize-only] --> F
    StagingShell[gh-staging<br/>shell only] -.-> Shell[staging-preview job<br/>no data write]
    StagingFull[gh-staging-pipeline] --> B
    StagingFast[gh-staging-finalize] --> F
    style Full fill:#1e3a5f,color:#fff
    style StagingFast fill:#0f5132,color:#fff
    style StagingFull fill:#664d03,color:#fff
    style Resume fill:#155e75,color:#fff
    style Finalize fill:#155e75,color:#fff
    style StagingShell fill:#495057,color:#fff
Loading

The finalize job branches on the deploy target: prod runs by default, the staging repo receives the push when either staging_with_pipeline or staging_with_finalize is on. This is a hard mutual exclusion in the workflow gates, tested by tests/test_staging_workflow_gates.py. Missing either check is the exact bug that once force-pushed prod's frontend over a staging preview.

Stage 1 -- build

  1. Process Official Reports -- Parses the bdefore/protondb-data monthly dump into data/{appId}/{year}.json year-bucket files.
  2. Manifest Backfill -- Apps listed in config/live_backfill_app_ids.json that are missing from the dump are fetched from ProtonDB's live detailed report endpoint.
  3. Build Probe Chunk Plan -- Computes which Steam app IDs still need a ProtonDB summary probe, splits them into chunks for parallel execution, and outputs the chunk matrix.

Outputs: probe-input artifact (data files + pipeline-state.json) and the chunk matrix for Stage 2.

Stage 2 -- probe-chunks

Runs the ProtonDB summary probe in sequential chunks (max-parallel: 1). Each chunk:

  • Restores the pipeline cache (.cache/protondb-summary-probe-cache.json)
  • Idempotent skip check (scripts/probe-idempotent-check.sh, #171 Phase 4) -- hashes the current cache file and compares against this chunk's stored input_sha in the manifest. Match = state hasn't drifted since last successful run, so probe would find nothing new. Emits skip=true and the probe step gates itself off. Warm reruns become near-instant.
  • Probes a slice of app IDs against ProtonDB's summary API (skipped if idempotent check fired)
  • Saves the updated cache under a fresh cache key
  • Persists a durable snapshot (scripts/persist-chunk-state.sh) to gh-pages/.pipeline-state/chunks/chunk-NN.json.gz plus a manifest.json entry (status, app_count, input_sha, completed_at, run_id, run_attempt). Runs on if: always() so a stalled chunk still records whatever it wrote before dying. Push loop retries up to 3 times with jittered backoff for cross-run contention. Skipped when the idempotent check fired (nothing to persist).

Skipped entirely when chunk_count == 0 (all apps already cached).

Durable chunk state (#171)

The chunk-snapshot layout on gh-pages:

.pipeline-state/
  manifest.json           { updated_at, chunks: {"01": {status, app_count, input_sha, completed_at, run_id, run_attempt}, ...} }
  chunks/
    chunk-01.json.gz      full probe cache after chunk 01 ran
    chunk-02.json.gz      full probe cache after chunk 02 ran (contains chunk 01's data too)
    ...

Each snapshot is the FULL cache state after that chunk, not a delta, so recovery only needs the highest-numbered file. Finalize (Stage 3) verifies the manifest against expected_matrix (the full plan) and aborts loud with chunk NN missing since T if a gap is found -- surfaces runner-starvation incidents at the right layer instead of silently finalizing on stale data.

Stage 3 -- finalize

  1. Restore chunk state from gh-pages (scripts/restore-chunk-state.sh) -- Reads .pipeline-state/manifest.json, verifies every chunk in expected_matrix (the full plan from Stage 1) is present and completed, then gunzips the highest-numbered snapshot into .cache/protondb-summary-probe-cache.json. Aborts loud with chunk NN missing since T if a gap is found so runner-starvation incidents surface here instead of finalizing on stale data. See #171 Phase 2.
  2. Backfill Probe Discoveries -- Apps the probe found on ProtonDB but missing from local data get their live detailed reports fetched and written to year-bucket files.
  3. Merge Pulse Reports -- Pulls user-submitted configs from Supabase (user_configs table) and merges them into the same year-bucket files alongside ProtonDB data. See Pulse Report Merge below.
  4. Finalize -- Builds index.json, latest.json, coverage report, search index, and data-index page.
  5. Deploy -- Force-pushes an orphan gh-pages branch with the full dataset plus static web app files.

Pulse Report Merge

Pulse Reports are the plugin's own data: hardware-tagged configs submitted via the Decky UI or the web form. They live in Supabase (user_configs table) and were historically fetched live by the web UI. As of proton-pulse-data#TBD, the pipeline snapshots them into the same static JSON files that hold ProtonDB reports, so consumers (plugin, web, third-party scripts) only need to read one place.

Where it runs

scripts/pipeline/pulse.py -- called from finalize_output() right after generate_latest_files() and before app indexes are built, so latest/index files pick up the freshly merged Pulse records.

How records are tagged

Every report in a year file now carries a source field so consumers can filter cleanly:

source Origin Schema
"protondb" ProtonDB archive or live fetch Base shape: cpu, gpu, protonVersion, rating, notes, timestamp, ...
"pulse" Supabase user_configs row Base shape plus launchOptions, formResponses, configKey, gameOwned, vramMb, durationMinutes, pulseId, submissionSource

Pulse records preserve the granular submission origin in submissionSource (e.g. "user" for the Decky plugin, "web-linux" for the web form), so the broader source: "pulse" tag never loses that information.

Dedup behavior

The Supabase row id is stored on each Pulse record as pulseId. On every pipeline run, existing pulse records in the year file with a pulseId matching an incoming row are replaced, not duplicated. This means users editing their submissions in Supabase have those edits reflected in the static snapshot on the next pipeline run.

ProtonDB records continue to dedupe by timestamp (handled in process.py, unchanged).

Bucketing

Pulse rows are bucketed by year using created_at. A submission from 2025-03-14 lands in data/{appId}/2025.json. Years are extracted from the ISO timestamp via datetime.fromisoformat() in UTC.

From report to search rating

A Pulse report can land on a game that has no ProtonDB data at all. That game starts life as an extended Steam stub (No Rating, zero counts). The rating fills in on the next pipeline run through a fixed chain of steps in finalize_output():

  1. merge_pulse_into_data_dir() runs first and creates data/{appId}/{year}.json from the Supabase row, even if the app had no directory before.
  2. derive_index_keys_from_disk() walks data/ and picks up that new directory, so the generators see the app this run instead of only the apps reprocessed from ProtonDB.
  3. generate_search_index() calls _compute_game_summary(), which averages the per-report scores (a single PLATINUM report scores 1.0, mapping to the platinum tier) and emits the app into the primary search-index.json with pulseCount set.
  4. Because the app is now in the primary index, generate_extended_steam_index skips it, so the old No Rating stub drops out and the frontend dedup shows the real tier.

So a freshly submitted report shows No Rating until the next "Build Site Data" run, then appears with its computed tier. No manual step is involved. The tier math matches the Scoring Algorithm page.

Backfill safety

The merge function backfills source: "protondb" on any legacy untagged ProtonDB records it encounters in the same file. So after one full pipeline run, every record in every year file is self-describing.

Failure mode

If Supabase is unreachable (network, rate limit, schema mismatch), the function logs the error and returns without touching the year files. ProtonDB data still ships as normal.

Credentials

Defaults to the production Supabase project's anon publishable key, which is read-only by RLS policy. Forks / staging can override via env:

SUPABASE_URL=https://<project>.supabase.co/rest/v1 \
SUPABASE_ANON_KEY=<your_anon_key> \
make gh-run

Pipeline Cache

The .cache/ directory is persisted across runs via GitHub Actions actions/cache. It stores:

  • protondb-summary-probe-cache.json -- Which apps have been probed and their summary results
  • Steam title cache -- Resolved game titles to avoid repeated Steam Store API calls

Each stage saves its cache under a unique key (build, chunk-N, final) so they don't collide.

Environment Variables

Variable Default Purpose
PROTONDB_PROBE_LIMIT 5000 Max apps to probe per chunk
PROTONDB_PROBE_BACKFILL_LIMIT 0 (unlimited) Max apps to backfill from probe discoveries
PROTONDB_PROBE_LOG_EVERY 100 Log progress every N apps during probe

Dispatch modes

update-data.yml exposes several workflow_dispatch inputs so a manual run can target a specific slice of the pipeline. Regular scheduled runs use all defaults and run the full stack.

Input Make target What it does
(none, defaults) make gh-run Full pipeline. Build -> probe-chunks -> finalize, deploys to prod.
pages_only=true make gh-pages-only Skips the pipeline entirely, republishes only the HTML/JS/CSS shell + static assets against current gh-pages data. Used after a UI-only change is confirmed on staging.
staging_only=true make gh-staging Deploys shell files to the staging repo (proton-pulse-web-staging). No pipeline run, no prod push.
staging_with_pipeline=true make gh-staging-pipeline Full pipeline but deploys data + shell to the staging repo instead of prod. Slow (30+ min). Use when a pipeline-data change needs end-to-end verification before promotion (#117).
backfill_app_ids=<comma list> make gh-backfill-apps ... Fetches specific app IDs into a small backfill artifact, publishes them. Skips the full pipeline.
coverage_backfill_issue_type=... make gh-coverage-backfill ... Runs the coverage-issue backfill script for one class of missing data (no-titles / bad-app-id / no-protondb-data).
resume=true make gh-resume Reads gh-pages/.pipeline-state/manifest.json, subtracts completed chunks from the chunk matrix, and re-runs only the missing ones. Rescue path after a partial run (e.g. chunk 8 stalled while 1-7 + 9-10 completed -> new dispatch runs just [08]). Finalize verifies the full expected_matrix afterwards (#171 Phase 3).
finalize_only=true make gh-finalize-only Skips build + probe-chunks entirely, re-runs only finalize against whatever state is currently in gh-pages/.pipeline-state/. Use after a finalize-side bug fix when the probe cache is fine (#171 Phase 3).

Only one non-default mode should be set per dispatch. The gates in the workflow enforce this: e.g. resume and finalize_only are mutually exclusive with staging_only, staging_with_pipeline, backfill_app_ids, and coverage_backfill_issue_type.

Resolved: probe-backfill Re-fetching All Apps

Original symptom

The "Backfill Probe Discoveries" step in finalize took 4+ hours every run, re-fetching ~29,600 apps even when nothing had changed.

Root cause

backfill_probe_discoveries() in scripts/pipeline/backfill.py determined what needed backfilling by scanning data_output_path (/tmp/protondb-output/data), which is rebuilt fresh each run from the probe-input artifact. That artifact only contains apps from the official dump + manifest backfill (~8K apps). The ~21K apps that were probe-backfilled in previous runs only existed on the deployed gh-pages branch and never made it into the artifact.

So every run, all ~21K probe-discovered apps appeared "missing" and got re-fetched one by one (2-3 HTTP requests each).

Fix

Two-part fix, both landed:

Python side (scripts/pipeline/backfill.py):

  • backfill_probe_discoveries() takes an already_known_app_ids set and unions it into the on-disk scan
  • run_probe_backfill() passes indexed_app_ids | backfill_app_ids from pipeline state

Workflow side (.github/workflows/update-data.yml, finalize job):

  • Checkout gh-pages with fetch-depth: 1 into gh-pages-data/
  • cp -rn gh-pages-data/data/. /tmp/protondb-output/data/ before "Backfill Probe Discoveries" runs
  • -rn (no-clobber) preserves the artifact's fresh files; gh-pages only fills historical gaps

The workflow piece is the load-bearing change. The Python side was already in place, but data_output_path.iterdir() only saw the artifact's small set until gh-pages data was merged in.

Impact

Probe-backfill drops from ~4 hours to seconds on no-change runs. The full pipeline now finishes in under 15 minutes when there's nothing new to fetch from ProtonDB.

Original evidence

From run 24434781792 (2026-04-15):

03:27:12 [probe-backfill] Backfilling 29,619 probe-discovered app(s)
03:27:12 [probe-backfill] (1/29619) Title for 10: 'Counter-Strike' via provided-catalog
...
07:46:08 [probe-backfill] Summary: attempted 29,619 app(s), succeeded 29,591, missed 28

4 hours 19 minutes to re-fetch apps that were already successfully backfilled in the previous run.

Static Output Files

The pipeline writes these files to /tmp/protondb-output/ during Stage 3 (finalize). All are deployed to the gh-pages branch and served as static JSON.

File Script Purpose
data/{appId}/ finalize.py Per-game ProtonDB + Pulse report year-bucket files
search-index.json finalize.py [appId, title, tier, protondbCount, pulseCount, appType] rows. Primary index: real data + Epic/GOG catalogs + Steam stubs filtered to ProtonDB-known apps
search-index-steam-extended.json finalize.py Long-tail Steam catalog stubs not in the primary index (no ProtonDB gate). Lazy-loaded by the grouped search page only. See Extended Steam Index
recent-reports.json finalize.py Top 100 games sorted by most recent report timestamp
most_played.json most_played.py Steam's current top charts with ProtonDB tier, rated + unrated + catalog-only buckets
steam-catalog.json most_played.py { appId: name } for Steam chart games with no ProtonDB data (see below)
game-images.json game_images.py { appId: url } for games where the standard CDN header URL 404s (hashed paths)
game-images-cache.json game_images.py Unified probe cache tracking status + probed date per app ID
release-years-cache.json release_years.py { appId: year } cache for Steam apps whose titles collide with another game's. Release dates never change, so this accumulates over time (#109 committed to gh-pages so the 200/run probe cap isn't burned from scratch on every run).
stats.json stats.py Aggregate counts for the /stats page

Steam Catalog Stub Pages

Some games appear in Steam's most-played charts but have no ProtonDB reports and no entry in search-index.json. Without a name source, the site previously skipped them entirely.

The pipeline now runs a catalog-only pass at the end of build_most_played():

  1. Any Steam chart game not in search-index.json is looked up in the cached Steam game catalog (.cache/steam-game-catalog.json, populated by the steam-catalog CLI step using STEAM_API_KEY).
  2. Matched games are appended to most_played.json with rating: "catalog" and zero report counts.
  3. steam-catalog.json is written alongside: a minimal { appId: name } map of just these catalog-only games.

The frontend uses steam-catalog.json in two ways:

  • Home page: rating: "catalog" entries are included in the "Not rated yet" toggle bucket alongside rating: "pending" games.
  • Game page: when no reports exist and the search-index has no entry, steam-catalog.json is fetched as a title fallback so a proper stub page is shown (with a submit CTA) instead of the generic "not in our mirror" error.

game_images.py treats all IDs in steam-catalog.json as hot, meaning their header images are always probed on each run instead of waiting for the backlog queue.

Extended Steam Index

The primary search-index.json includes Steam catalog stubs only for apps that ProtonDB knows about (its curated compatibility_report_with_games.json signal plus our probe cache). That gate keeps the file small but hides real Steam games that have ProtonDB reports yet are not in the curated signal. Issue #134 hit this with "Thank You For Your Application" (app 2881370).

To close that gap, generate_extended_steam_index writes a companion file at the end of finalize: every Steam catalog entry that is not already in the primary index, with no ProtonDB gate. Same row shape as the primary so the frontend can reuse the same render helper.

The frontend loads it lazily. The grouped search results page (renderSearchPage, hit by pressing Enter on a query) calls loadExtendedSteamIndex() alongside loadSearchIndex() and merges matches deduped by appId. The instant dropdown (onSearchInput) stays primary-only so typing never triggers a multi-megabyte fetch.

Header images for these stubs are handled by the backlog drain, not an upfront fetch. Most extended apps render fine because the frontend card tries the standard Steam CDN header.jpg path client-side first, with no pipeline involvement. Only the minority whose standard path 404s need a hashed URL stored in game-images.json. game_images.py reads search-index-steam-extended.json and appends its uncached IDs to the tail of the backlog (_extended_steam_ids), behind apps that have real reports. The same PROBE_CAP per run bounds the work, so the long tail trickles in over many runs rather than probing 140k IDs at once and tripping Steam rate limits. Until an extended app gets probed, a hashed-path game shows no image (for example app 3761210).

Automated Backups

The backup.yml workflow backs up Supabase data and a site snapshot to proton-pulse-data-backup.

When it runs

Trigger Frequency Commit label
Cron schedule Every Sunday at 04:00 UTC backup (scheduled): YYYY-MM-DD
After "Update ProtonDB Data" succeeds Every successful CI run backup (post-deploy): YYYY-MM-DD
Manual (workflow_dispatch) On demand backup (manual): YYYY-MM-DD

What is backed up

Archive Contents PII handling
backup-YYYY-MM-DD-schema.tar.gz SQL DDL for all public tables, RLS policies None
backup-YYYY-MM-DD-user_configs.tar.gz All visible game compatibility reports client_id + proton_pulse_user_id HMAC-pseudonymized; notes sanitized
backup-YYYY-MM-DD-author_avatars.tar.gz Steam avatar cache steam_id excluded; proton_pulse_user_id HMAC-pseudonymized
backup-YYYY-MM-DD-site.tar.gz Snapshot of deployed gh-pages HTML/JS/CSS None

Auth data (auth.users, admins, banned_users, claimed_client_ids) is never exported.

Backup log

Every run appends one line to backups.jsonl at the root of the backup repo. Each entry contains:

{
  "ts": "2026-06-22T16:00:00.000Z",
  "date": "2026-06-22",
  "trigger": "scheduled",
  "run_id": "12345678",
  "run_url": "https://github.com/mdeguzis/proton-pulse-web/actions/runs/12345678",
  "source_sha": "abc1234",
  "source_url": "https://github.com/mdeguzis/proton-pulse-web/commit/abc1234",
  "files": [
    { "name": "backup-2026-06-22-schema.tar.gz", "size_bytes": 8192, "row_count": null },
    { "name": "backup-2026-06-22-user_configs.tar.gz", "size_bytes": 204800, "row_count": 1234 }
  ]
}

How to trigger a manual backup

In the proton-pulse-web repo, go to Actions > Backup > Run workflow. Choose the backup type (default: all).

How to restore

See RESTORE.md in claude-configs for the full team handoff and restore guide.

Relationship to Other Wiki Pages

  • ProtonDB Data Resolution -- Documents the plugin-side lookup strategy (mirror -> live-detailed -> live-summary) that consumes the data this pipeline produces.
  • Architecture -- The src/lib/protondb.ts section covers how the plugin reads from the GitHub Pages mirror endpoints.
  • Supabase Voting -- Describes the user_configs Supabase table that the Pulse merge step reads from.

Clone this wiki locally