Skip to content

Data Pipeline

Jump to bottom
mdeguzis edited this page Jun 9, 2026 · 4 revisions

Data Pipeline

CI/CD architecture for the proton-pulse-web GitHub Pages mirror. This pipeline runs daily via GitHub Actions and produces the per-game report files the plugin consumes.

Pipeline Stages

The update-data.yml workflow has three main stages:

Stage 1: build          Stage 2: probe-chunks        Stage 3: finalize
-----------------       ---------------------        ------------------
Process official     -> Probe ProtonDB summaries  -> Backfill probe
  ProtonDB dump           in parallel chunks           discoveries
Manifest backfill       (resumable, cache-backed)    Merge Pulse Reports
Compute probe plan                                   Build indexes
                                                     Coverage report
                                                     Deploy to gh-pages

Stage 1 -- build

  1. Process Official Reports -- Parses the bdefore/protondb-data monthly dump into data/{appId}/{year}.json year-bucket files.
  2. Manifest Backfill -- Apps listed in config/live_backfill_app_ids.json that are missing from the dump are fetched from ProtonDB's live detailed report endpoint.
  3. Build Probe Chunk Plan -- Computes which Steam app IDs still need a ProtonDB summary probe, splits them into chunks for parallel execution, and outputs the chunk matrix.

Outputs: probe-input artifact (data files + pipeline-state.json) and the chunk matrix for Stage 2.

Stage 2 -- probe-chunks

Runs the ProtonDB summary probe in sequential chunks (max-parallel: 1). Each chunk:

  • Restores the pipeline cache (.cache/protondb-summary-probe-cache.json)
  • Probes a slice of app IDs against ProtonDB's summary API
  • Saves the updated cache under a fresh cache key

The chunked design with cache checkpoints means an interrupted multi-hour run can resume from the latest completed chunk instead of restarting.

Skipped entirely when chunk_count == 0 (all apps already cached).

Stage 3 -- finalize

  1. Backfill Probe Discoveries -- Apps the probe found on ProtonDB but missing from local data get their live detailed reports fetched and written to year-bucket files.
  2. Merge Pulse Reports -- Pulls user-submitted configs from Supabase (user_configs table) and merges them into the same year-bucket files alongside ProtonDB data. See Pulse Report Merge below.
  3. Finalize -- Builds index.json, latest.json, coverage report, search index, and data-index page.
  4. Deploy -- Force-pushes an orphan gh-pages branch with the full dataset plus static web app files.

Pulse Report Merge

Pulse Reports are the plugin's own data: hardware-tagged configs submitted via the Decky UI or the web form. They live in Supabase (user_configs table) and were historically fetched live by the web UI. As of proton-pulse-web#TBD, the pipeline snapshots them into the same static JSON files that hold ProtonDB reports, so consumers (plugin, web, third-party scripts) only need to read one place.

Where it runs

scripts/pipeline/pulse.py -- called from finalize_output() right after generate_latest_files() and before app indexes are built, so latest/index files pick up the freshly merged Pulse records.

How records are tagged

Every report in a year file now carries a source field so consumers can filter cleanly:

source Origin Schema
"protondb" ProtonDB archive or live fetch Base shape: cpu, gpu, protonVersion, rating, notes, timestamp, ...
"pulse" Supabase user_configs row Base shape plus launchOptions, formResponses, configKey, gameOwned, vramMb, durationMinutes, pulseId, submissionSource

Pulse records preserve the granular submission origin in submissionSource (e.g. "user" for the Decky plugin, "web-linux" for the web form), so the broader source: "pulse" tag never loses that information.

Dedup behavior

The Supabase row id is stored on each Pulse record as pulseId. On every pipeline run, existing pulse records in the year file with a pulseId matching an incoming row are replaced, not duplicated. This means users editing their submissions in Supabase have those edits reflected in the static snapshot on the next pipeline run.

ProtonDB records continue to dedupe by timestamp (handled in process.py, unchanged).

Bucketing

Pulse rows are bucketed by year using created_at. A submission from 2025-03-14 lands in data/{appId}/2025.json. Years are extracted from the ISO timestamp via datetime.fromisoformat() in UTC.

Backfill safety

The merge function backfills source: "protondb" on any legacy untagged ProtonDB records it encounters in the same file. So after one full pipeline run, every record in every year file is self-describing.

Failure mode

If Supabase is unreachable (network, rate limit, schema mismatch), the function logs the error and returns without touching the year files. ProtonDB data still ships as normal.

Credentials

Defaults to the production Supabase project's anon publishable key, which is read-only by RLS policy. Forks / staging can override via env:

SUPABASE_URL=https://<project>.supabase.co/rest/v1 \
SUPABASE_ANON_KEY=<your_anon_key> \
make gh-run

Pipeline Cache

The .cache/ directory is persisted across runs via GitHub Actions actions/cache. It stores:

  • protondb-summary-probe-cache.json -- Which apps have been probed and their summary results
  • Steam title cache -- Resolved game titles to avoid repeated Steam Store API calls

Each stage saves its cache under a unique key (build, chunk-N, final) so they don't collide.

Environment Variables

Variable Default Purpose
PROTONDB_PROBE_LIMIT 5000 Max apps to probe per chunk
PROTONDB_PROBE_BACKFILL_LIMIT 0 (unlimited) Max apps to backfill from probe discoveries
PROTONDB_PROBE_LOG_EVERY 100 Log progress every N apps during probe

Resolved: probe-backfill Re-fetching All Apps

Original symptom

The "Backfill Probe Discoveries" step in finalize took 4+ hours every run, re-fetching ~29,600 apps even when nothing had changed.

Root cause

backfill_probe_discoveries() in scripts/pipeline/backfill.py determined what needed backfilling by scanning data_output_path (/tmp/protondb-output/data), which is rebuilt fresh each run from the probe-input artifact. That artifact only contains apps from the official dump + manifest backfill (~8K apps). The ~21K apps that were probe-backfilled in previous runs only existed on the deployed gh-pages branch and never made it into the artifact.

So every run, all ~21K probe-discovered apps appeared "missing" and got re-fetched one by one (2-3 HTTP requests each).

Fix

Two-part fix, both landed:

Python side (scripts/pipeline/backfill.py):

  • backfill_probe_discoveries() takes an already_known_app_ids set and unions it into the on-disk scan
  • run_probe_backfill() passes indexed_app_ids | backfill_app_ids from pipeline state

Workflow side (.github/workflows/update-data.yml, finalize job):

  • Checkout gh-pages with fetch-depth: 1 into gh-pages-data/
  • cp -rn gh-pages-data/data/. /tmp/protondb-output/data/ before "Backfill Probe Discoveries" runs
  • -rn (no-clobber) preserves the artifact's fresh files; gh-pages only fills historical gaps

The workflow piece is the load-bearing change. The Python side was already in place, but data_output_path.iterdir() only saw the artifact's small set until gh-pages data was merged in.

Impact

Probe-backfill drops from ~4 hours to seconds on no-change runs. The full pipeline now finishes in under 15 minutes when there's nothing new to fetch from ProtonDB.

Original evidence

From run 24434781792 (2026-04-15):

03:27:12 [probe-backfill] Backfilling 29,619 probe-discovered app(s)
03:27:12 [probe-backfill] (1/29619) Title for 10: 'Counter-Strike' via provided-catalog
...
07:46:08 [probe-backfill] Summary: attempted 29,619 app(s), succeeded 29,591, missed 28

4 hours 19 minutes to re-fetch apps that were already successfully backfilled in the previous run.

Relationship to Other Wiki Pages

  • ProtonDB Data Resolution -- Documents the plugin-side lookup strategy (mirror -> live-detailed -> live-summary) that consumes the data this pipeline produces.
  • Architecture -- The src/lib/protondb.ts section covers how the plugin reads from the GitHub Pages mirror endpoints.
  • Supabase Voting -- Describes the user_configs Supabase table that the Pulse merge step reads from.

Clone this wiki locally