Store APIs and Images

Store APIs and Images (Dev Reference)

How the pipeline and the web app fetch game data and header images from each supported storefront. Living document; update in the same session as any pipeline change that touches a store API.

Steam

appdetails (metadata + header URL)

• Endpoint: https://store.steampowered.com/api/appdetails?appids=<id>&filters=basic
• No auth required. Rate limited (~200 requests / 5 min from a single IP; pipeline throttles via cache).
• Response: { "<id>": { "success": true, "data": { name, header_image, content_descriptors: {ids, notes}, ... } } }
• Used by: scripts/pipeline/game_images.py, scripts/pipeline/common.py (fetch_steam_title_with_source, fetch_steam_content_descriptors).
• Cache: .cache/steam-title-cache.json (30d TTL); .cache/steam-content-descriptors-cache.json (30d TTL).

Content descriptor IDs (adult filter)

Steam publishes numeric descriptor IDs on each app. The adult filter treats 3 and 4 as adult:

ID	Meaning	Filtered?
1	Some Nudity or Sexual Content	No (too broad — BG3, Cyberpunk 2077, Rust, GTA V)
2	Frequent Violence or Gore	No (most action games)
3	Adult Only Sexual Content	Yes
4	Frequent Nudity or Sexual Content	Yes
5	General Mature Content	No (CS2, DBD, Rust — mainstream M-rated)

Trust Steam's developer self-flagging: genuine adult titles (porn / hentai / VNs) carry ID 3 or 4. Mainstream M-rated games only get 1, 2, or 5.

Constant lives in scripts/pipeline/common.py::ADULT_DESCRIPTOR_IDS. Adding a new ID is a conscious edit; drop the test in tests/test_content_descriptors.py too.

Storefront tags vs content descriptors — do NOT confuse them. The Steam store page's Tags row ("Sexual Content", "Nudity", "Visual Novel") is user-curated community tagging. Content descriptors are a separate developer-set field returned via content_descriptors.ids in the appdetails JSON. Our filter uses only descriptors. A game with the "Sexual Content" tag but no descriptor 3/4 will not be filtered (correctly — mainstream games commonly carry that tag).

filters=basic gotcha — STRIPS content_descriptors

Steam's appdetails endpoint accepts a filters query parameter to trim the response. filters=basic removes content_descriptors from the payload. Any code path that needs the adult flag MUST call appdetails WITHOUT filters=basic:

• ✅ common.py::STEAM_APP_DETAILS_URL (no filter) — used by fetch_steam_content_descriptors. Correct.
• ⚠️ game_images.py::STEAM_APPDETAILS_URL (has filters=basic) — only used for header_image extraction. Do NOT reuse this URL if you need descriptors.

If the pipeline flips this by accident, every game silently ends up with adult: false because content_descriptors is None in the response. Caught this bug in prod on 2026-07-02 for Naughty Chat and 15k+ catalog stubs.

Cache poisoning on rate-limited fetches (#185)

A second, separate failure mode. Steam's appdetails is aggressively rate-limited (HTTP 429). The old caching wrote an empty descriptor list to .cache/steam-content-descriptors-cache.json on ANY non-success, including a throttled/failed fetch, and kept it for the full 30-day TTL. A single 429 during a run therefore locked a real adult game as adult: false for a month.

Concrete case: Naughty Chat (app 3580330). Its appdetails returns

"content_descriptors": { "ids": [1, 3, 4, 5], "notes": "sexting, nudity or sexual content, general mature content, sexual acts." }

3 and 4 are both in ADULT_DESCRIPTOR_IDS, so it should be flagged — but a throttled fetch had cached [].

Fix in common.py::fetch_steam_content_descriptors:

• A network error / 429 is NOT cached at all — the next run retries instead of poisoning the entry.
• success:false is cached with a short negative TTL (STEAM_DESCRIPTORS_NEGATIVE_TTL_SECONDS, 3 days) since it can be a transient rate-limit response, not a removed app.
• success:true (even with an empty id list) is a definitive result, cached for the full 30 days with an ok: true flag.
• Legacy entries with no ok flag: a non-empty list is treated as confirmed (keep 30d); an empty list is a suspect false negative and gets the short TTL so it re-fetches and self-heals.

The stale prod data heals on the next full pipeline run after the fix ships. Tests: tests/test_content_descriptors.py.

Which entries get descriptor-checked

Not every search-index row hits appdetails. Cost tradeoff:

Entry type	Descriptor check?	Notes
Steam game with local reports	Always	~16k API calls first run; cached 30d after that
Steam catalog stub (no reports) with adult-hint title	Yes	Only ~58 of 15k stubs match the regex (naughty|hentai|nsfw|adult|erotic|sexy|xxx|nude|nudity|lewd|kinky|18\+|sensual|waifu|ecchi|yuri|yaoi|bimbo). Force-refreshes to heal poisoned cache entries (#185).
Extended-index stub (no reports, no hint)	Gradual (#176)	Cached entries are used for free. Uncached stubs get descriptor-checked up to PIPELINE_STUB_ADULT_ENRICH_BUDGET per run (default 500 -- roughly 12 min at Steam's throttle). ~30 runs covers all ~15k stubs. Stable numeric-app_id sort order so the sequence is deterministic across runs. Extended index is 9-column shape ([id, title, tier, pdb, pulse, appType, releaseYear, delisted, adult]) so the frontend adult filter works uniformly.
GOG / Epic (any)	Skipped	No public no-auth API returns Steam-equivalent descriptors for those stores. Falls back to admin curation (#173).

is_adult_app_cached(app_id) in common.py is the read-only variant used by the extended-index enrichment path: returns None on cache miss so the caller can decide whether to spend the per-run budget. Stale negatives (past the short TTL) are treated as misses to self-heal poisoned entries (#185). Full-catalog enrichment (#176) shipped as gradual pass; expand the hint regex when new false-negatives surface between full-cache cycles.

Header image URL chain

Steam header images live at multiple CDNs. The frontend walks the chain on 404 (see js/app/lib/steam-img.js):

1. https://shared.akamai.steamstatic.com/store_item_assets/steam/apps/<id>/header.jpg — canonical.
2. https://cdn.cloudflare.steamstatic.com/steam/apps/<id>/header.jpg — fallback CDN.
3. Admin override URL from game-images.json — the pipeline seeds this from the box_art_overrides Supabase table before probing. Admin sets these via the Box Art Manager (Set URL / Upload buttons). Overrides beat every other source and survive every pipeline rerun. See "Admin overrides" section below.
4. Hashed URL from game-images.json — the pipeline finds the current asset hash via appdetails (data.header_image) when the canonical path 404s and stores the hashed URL there. Format: .../steam/apps/<id>/<sha>/header.jpg.
5. SteamGridDB URL from game-images.json — the pipeline falls through to SGDB when the standard CDN 404s AND appdetails has no header (usually delisted / freshly-missing apps). Community-contributed artwork; requires SGDB_API_KEY set. See the SGDB section below.
6. Hidden if all five fail.

Admin overrides

Supabase table box_art_overrides (app_id, image_url, source, set_by, created_at, updated_at) stores admin-curated header URLs. Two write paths, both gated on the manage_box_art permission:

• Set URL — admin pastes a URL; the edge function HEAD-probes it before writing so broken URLs never land in the table.
• Upload — admin picks a PNG/JPG/WebP up to 2 MB; the edge function uploads to the boxart Supabase Storage bucket (public read) and writes the resulting public URL into the table.
• Clear — deletes the row AND any bytes under boxart/<app_id>.<ext>.

Pipeline behaviour (scripts/pipeline/game_images.py):

• _fetch_admin_overrides() pulls all rows from /rest/v1/box_art_overrides at the start of build_game_images(), using SUPABASE_URL + SUPABASE_ANON_KEY (RLS grants SELECT to anon).
• Each override seeds the cache with status='override' before any probing, and the override app IDs are excluded from hot / backlog / extended-stub probe lists so we never overwrite an admin choice.
• The frontend export (game-images.json) includes override URLs first, then hashed, then SGDB.

Endpoints on the shared edge function image-refetch:

• POST source=set_override {app_id, url} — insert/update the row (JWT + manage_box_art required)
• POST source=upload_override multipart with app_id + file (JWT + manage_box_art required)
• POST source=clear_override {app_id} — delete the row and any bytes in the bucket (JWT + manage_box_art required)

Web scrape fallback

For apps behind an age gate (18+) where the JSON API returns success: false, common.py::_scrape_steam_store_title fetches https://store.steampowered.com/app/<id> with the wants_mature_content=1 cookie and greps apphub_AppName for the title.

SteamGridDB (fallback tier)

Community-contributed artwork DB. Used as the last-resort fallback — only when Steam's own APIs (standard CDN + appdetails) yield no header URL for a given appId. Prevents wasting SGDB rate limit (500 req/hour free tier) on apps Steam already covers.

• API base: https://www.steamgriddb.com/api/v2
• Auth: Authorization: Bearer <SGDB_API_KEY> on every request
• Key setup: get one at https://www.steamgriddb.com/profile/preferences/api. Set as SGDB_API_KEY in:
  • GitHub Actions secrets (pipeline: game_images.py)
  • Supabase edge function secrets (admin interactive refetch: image-refetch)
• Endpoints used:
  • GET /games/steam/<appid> → { data: { id } } — maps a Steam appid to SGDB's internal game id
  • GET /grids/game/<game_id>?dimensions=460x215&types=static → { data: [{ url, mime, ... }] }. We ask for header-shaped static grids only (460x215 matches Steam's header ratio; static excludes animated). Prefer PNG over JPG.
• Pipeline behavior: on fallback trigger, _fetch_sgdb_header(app_id) returns the top-voted URL (PNG-preferred), which then goes through _url_is_ok verification before landing in game-images.json with status='sgdb'.
• Silent on any failure: no API key, HTTP error, empty result, or 404 on the returned URL all fall through to status='missing' (same behaviour as before SGDB was wired).
• Non-Steam (GOG / Epic) SGDB integration is not in the pipeline yet — SGDB doesn't have first-class id lookup for those stores; needs a title-search path. Tracked as follow-up.

GOG

Catalog list

• Endpoint: https://catalog.gog.com/v1/catalog?limit=48&locale=en-US&page=<n>&order=asc:releaseDate&productType=in:game,pack
• No auth. Paginated (~48/page).
• Response contains products[].id (numeric productId) and products[].slug.
• Used by: scripts/pipeline/gog_catalog.py.
• Cache: .cache/gog-catalog-cache.json. Refreshed on every pipeline run (catalog is small).

Header images

• URL format: https://images.gog-statics.com/<sha256>.png where the SHA is stored per game.
• The pipeline extracts image URLs from catalog product responses and writes them to nonsteam-images.json keyed by gog:<productId>.
• No fallback CDN. If the URL 404s the image is broken.

Epic

GraphQL storefront

• Endpoint: https://store.epicgames.com/graphql
• No auth (public storefront queries), but requires:
  • Origin: https://store.epicgames.com
  • Referer: https://store.epicgames.com/en-US/browse
  • Standard browser User-Agent (Cloudflare gates on this).
• Used by: scripts/pipeline/epic_catalog.py.
• Response includes namespace, title, keyImages[] (with type = "OfferImageTall", "OfferImageWide", etc.).

Header images

• URLs come from keyImages[] in the GraphQL response — Epic hosts on their own CDN with hashed paths.
• Written to nonsteam-images.json keyed by epic:<namespace>.
• No fallback CDN.

Data files summary

File	Written by	Keyed by	Value
game-images.json	game_images.py (pipeline)	Steam appId	Hashed header URL for apps where the canonical path 404s
nonsteam-images.json	game_images.py + catalog scripts	gog:<id> or epic:<namespace>	Store CDN URL for GOG/Epic games
steam-catalog.json	most_played.py	Steam appId	Title for catalog-only apps (in Steam charts but not in the ProtonDB signal export)
.cache/steam-title-cache.json	common.py (local, not deployed)	Steam appId	{title, source, ts} — 30d TTL
.cache/steam-content-descriptors-cache.json	common.py (local, not deployed)	Steam appId	{ids, ts} — 30d TTL
.cache/gog-catalog-cache.json	gog_catalog.py (local, not deployed)	GOG productId	Full catalog snapshot
.cache/epic-catalog-cache.json	epic_catalog.py (local, not deployed)	Epic namespace	Full catalog snapshot

Store URL parsing (client-side)

js/lib/store-url-parser.js (with an inline copy in js/lib/topbar.js — kept in sync manually) parses pasted store URLs and returns { store, appId, canonicalId, slug }.

• Steam URLs yield the canonical numeric appId directly from the path.
• GOG / Epic URLs carry a human slug (not the internal canonical id). Slug → id resolution requires reading the catalog cache; the parser leaves that to the caller.
• Tests: tests/storeUrlParser.test.js.

Admin: Missing Box Art tab

/admin.html -> Missing Box Art loads all three cache files in the browser and lets an admin:

• Probe: HEAD the canonical URL for a row (Steam walks akamai -> cloudflare -> pipeline fallback; GOG/Epic re-probe the cached URL).
• Refetch: for Steam, hit appdetails and return the current header_image URL. For GOG/Epic, there is no public no-auth manifest API, so refetch is a re-probe of the cached URL.

Failed probes surface the HTTP status and error text so an admin can tell CORS, 404, region-lock, or removed-app apart. See js/admin/api/boxart.js and js/admin/components/boxart.js.

Update policy

Any change to a store API URL, response shape, or auth requirement lands in the same commit as the code that uses it. Cache files under .cache/ are gitignored — bumping their TTL constants is a code change and needs a matching update here.