-
Notifications
You must be signed in to change notification settings - Fork 0
Steam Depot Data
Deep dive into Steam's depot system, how SteamDB tracks depot creation/update dates, and practical approaches for proton-pulse-web to replicate this functionality.
Steam's official public Web API (partner.steamgames.com/doc/webapi) does not expose per-depot timestamps, manifest history, or OS-specific update dates. The store/api/appdetails endpoint returns app-level release dates but nothing about individual depots. We need this data to show users when a game's Linux depot was first added and when it was last updated.
App (e.g. 367520 Hollow Knight)
-> Depot 367521 (Windows content, oslist: "windows")
-> Depot 367522 (macOS content, oslist: "macos")
-> Depot 367523 (Linux content, oslist: "linux")
-> Depot 367524 (shared content, no oslist)
Each depot has:
- A depot ID (unique uint32)
- A config section with fields like
oslist(e.g. "windows", "macos", "linux") - A manifests section listing branch -> manifest_gid + timeupdated
- The manifest itself has a
creation_timein its binary header
PICS (Product Info Cache System) is Steam's internal metadata distribution system. It stores all app and package metadata as KeyValue (VDF) blobs. When you query PICS for an app, the depot section looks like:
"depots"
{
"367521"
{
"config"
{
"oslist" "windows"
}
"manifests"
{
"public"
{
"gid" "1234567890123456789"
"size" "5368709120"
"download" "2147483648"
}
}
"maxsize" "5368709120"
"depotfromapp" "367520"
}
"branches"
{
"public"
{
"buildid" "12345678"
"timeupdated" "1688000000"
}
}
}Critical insight: The timeupdated field lives at the BRANCH level (depots.branches.public.timeupdated), NOT per-depot. This means all depots in the same branch share one timestamp. This is why issue #226 exists: you cannot derive per-OS update dates from this field alone.
A common misconception: PICS change numbers are sequential unsigned integers, not Unix timestamps. They increment globally across all Steam apps/packages. The current numbers are in the millions (far below Unix timestamp range). When you call PICSGetChangesSince, the response contains change numbers and app/package IDs, but no timestamp fields. SteamDB records the wall-clock time when they observe each change number, which is how they derive "last updated" dates.
Steam only allows querying approximately 5,000 change numbers into the past. Beyond that, you get only the current change number with no historical data. Given thousands of daily PICS changes across Steam's catalog, this covers only hours to days of history. This is why continuous polling is mandatory: miss a few days and you lose that window permanently.
Each manifest (identified by gid) is a binary blob with a protobuf header. That header includes:
depot_id: uint32
creation_time: uint32 (unix timestamp - when THIS manifest was created/uploaded)
size_original: uint64
size_compressed: uint64
filenames_encrypted: bool
The creation_time in the manifest header IS per-depot and represents when that specific build was uploaded. This is the closest thing to a per-OS "last updated" date, but retrieving it requires downloading the manifest itself (a separate network call per depot).
Important ambiguity: SteamKit2's XML doc says CreationTime is "the depot creation time," but since each depot has MANY manifests over its lifetime (one per build), each manifest has its own creation_time. It likely represents when that specific manifest revision was generated, not when the depot was first created. To get the depot's original creation date, you would need the FIRST manifest ever published for that depot, which requires historical data you can only accumulate over time.
Manifest metadata fields (from content_manifest.proto): depot_id, gid_manifest, creation_time, filenames_encrypted, cb_disk_original, cb_disk_compressed, unique_chunks, crc_encrypted, crc_clear. That's it. No update_time, no OS field, no changelist number. OS info lives only in the PICS KeyValues blob.
SteamDB explains their methodology directly:
"We use SteamKit to interface with the Steam network. We request changes for all applications and packages once in a while, but mostly rely on Steam's own update system which tells us which applications and packages have updated."
"The basic app and package info we provide is publicly available from Steam itself, and can be acquired by anyone with a regular Steam account. If you launch Steam with
-console[...] and typeapp_info_print 440, it will display most of the information we have on our page for Team Fortress 2."
On changelists:
"When an app or package changes, Steam creates a 'changelist' to notify all clients. The changenumber increments globally and is not a per-app thing."
On manifests and depot access:
"Getting depot manifests requires ownership on your account." Since 2022, "manifest request codes are required for downloading manifests" and are "valid only for 15 minutes, and requires ownership of the depot to get a code."
On crowdsourcing access:
"App tokens allow our site to display information for upcoming games, deleted games, private betas, and otherwise hidden apps. Package tokens allow our site to display information for packages (subs) such as which apps they contain. Depot keys allow our site to display file list and file names, as well as tracking their updates."
SteamDB's FAQ points to xPaw/SteamWebPipes as a smaller-scale example of their approach. This is a C# application (by xPaw, who also maintains steamapi.xpaw.me and is a SteamDB maintainer) that:
- Connects to Steam CM servers using SteamKit2
- Subscribes to PICS changelist notifications
- Retransmits change events to WebSocket clients in real-time
Architecture:
Steam CM Servers -> [SteamKit2 PICS listener] -> SteamWebPipes -> [WebSocket] -> Clients
|
MySQL (app/package name lookup)
The repo powered SteamDB's real-time feed at steamdb.info/realtime/ until it was archived in October 2023 (likely replaced by their internal tooling). It demonstrates the exact pattern: connect once, listen for changelist pushes, fetch product info for changed apps.
-
PICS Change Listener - A long-lived Steam client connection (built on SteamKit2/.NET) subscribes to changelist notifications. Steam pushes a global changelist number every time any app or package metadata changes. SteamDB calls
PICSGetChangesSince(lastChangeNumber)to get a list of which apps/packages changed. -
Product Info Fetcher - For each changed app, they call
PICSGetProductInfo(appId)which returns the full VDF blob (including depot configs, branch timestamps, manifest GIDs). -
Manifest Fetcher - They download the actual manifests for changed depots to get file lists and
creation_time. This requires either owning the game OR having a depot key + access token (crowdsourced via Token Dumper). -
Historical Database - They store every changelist event with its timestamp. "First seen" dates come from their database of 10+ years of observations, not from any Steam API field. When depot 367523 (Linux) first appears in an app's PICS data, the changelist timestamp becomes its "creation date" in SteamDB.
-
Non-PICS sources - They also run "various scripts that check web APIs, store, community and others." Changes from these are "prefixed with
U:in history" since they lack a PICS changenumber.
SteamDB's "Date added" for a depot is the date they first observed that depot in the PICS data. There is no Steam API that returns "when was this depot first created." SteamDB has been tracking since ~2012, so for most games their first-observation date IS effectively the creation date. But if a game launched before SteamDB started tracking, the "added" date may be slightly wrong.
| Repo | Purpose | Useful to Us? |
|---|---|---|
| SteamKit2 | .NET library for Steam client protocol | Yes - PICS queries |
| ValvePython/steam | Python library for Steam client protocol | Yes - our pipeline is Python |
| SteamTokenDumper | Crowdsources access tokens | No - we use anonymous login |
| FileDetectionRuleSets | Detect engines from file lists | Maybe - secondary feature |
| SteamTracking | Tracks Steam client protobufs | Reference for protocol changes |
SteamDB's core crawler is NOT open source. Their backend (the changelist listener, fetcher, and database) is proprietary.
steamcmd +login anonymous +app_info_print 367520 +quitReturns the full PICS VDF blob. Anonymous login gives us:
- All depot IDs and their
config.oslist - Branch-level
timeupdated(shared across all depots in that branch) - Manifest GIDs per depot per branch
- Depot max sizes
Does NOT give us:
- Per-depot creation/update timestamps
- Historical changelist data
- Manifest
creation_time(that requires downloading the manifest)
Limitation: One timeupdated per branch, shared by ALL depots. If Windows and Linux depots both update in the same build, they get the same timestamp. If only Windows updates, ALL depots in that branch still show the new timestamp.
from steam.client import SteamClient
from steam.client.cdn import CDNClient
client = SteamClient()
client.anonymous_login()
# Get PICS product info (same data as steamcmd but programmatic)
info = client.get_product_info(apps=[367520])
app_info = info['apps'][367520]
depots = app_info.get('depots', {})
# Get changelist stream
changes = client.get_changes_since(last_change_number,
app_changes=True,
package_changes=False)
# changes.current_change_number -> new baseline
# changes.app_changes -> list of {appid, change_number}
# Get manifest (requires ownership or depot key for non-free games)
cdn = CDNClient(client)
manifests = cdn.get_manifests(367520)
for m in manifests:
print(f"Depot {m.depot_id}: creation_time={m.creation_time}")Key methods:
-
get_product_info(apps=[...])- same as steamcmd app_info_print but returns parsed dict -
get_changes_since(change_number)- poll for changelist updates -
get_access_tokens(app_ids=[...])- get tokens for owned apps -
CDNClient.get_manifests(app_id)- download manifests (hascreation_timeper manifest)
Anonymous login limitations:
- Can query PICS metadata for all apps (depot list, config, branch info)
- CANNOT download manifests for paid games (need ownership or depot keys)
- CAN download manifests for free-to-play games
GET https://api.steampowered.com/IContentServerDirectoryService/GetDepotPatchInfo/v1/
?appid=367520
&depotid=367521
&source_manifestid=<old_gid>
&target_manifestid=<new_gid>
Returns patch info between two manifests. Requires knowing both manifest GIDs. Useful for understanding delta size but NOT for getting creation dates or history.
GET https://partner.steam-api.com/ISteamApps/GetAppBuilds/v1/?appid=367520&depot_details=true
GET https://partner.steam-api.com/ISteamApps/GetAppDepotVersions/v1/?appid=367520
These return build history WITH depot details and depot versions. BUT they require a Steamworks publisher API key for that specific app. We don't have publisher access to other people's games, so this is not viable for a general tracker.
There is NO Steam API that returns "when was this depot first created." Here's what exists:
| Data Point | Source | Availability | Per-Depot? |
|---|---|---|---|
Branch timeupdated
|
PICS (steamcmd/SteamKit) | Anonymous | No (per-branch) |
Manifest creation_time
|
Manifest binary header | Ownership required | Yes |
| Manifest GID changes | Observational (track over time) | Anonymous | Yes |
| "First seen" date | Your own database of observations | Self-tracked | Yes |
| Build history | ISteamApps publisher API | Publisher key required | Yes |
Bottom line: To get a per-depot "first seen" / "last updated" that is truly per-OS, you must either:
- Track manifest GID changes over time (observe when they change) - this is what SteamDB does
- Download the manifest to read
creation_time(requires game ownership for paid games) - Use the branch
timeupdatedas an approximation (shared across all OS depots)
What you get: depot list, OS mapping, branch-level timeupdated, manifest GIDs.
Store in steam_depot_updates:
CREATE TABLE steam_depot_updates (
app_id integer NOT NULL,
depot_id integer NOT NULL,
os text NOT NULL, -- from config.oslist
manifest_id text, -- current manifest GID
last_updated_at timestamptz, -- branch timeupdated (approximate)
PRIMARY KEY (app_id, depot_id, os)
);Limitation: last_updated_at is branch-level, same for all OS depots in one app. This is your MVP.
Track manifest GID changes over time to build your own "first seen" data:
CREATE TABLE steam_depot_manifest_history (
app_id integer NOT NULL,
depot_id integer NOT NULL,
os text NOT NULL,
manifest_id text NOT NULL,
first_observed_at timestamptz NOT NULL DEFAULT now(),
latest_observed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (app_id, depot_id, os, manifest_id)
);On each nightly run:
- If
(app_id, depot_id, os, manifest_id)is new -> INSERT (first observation of this build) - If it exists -> UPDATE
latest_observed_at - First ever observation of a depot_id for an OS = "first seen this OS"
What this gives you that Phase 1 doesn't:
- Per-OS "first seen" dates (when a Linux depot FIRST appeared for a game)
- Per-OS "last updated" (when the manifest GID last changed for THAT depot)
- Build history (how many manifest changes per depot over time)
Replace nightly polling with a persistent connection:
from steam.client import SteamClient
client = SteamClient()
client.anonymous_login()
last_change = load_last_change_number()
while True:
changes = client.get_changes_since(last_change, app_changes=True)
if changes is None:
sleep(30)
continue
for app_change in changes.app_changes:
if app_change.appid in our_tracked_apps:
info = client.get_product_info(apps=[app_change.appid])
process_depot_update(info)
last_change = changes.current_change_number
save_last_change_number(last_change)
sleep(5) # Steam pushes changes frequentlyBenefits over nightly polling:
- Near-real-time updates (seconds, not 24h)
- No cold-start overhead (persistent connection)
- Precise timestamps (changelist events have server-side timestamps)
- Can track thousands of apps without rate limit issues
Requirements:
- Long-lived process (VPS, Fly.io, etc. - not serverless)
- Session persistence (save credentials, reconnect on drop)
- State persistence (last change number)
For free-to-play games, you can get the actual manifest creation_time:
cdn = CDNClient(client)
manifests = cdn.get_manifests(app_id, filter_func=lambda did, info: 'linux' in info.get('config', {}).get('oslist', ''))
for m in manifests:
# m.creation_time is the REAL per-depot upload timestamp
store_manifest_creation_time(app_id, m.depot_id, m.creation_time)For paid games, this requires the SteamTokenDumper approach (crowdsourcing depot keys) or owning the game on a bot account. Not practical at scale without significant investment.
Short answer: Partially. Here's what we can and cannot replicate:
- Get current depot list per app with OS mapping (anonymous PICS)
- Get branch-level
timeupdated(approximate last update) - Get current manifest GIDs (track changes going forward)
- Get manifest
creation_timefor free games (real per-depot timestamp) - Track all future changes with precise timestamps
- Historical "first seen" dates for depots added before we start tracking. SteamDB has been observing since ~2012. If Hollow Knight added a Linux depot in 2017, SteamDB knows because they saw that changelist. We would only know "it existed when we first checked."
- Manifest creation_time for paid games (without buying them or crowdsourcing keys)
- Use the app's Steam release date as a floor - if a game released in 2017 and has a Linux depot, the depot was added between release and now.
- Use ProtonDB report dates - if the first Linux report appeared in 2018, the Linux depot likely existed by then.
- Use SteamDB as a reference for historical data (manual lookups for important games, or a one-time backfill from their public pages).
- Accept "first seen by us" dates - clearly label them as "Tracked since " rather than implying they're creation dates.
- Manifest creation_time for free games gives you real dates without any history.
This site documents both official and undocumented Steam Web API endpoints parsed from protobuf files and GetSupportedAPIList. Key findings:
| Interface | Method | Auth | Useful? |
|---|---|---|---|
| ISteamApps | GetAppBuilds | Publisher | Yes but we don't have publisher keys |
| ISteamApps | GetAppDepotVersions | Publisher | Same limitation |
| ISteamApps | UpToDateCheck | Public | Only checks if current, no history |
| IContentServerDirectoryService | GetDepotPatchInfo | Public | Needs both manifest GIDs, no timestamps |
| IContentServerDirectoryService | GetServersForSteamPipe | Public | CDN server list, not useful for metadata |
| IProductInfoService | (internal only) | N/A | NOT exposed via web API |
- PICS is NOT available via Web API. The only way to access PICS data is through the Steam client protocol (SteamKit2, ValvePython/steam, steamcmd). There is no REST endpoint for it.
- Publisher endpoints exist for builds/depots but require per-app publisher keys.
- No undocumented public endpoint exists for depot history or creation dates.
Phase 1 (MVP, now): steamcmd nightly poll -> branch-level timeupdated per app
Phase 2 (next): Track manifest_id changes over time -> per-OS first_seen / last_changed
Phase 3 (scale): ValvePython/steam persistent listener -> near-real-time updates
Phase 4 (nice-to-have): Manifest creation_time for free games -> real per-depot upload dates
The key architectural decision: accept that "first seen" will be observational from our start date forward. Clearly communicate this in the UI ("Tracked since 2026-07-07" or "Linux depot first observed: ..."). SteamDB had 10+ years to build their dataset. We start accumulating now and the data improves every day.
SteamDB FAQ recommends DepotDownloader for downloading specific depots and manifests. Built on SteamKit2. Can download historical manifests by ID (-manifest <id> flag) but cannot enumerate available manifests or browse depot history. You must already know the manifest GID (typically from SteamDB). GitHub Issue #260 requesting manifest listing remains open since 2021.
Node.js equivalent of ValvePython/steam. Has PICS support including getProductInfo(), getProductChanges(), and access token methods. Default polling interval for changelists is 60 seconds (changelistUpdateInterval), configurable down to 1 second minimum. Relevant if you ever move the listener to a Node.js runtime.
A small tool that "downloads appinfo files when new releases are seen via SteamWebPipes." Demonstrates the pattern of reacting to PICS change events and fetching product info.
CLI tool built on ValvePython/steam. Can query PICS data from the command line similar to steamcmd but with more scripting-friendly output formats.
-
Does manifest
creation_timerepresent the depot's original creation or just that manifest revision? If you fetch the current manifest for depot 367521, does creation_time give you the date the depot was first created, or only the date the latest manifest was built? Testing with a free game would answer this. -
Can we enumerate historical manifest GIDs for a depot? Steam's undocumented GetManifestRequestCode and CDN endpoints may have unlisted capabilities. DepotDownloader can't do this (Issue #260 open since 2021).
-
What real-time window does the ~5,000 changelist lookback cover? Given thousands of daily changes, this is probably hours not days. Determines how quickly a new tracker must start polling to avoid gaps.
-
Does the PICS KeyValues blob contain any depot-level timestamp fields beyond what we've identified? Some games may have additional keys not present in others.
Alongside the Supabase tables, the pipeline emits a compact JSON per Steam app under data/{appId}/depots.json on gh-pages. This gives us three things:
- A stable, cacheable URL any consumer (Metadata modal, CLI script, external tool) can hit without going through Supabase.
- A verbatim copy of the full parsed PICS
depotsblock (branches, sharedinstall, dlc_appids, encryptedmanifests, config.oslist, manifests.public.gid + timeupdated, ...). Storing the raw parse means adding a new UI field never requires a re-fetch. - A deep-link target for the Metadata modal's brown package icon so admins can jump to the raw data behind any row.
Shape:
{
"app_id": 367520,
"fetched_at": "...",
"status": "ok",
"os": {
"linux": {
"depots": 1,
"last_updated": "2026-03-27T00:00:00Z",
"tracked_since": "2025-06-15T00:00:00Z",
"manifests": [
{ "depot_id": 367521, "manifest_id": "1", "first_observed_at": "...", "latest_observed_at": "..." }
]
},
"windows": { "..." : "..." }
},
"raw_pics": {
"367521": { "config": {}, "manifests": {} },
"branches": { "public": {} }
}
}Population:
-
scripts/pipeline/steam_metadata.pyruns nightly (steam-metadata-fetch.yml), parses+app_info_printoutput, and writes:- Narrow rows to
steam_depot_updates(OS rollup,last_updated_at). - Observation rows to
steam_depot_manifest_history(first_observed_at,latest_observed_at). - The full parsed
depotsdict tosteam_depot_fetch_status.raw_pics(JSONB).
- Narrow rows to
-
scripts/pipeline/write_depot_files.pyruns during finalize (update-data.yml), joins those three tables per app, and writesdata/{appId}/depots.jsonundergh-pages.
Rules:
-
tracked_sinceis only reported when we have a realfirst_observed_atfor that OS. Fallback tolast_updatedor the release date would be a lie. - Non-Steam ids (GOG, Epic) are skipped -- no PICS.
- The Metadata modal's
steam-depot-infoedge fn reads the tables directly (fresh reads for accuracy). The staticdepots.jsonfile is a deep-link + archive target, not the modal's primary data source.
- ValvePython/steam docs - Python PICS client
- SteamRE/SteamKit - .NET PICS client (used by SteamDB)
- xPaw/SteamWebPipes - Reference impl of PICS listener (archived, C#)
- SteamDatabase/SteamTokenDumper - How SteamDB crowdsources access
- SteamDB FAQ - How SteamDB collects data (primary source)
- steamapi.xpaw.me - Undocumented Steam API reference
- Steamworks Depot Docs - Official depot documentation
- SteamDatabase/SteamTracking - Tracks Steam protobuf changes
- proton-pulse-web issues: #214, #215, #216, #226