Regenerating Protobufs

Regenerating Protobufs

Steam periodically changes its wire format — new messages, renamed fields, deleted RPCs. Keeping the fork current means regenerating the _pb2.py + _pb2.pyi files under steam/protobufs/ from fresh .proto sources.

This page covers the four poetry-registered scripts that do the work.

Prerequisites

• protoc on PATH. poetry run pb-compile refuses to run without it.
  • macOS: brew install protobuf
  • Debian / Ubuntu: sudo apt install protobuf-compiler
  • Verify: protoc --version — expect libprotoc 3.19+, ideally 5.x or 6.x.
• Dev deps: poetry install --with dev — provides mypy-protobuf (emits the .pyi companions) and types-protobuf (typeshed for the runtime).
• Internet — pb-fetch downloads from GitHub.

Fast path: full refresh

poetry run pb-update

Runs the four sub-scripts in order (pb-fetch → pb-compile → pb-services → pb-gen-enums). Short-circuits on the first non-zero exit code so a partial refresh doesn't leave the tree in a half-migrated state.

For most changes — Valve added a new message, renamed a field, added a service — that's all you need.

Individual steps

poetry run pb-fetch

Downloads .proto files listed in protobuf_list.txt into protobufs/.

Reads URLs from protobuf_list.txt line by line, skipping blank lines and # comments. Downloads them in parallel (8 concurrent workers). For each downloaded file:

• Rename *.steamclient.proto → *.proto.
• Prepend syntax = "proto2"; if missing (idempotent).
• Swap cc_generic_services → py_generic_services.
• Rewrite intra-file .steamclient.proto references to .proto.

Locally-maintained protos are protected. gc.proto and test_messages.proto are renamed to *.notouch before the fetch runs and restored after, so an upstream 404 or the file being missing from protobuf_list.txt can never nuke them.

Output: prints downloaded N ok, M missing at the end.

poetry run pb-compile

Invokes protoc to turn protobufs/*.proto → steam/protobufs/*_pb2.{py,pyi}.

1. Wipes any existing steam/protobufs/*_pb2.{py,pyi} first — a failure surfaces as "no files written" instead of a mix of stale and new.
2. Single protoc invocation over all .proto files:

   protoc --python_out=steam/protobufs --mypy_out=steam/protobufs
          --proto_path=protobufs protobufs/*.proto
   

3. Post-process:
   • .py — sibling protobuf imports get the steam.protobufs. prefix so runtime import works without steam/protobufs/ on sys.path.
   • .pyi — sibling imports get the same prefix; per-message DESCRIPTOR: _descriptor.Descriptor overrides are stripped. See Type Checking for why.

Output: compiled N .py + M .pyi under steam/protobufs.

poetry run pb-services

Regenerates the ServiceName -> _pb2 module map in steam/core/msg/unified.py.

Scans every protobufs/*.proto for top-level service Foo { rpc Bar(...); } declarations and rewrites the dict body between MARK_SERVICE_START and MARK_SERVICE_END inline-comment markers.

The service_lookup dict is what makes client.send_um('Player.GetGameBadgeLevels#1', ...) work — it maps the service name (Player) to the module that defines its methods.

Output: wrote N service registrations to unified.py.

poetry run pb-gen-enums

Regenerates steam/enums/proto.py from every compiled *_pb2 module.

Scans steam/protobufs/*_pb2.py for top-level EnumTypeWrapper instances, filters out any already declared in steam/enums/common.py (which is hand-written), strips the k_<EnumName>_ prefix that Valve puts on member names, and emits a Python file of proper SteamIntEnum classes.

Members that collide with Python keywords or start with a digit fall back to the factory form SteamIntEnum(name, {member: value}) — the class X(SteamIntEnum): NAME = VALUE syntax can't express those.

Output: wrote N enum classes to steam/enums/proto.py.

Adding a new upstream proto file

Say Valve just added a new proto file at SteamDatabase/Protobufs/steam/foo_bar_baz.proto.

1. Edit protobuf_list.txt — add the raw GitHub URL:

   https://raw.githubusercontent.com/SteamDatabase/Protobufs/master/steam/foo_bar_baz.proto
   

   Keep the file sorted by domain / directory to match the existing structure. Comments (# lines) are ignored, feel free to annotate.

2. Run the full refresh:

   poetry run pb-update

3. Confirm the new module appears:

   ls steam/protobufs/foo_bar_baz_pb2.py

4. If it declared any services, pb-services will have already picked them up. If it declared any enums, pb-gen-enums folded them into steam/enums/proto.py.

5. Run the test suite to make sure nothing regressed:

   poetry run pytest

Removing a stale proto

Delete the URL from protobuf_list.txt, delete protobufs/foo.proto and steam/protobufs/foo_pb2.{py,pyi}, then re-run pb-update. pb-services and pb-gen-enums rescan from scratch so stale entries drop out automatically.

Locally-maintained protos

Two protos in protobufs/ are not in protobuf_list.txt because they don't exist upstream in fetchable form:

• gc.proto — game-coordinator base messages.
• test_messages.proto — used by the test suite.

pb-fetch protects both via .notouch renaming.

Layout summary

protobufs/                       # .proto sources (fetched)
protobuf_list.txt                # URL list for pb-fetch
steam/protobufs/                 # generated *_pb2.py + *_pb2.pyi (do not hand-edit)
steam/core/msg/unified.py        # service map (regen'd by pb-services)
steam/enums/proto.py             # protobuf enums (regen'd by pb-gen-enums)
scripts/pb_*.py                  # the four scripts + pb_postprocess + pb_update runner

Where to go next

• Type checking hiccups after regeneration? Type Checking covers the mypy-protobuf / pyright interplay.
• Wondering what changed vs. upstream in the last regen? Fork Changes has the fork's protobuf-modernization diff.