Skip to content

Visual Mods Pipeline

M T edited this page Oct 4, 2026 · 1 revision

Visual Mods Pipeline

Halo Vision ships optional, exact-byte visual replacements: a native texture pack (TextureMods.hvt, format HVTEX001, 1,068 entries) and a pixel-shader pack (ShaderMods.hvs, format HVSHD001, 13 entries), accompanied by CEnshineSources.zip (upstream shader source and license). None of these binaries are in Git. They are published as a checksum-pinned release archive, fetched and verified by tools/visual_assets.py during setup and every build, bundled into the app as resources, and memory-mapped or loaded by the Direct3D9 bridge at runtime, where they replace original textures by CRC and original pixel shaders by exact bytecode. This page documents the manifest, the download and tamper-resistance logic, both pack formats, the three offline authoring tools (TexMod import, HD merge, CEnshine shader adaptation), and the runtime lookup.

Source files

File Role
mods/visual-assets.json Public manifest pinning each pack's size and SHA-256, entry counts, and the release archive URL, size and SHA-256.
tools/visual_assets.py ensure_visual_assets() (reuse, Complete-bundle copy, or verified download and extraction) and bundle_visual_assets() (copy into an app and verify).
tools/import_texmod_pack.py Converts an owner-supplied TexMod .tpf (or a ZIP containing one) into an HVTEX001 pack.
tools/merge_hd_texture_pack.py Merges reviewed PNG textures into an existing HVTEX001 pack, losslessly and digest-bound.
tools/import_censhine_pack.py Adapts the reviewed 13-program CEnshine 1.0.0 subset to the retail shader constant ABI and writes an HVSHD001 pack.
native/EngineHost/texture_mod_pack.h, texture_mod_runtime.inc Runtime HVTEX001 validation, CRC lookup and upload.
native/EngineHost/shader_mod_pack.h, shader_mod_runtime.inc Runtime HVSHD001 validation and exact-byte lookup.
native/EngineVision/Sources/EngineVisionRuntime.m:651-654 Points the runtime at the bundled packs.
Tests: tools/test_visual_assets.py, tools/test_texmod_import.py, tools/test_hd_texture_merge.py, tools/test_censhine_import.py, native/EngineHost/tests/test_texture_mod_pack.c, test_shader_mod_pack.c, test_d3d9_texture_mod.c See Testing.
docs/ASSET_NOTICES.md Provenance and licensing of the packs.

End-to-end flow

flowchart TD
    subgraph Authoring["Offline authoring (maintainer, optional)"]
        TPF["Owner-supplied TexMod .tpf"] --> TI["import_texmod_pack.py"] --> HVT0["HVTEX001 base pack + receipt"]
        PNG["Reviewed HD PNGs + review manifest"] --> HM["merge_hd_texture_pack.py"]
        HVT0 --> HM --> HVT["TextureMods.hvt + receipt"]
        FX["Composer-decoded retail fx.bin"] --> CI["import_censhine_pack.py"]
        CE["Decoded CEnshine 1.0.0 collection"] --> CI --> HVS["ShaderMods.hvs + report"]
    end
    HVT --> ZIP["MasterChef-v1.0.3-visual-assets.zip (GitHub release)"]
    HVS --> ZIP
    SRC["CEnshineSources.zip"] --> ZIP
    ZIP --> EVA["ensure_visual_assets(): verify archive, extract, verify files"]
    CB["Complete bundle Release/HaloVision.app"] --> EVA
    MAN["mods/visual-assets.json"] --> EVA
    EVA --> VM[".setup/VisualMods/ + VisualModsManifest.json"]
    VM --> XC["Xcode resources (project.yml)"]
    VM --> DB["bundle_visual_assets() into direct-build app"]
    XC --> APP["HaloVision.app resources"]
    DB --> APP
    APP --> RT["EngineVisionRuntime.m sets HALO_TEXTURE_PACK / HALO_SHADER_PACK"]
    RT --> TR["texture_mod_runtime.inc: mmap, CRC lookup per texture upload"]
    RT --> SR["shader_mod_runtime.inc: exact-byte lookup at CreatePixelShader"]
Loading

The public manifest (mods/visual-assets.json)

Field Value at this commit Meaning
formatVersion 1
version 1.0.3 Release the packs belong to
runtimeBaseline Build91 The owner build whose installed packs these are byte-identical to
textureEntries / shaderEntries 1068 / 13 Printed after a successful install
files[] TextureMods.hvt 2,876,332,440 bytes; ShaderMods.hvs 16,268 bytes; CEnshineSources.zip 105,408 bytes, each with SHA-256 The exact files that must exist after installation
archive.url https://github.com/mitchaiet/master-chef/releases/download/v1.0.3/MasterChef-v1.0.3-visual-assets.zip Download source
archive.bytes / archive.sha256 963,216,022 / e1cbe0fafd0367bc039f8ce0dc8555889ceeb12eac709b9bd6532cc22f1c6297 Outer archive pin
notes "Exact runtime packs from the installed Build91; no product keys, saved profiles or Apple provisioning."

The set of file names is hard-coded as NAMES = {'TextureMods.hvt', 'ShaderMods.hvs', 'CEnshineSources.zip'} (visual_assets.py:19); a manifest listing anything else is rejected. ASSET_NOTICES.md states the 1,068 entries are "the full installed pack, not a claim that every original texture was repainted", combining HD-mod inputs credited to Delta117, source-faithful upscales and reviewed image edits.

The sibling mods/runtime-settings.json is a record of the compiled startup defaults (render resolution 2048x1536, 30 FPS target, environment defaults such as HALO_PANORAMA=1); FEATURES.md says "it is not a separate settings loader", and no code in the repository reads it. See Runtime Settings.

Installing the packs (ensure_visual_assets)

ensure_visual_assets(root=ROOT, archive=None) (visual_assets.py:48-101) is called by setup_halo.build_engine() and twice by build_engine_vision.py (preamble and direct bundle step). It returns the path .setup/VisualMods.

flowchart TD
    A["load mods/visual-assets.json; file names must equal NAMES"] --> B{".setup or .setup/VisualMods is a symlink?"}
    B -->|yes| X1["ValueError: must not be symbolic links"]
    B -->|no| C["mkdir .setup (0700); flock .setup/visual-assets.lock (blocking)"]
    C --> D{".setup/VisualMods exists?"}
    D -->|yes| E["verify_files(VisualMods); return (no network)"]
    D -->|no| F{"Release/HaloVision.app exists?"}
    F -->|yes| G["verify_files(bundle); clone_copy into temp; write VisualModsManifest.json; rename to VisualMods"]
    F -->|no| H["disk check: files + archive + 1 GiB"]
    H --> I{"--archive given?"}
    I -->|no| J["URL must start with the project's releases/download prefix; stream download; abort if larger than pinned size"]
    I -->|yes| K["use local ZIP"]
    J --> L["archive size and SHA-256 must match"]
    K --> L
    L --> M["extract_verified() into temp/unpacked"]
    M --> N["write VisualModsManifest.json; rename to VisualMods"]
    N --> O["print: Verified 1068 textures and 13 shader replacements"]
Loading

Details:

  • Reuse never touches the network and never "repairs" a modified pack. verify_files() (visual_assets.py:25-29) requires each manifest file to be a regular, non-symlink file with the exact byte size and SHA-256; any difference raises "Visual asset is missing or changed: <name>" and the existing file is left in place.
  • Complete bundle. When Release/HaloVision.app exists (an extracted Complete release), the three packs are verified inside the bundle and clone-copied, so no second ~0.9 GiB download is needed.
  • Download. The URL must start with https://github.com/mitchaiet/master-chef/releases/download/ ("Unexpected visual asset download origin" otherwise). The request uses User-Agent: MasterChef-Setup and a 120 s timeout, is written to download.zip opened with xb (exclusive create), and is aborted the moment it exceeds the pinned byte count. The approximate size in GiB is printed first.
  • Offline. python3 tools/visual_assets.py --archive /path/to/MasterChef-v1.0.3-visual-assets.zip uses a pre-downloaded archive (visual_assets.py:115-122); errors exit with status 2 and "Visual asset setup failed: ...".
  • Disk. Free space in .setup must cover the three files, the archive, and a 1 GiB reserve.
  • Atomicity. All work happens in .setup/visual-assets-* temporary directories; only a fully verified unpacked/ directory is renamed to .setup/VisualMods. A failure leaves no VisualMods directory (test_bad_archive_hash_leaves_no_install).
  • Concurrency. The blocking flock on .setup/visual-assets.lock serialises concurrent setup/build processes.
  • VisualModsManifest.json (a copy of the public manifest) is written next to the packs; it is also bundled into the app.

Safe extraction (extract_verified)

visual_assets.py:31-46 treats the ZIP as untrusted even after the outer hash matches:

Check Defends against
Manifest rows must be exactly NAMES with no duplicates Unexpected payloads
namelist() must have exactly as many entries as rows and the same set of names Extra entries, duplicates, path traversal names such as ../escape
Entry mode bits (external_attr >> 16) must not be a symlink Symlink entries pointing outside the destination
info.file_size must equal the manifest size Size mismatch before writing
Each file opened with xb and chmod 0600 Overwriting existing files
verify_files() on the destination afterwards Wrong inner content even when the outer archive hash matches

Bundling into the app (bundle_visual_assets)

visual_assets.py:103-113 re-verifies the source directory, then for each of the three packs plus VisualModsManifest.json: if the destination exists in the app, it must not be a symlink and must have the same SHA-256 ("Existing app has different visual assets; rebuild with --clean"); otherwise it is clone-copied. The app copy is verified once more. In the Xcode route the same four files are listed as resources in project.yml:29-36. CHANGELOG 1.0.3 records that both build routes previously omitted the packs.

Texture pack format (HVTEX001)

All fields little-endian. Written by import_texmod_pack.py and merge_hd_texture_pack.py using struct.Struct('<8sIIQQ') for the header and struct.Struct('<IIIIQQ') for entries; validated at runtime by htm_open() (texture_mod_pack.h:25-43).

Header (32 bytes):

Offset Size Field Required value
0 8 magic HVTEX001
8 4 count 1..65536
12 4 entry stride 32
16 8 table offset 32
24 8 file size equals the actual file size

Entry (32 bytes each, immediately after the header):

Offset Size Field Rule
0 4 hash TexMod CRC32 of mip level 0; strictly increasing across entries (enables binary search)
4 4 width 1..8192
8 4 height 1..8192
12 4 flags must be 1 (BGRA8, single level)
16 8 offset at or after the end of the previous data (no overlap), within the file
24 8 bytes exactly width * height * 4

Pixel data is raw BGRA8, top-down, one mip level. The runtime caps the file at 4 GiB (htm_file_size_valid), since "Full-game artwork exceeds 2 GiB".

Runtime texture replacement

texture_mod_runtime.inc:

  1. Initialisation (once, pthread_once, on the first texture upload): reads HALO_TEXTURE_MODS and HALO_TEXTURE_PACK. If mods are disabled ("0") or no path is set, originals are used. Otherwise the file is mmaped read-only (MAP_PRIVATE), validated with htm_open, and the log reports [texture-mod] loaded N replacements, M mapped bytes. "Pages are demand loaded; replacement uploads use the existing bounded Metal LRU cache." Invalid or oversized packs log a message and fall back to originals.
  2. Lookup (uploaded_texture_mod(), lines 35-54), called from uploaded_texture() in d3d9_render.inc:329 before the normal decode path: render-target textures (usage & 1) are skipped. The CRC of the guest's level-0 bytes (in the texture's native format and size) is computed once per content generation, using the ARMv8 CRC32 instructions when available (HTM_NATIVE_CRC32) or a table-driven scalar fallback. The stored value is the uncomplemented IEEE CRC32, matching TexMod. A binary search finds the entry.
  3. Upload: the replacement is uploaded into the renderer's texture cache under key 0x48565458_00000000 | hash ("HVTX"), reused on later hits, and the first hit of each entry is logged. "Allocation failure falls back to the original, never black."

TexMod import (import_texmod_pack.py)

python3 tools/import_texmod_pack.py <source.tpf | source.zip> <output.hvt>

convert() (import_texmod_pack.py:67-121). Requires Pillow and numpy; "No Windows program is executed", and only format facts from the cited references are used.

  1. Container. A .tpf is used directly; any other file is opened as a ZIP that must contain exactly one .tpf.
  2. De-obfuscation (open_tpf()): every little-endian 32-bit word is XORed with 0x3FA43FA4, trailing bytes with 0xA4; the result is a ZIP read with TexMod's fixed archive password constant (TPF_PASSWORD). Reading entries through zipfile also checks each entry's ZIP CRC.
  3. Definitions. texmod.def lines are crc|name (hex or decimal CRC via int(x, 0)). CRCs must fit 32 bits. Duplicate CRCs keep the first definition and are reported in the receipt as duplicateDefinitions (hash, selected, ignored) "rather than hiding it".
  4. Decoding (pixels()):
    • DDS DXT2/DXT4 headers are rewritten to DXT3/DXT5 (same block layout) and the decoded RGB is un-premultiplied: min(255, (c*255 + a/2) / max(a,1)), zeroed where alpha is 0.
    • 32-bit BI_RGB BMPs are decoded manually (Pillow would drop the fourth byte): bottom-up and top-down orientation honoured, truncation and dimensions over 8192 rejected. If every alpha byte is 0 the image is made opaque ("BI_RGB files with an entirely zero spare byte have no alpha (e.g. this pack's crewman uniform)").
    • Everything else goes through Pillow convert('RGBA'). Dimensions must be 1..8192.
  5. Output. Entries are written sorted by CRC into <output>.partial (header and table reserved first, then patched), then renamed. The output must not already exist (FileExistsError). Progress prints every 25 textures.
  6. Receipt. <output stem>.json with sourceSHA256, tpfSHA256, packSHA256, count, bytes, duplicateDefinitions, and per-texture hash, source, width, height, offset, bytes, sha256 (of the BGRA pixels). A summary without the texture list is printed.

HD merge (merge_hd_texture_pack.py)

python3 tools/merge_hd_texture_pack.py <base.hvt> <review-manifest.json> <output.hvt>

merge() (merge_hd_texture_pack.py:39-111) adds reviewed HD images to an existing pack. The docstring: "Pixels are only decoded to native BGRA8. Masked images require the explicit source-nearest alpha policy and exact verification. Replacing an existing entry requires its reviewed pixel digest. Original packs/images stay unchanged."

Review manifest: a JSON array of rows. Only rows with "status": "accepted" are used (others such as held are ignored; a manifest with no accepted rows fails with "No accepted images").

Row field Required Meaning
status yes accepted to include
hash yes 8-hex texture CRC; must be unique among accepted rows
original, generated yes Paths to the original-resolution PNG and the HD PNG
originalPNG_SHA256, generatedPNG_SHA256 yes Must match the files: "Image changed after review" otherwise
alphaPolicy when the original has any non-opaque pixel Must be source-nearest; the HD alpha channel must equal the original alpha resized with nearest-neighbour exactly
replacesPixelSHA256 when hash already exists in the base pack SHA-256 of the existing entry's BGRA bytes; must match ("Existing replacement changed after review")

Image rules: HD width/height between the original's and 2048 inclusive; aspect ratio exactly preserved (w * src.h == h * src.w); fully opaque originals require a fully opaque HD image ("Unexpected transparency").

Algorithm: validate the base with entries() (same rules as htm_open, plus the base is memory-mapped read-only); require the set of replaced hashes to equal the set of rows carrying replacesPixelSHA256; write the union of entries in ascending CRC order into <output>.partial (opened x+b); re-open and re-validate the new pack, checking byte-for-byte that every untouched old entry and every added entry is identical; confirm the base pack's SHA-256 did not change during the run; rename into place. The receipt <output stem>.json records baseSHA256, manifestSHA256, packSHA256, originalEntries, addedEntries, replacedEntries, count, bytes, preservedAllExistingPixels (false when any replacement happened), preservedAllUnselectedPixels: true, resampled: false, alphaReconstructed: false, and the per-texture notes with pixelSHA256.

Shader pack format (HVSHD001)

Written by make_pack() (import_censhine_pack.py:106-122); validated by hsm_open() (shader_mod_pack.h:26-44).

Offset Size Field Rule
0 8 magic HVSHD001
8 4 count 1..128 (HSM_MAX_ENTRIES)
12 4 reserved 0
16 ... entries u32 original_size, u32 replacement_size, original bytes, replacement bytes, repeated count times

Whole file 16 bytes to 4 MiB (HSM_MAX_BYTES); the entries must consume the file exactly. Each shader must pass hsm_shader(): 8..262,144 bytes, 4-byte aligned, version token 0xFFFF0200 (ps_2_0), a bounded token walk (opcodes up to 96, comment tokens skipped by their length, instruction lengths inside the buffer) ending with the END token 0x0000FFFF as the last word. Duplicate originals are rejected. The header comment adds that "The release gate also translates and compiles every replacement with the production MojoShader/Metal path."

Runtime shader replacement

shader_mod_runtime.inc loads the pack once (pthread_once) from HALO_SHADER_PACK unless HALO_SHADER_MODS=0, reading it fully into memory; an invalid pack logs [shader-mod] invalid pack; using originals. In the Direct3D9 bridge, IDirect3DDevice9::CreatePixelShader (vtable index 106) measures the guest token stream through its END token and calls shader_mod_lookup() with those bytes (d3d9.c:785-788). On an exact byte match the replacement token stream is used instead, before the shader is hashed and translated by MojoShader (see Shader Translation). Lookups happen "only when a shader is created, never for every draw or panorama bearing". Vertex shaders are never replaced.

CEnshine adaptation (import_censhine_pack.py)

python3 tools/import_censhine_pack.py /path/to/decoded/fx.bin /path/to/decoded/collection /path/to/output/ShaderMods.hvs

Inputs are a locally supplied, Composer-decoded retail fx.bin and the decoded CEnshine 1.0.0 collection. The tool does not download inputs or include Composer (BUILDING.md). Ordinary setup never runs it; it exists so the published pack can be reproduced and reviewed. The same adapter, plus upstream source and the GPL-3.0 license, is inside CEnshineSources.zip (upstream commit ce22b5456ed29ae0c53cf19711085b4b9ee93d6a).

  1. Retail effects (retail_shaders(), lines 24-47): fx.bin is a sequence of length-prefixed blocks, each a D3DX effect beginning 0xFFFFFFFF. The parser walks the parameter table (rejecting tables over 4096 entries), records the names of pixel-shader object parameters (type 15, class 4, no elements), skips techniques/passes/annotations/states, and collects the bytecode of each named shader object from the string/object section. A bounds-checked Reader rejects any truncation.
  2. Collection (collection_shaders(), lines 49-64): verifies the inner footer (32 ASCII hex MD5 of the body plus NUL), requires exactly 126 groups ("Unexpected collection revision"), at most 256 functions per group, no duplicate names, and no trailing data.
  3. Selection (make_pack()): only groups environment_lightmap_normal and model_environment; names are mapped PS_EnvironmentLightmapNormal to retail PS_LightmapNormal; each must match exactly one retail shader whose bytes start 00 02 ff ff and end ff ff 00 00. Programs whose retail bytecode is identical must adapt identically (else "Ambiguous identical original bytecode") and are emitted once. Exactly 13 rows must result ("Expected the reviewed 13-program subset").
  4. ABI adaptation (adapt(), lines 66-104): the replacement must be stripped ps_2_0 (no comment tokens), with valid instruction lengths and an END token, and no relative addressing. Constant-register operands are remapped:
Program kind CE register Retail meaning Remapped to
model (model_environment) c0 tint c16 (defined as 1,1,1,1: "retail's white tint")
model c1, c2, c3 fog correction vectors c0, c1, c2 (retail exposes fog in c0..c2)
model c4, c5 illumination / alpha reference c17 (defined as 0,0,0,0: "no extra illumination and renderer-side alpha test")
model, non-ComplexFog c6 c17
model, non-ComplexFog v0.wwww (CE's in-shader fog) c17.wwww ("retain its external fog")
lightmap (environment_lightmap_normal) c6 c17

DEF literal instructions are not remapped, but a DEF targeting c16 or c17 is rejected ("Reserved constant conflict"). The tool prepends def c17, 0,0,0,0 (and for model programs def c16, 1,1,1,1) right after the version token. 5. Output: HVSHD001 pack plus <output stem>.json with format, count, pack sha256, retailSHA256, collectionSHA256, and per-entry name, group, original and replacement SHA-256 and byte counts.

Tamper and safety checks summary

Stage Check
Manifest Exactly three known file names
Install location .setup and .setup/VisualMods must not be symlinks
Download Fixed origin prefix; streamed size cap; exclusive file creation
Archive Exact byte size and SHA-256
Extraction Exact entry set (no extras, no duplicates, no traversal), no symlink entries, per-entry size
Installed files Exact size and SHA-256 on every reuse; never overwritten
App bundle Existing different packs refused; copied packs re-verified
Runtime (HVTEX001) Header, sorted unique CRCs, dimension and size bounds, non-overlapping ranges; failure falls back to originals
Runtime (HVSHD001) Size limits, count, reserved word, per-shader token walk, no duplicate originals, exact length; failure falls back to originals

Unknown or modified packs are never silently used (CHANGELOG 1.0.3).

Environment variables

Variable Default Effect Read at
HALO_TEXTURE_PACK unset on desktop; on visionOS set to the bundled TextureMods.hvt if present (without overwriting an existing value) Path of the HVTEX001 pack texture_mod_runtime.inc:17, set at EngineVisionRuntime.m:652
HALO_TEXTURE_MODS unset (enabled) 0 disables texture replacement texture_mod_runtime.inc:17-18
HALO_SHADER_PACK unset on desktop; on visionOS the bundled ShaderMods.hvs if present (no overwrite) Path of the HVSHD001 pack shader_mod_runtime.inc:7, set at EngineVisionRuntime.m:654
HALO_SHADER_MODS unset (enabled) 0 disables shader replacement shader_mod_runtime.inc:7-8

Failure modes

Message Cause Fix
Visual asset is missing or changed: <name> A file in .setup/VisualMods (or the Complete bundle) differs from the manifest Restore the original file or move the modified directory aside; it is never overwritten automatically.
Visual archive checksum mismatch Corrupt or different download Re-download or supply the correct --archive.
Visual archive exceeds its expected size Download larger than pinned Check the URL/network proxy.
Unexpected or duplicate archive entries / Invalid archive entry: ... Archive contents not exactly as published Use the official archive.
Insufficient free space for the verified visual assets and 1 GiB reserve Disk Free space.
Visual asset directories must not be symbolic links .setup or VisualMods is a symlink Use real directories.
Existing app has different visual assets; rebuild with --clean Stale direct product build_engine_vision.py --direct --clean.
Runtime log [texture-mod] invalid pack; using originals / [shader-mod] invalid pack; using originals Pack failed runtime validation Rebuild the pack.

Testing

Test Run by run_source_checks.py Asserts
tools/test_visual_assets.py (9 tests) yes Exact pack import, resume without network, and bundling; outer checksum rejection leaves no install; a modified installed pack is preserved and rejected; ../escape entry rejected; symlink entries rejected; wrong inner content rejected even when the outer hash matches; symlinked .setup rejected; Complete bundle used without network; game-content selection keeps configuration/movies and excludes saves/registry.
tools/test_censhine_import.py (4 tests) yes Complex-fog register ABI and defined white tint; simple-fog v0.wwww becomes c17.wwww; lightmap keeps lighting registers and remaps c6; malformed inputs rejected (empty, unaligned, wrong shader version, a truncated DEF instruction, tokens after END, bad collection checksum, bad retail block, truncated reader).
tools/test_texmod_import.py (3 tests) no (needs Pillow) BMP alpha and bottom-up orientation, truncation, zero spare byte made opaque; PNG channel order to BGRA; DXT2 un-premultiplication.
tools/test_hd_texture_merge.py (13 tests) no (needs Pillow) Old and new pixels exact; duplicate replacement without digest rejected; source/generated alpha rules; reviewed source-nearest alpha exact and mismatch rejected; digest-bound replacement and wrong/absent digests; image changed after review; aspect change; truncated and overlapping ranges; held candidate not promoted.
native/EngineHost/tests/test_texture_mod_pack.c yes (portable) Pack parsing and native CRC32 against the scalar algorithm over differential cases (with a CPU microbenchmark printout).
native/EngineHost/tests/test_shader_mod_pack.c yes (portable) hsm_open bounds, exact-match lookup, truncation at every length, size and token validation.
native/EngineHost/tests/test_d3d9_texture_mod.c yes (portable, with texture_decode.c) The real D3D upload boundary with a temporary HALO_TEXTURE_PACK, generation changes and recorded GPU doubles.

Run the two Pillow-based tests manually from the activated .venv: python3 tools/test_texmod_import.py and python3 tools/test_hd_texture_merge.py.

Related pages

Clone this wiki locally