Repository navigation
PSX PSP stage overlay import runbook
Use this document when asked to "import overlay X". Replace <overlay> with
the lowercase overlay argument accepted by the extractor, <OVL> with the
disc basename, and fill in the actual version/path values discovered in the
workspace. Keep an overlay-specific overlay_import_log.txt alongside this
runbook with every command, failure, fix, hash, and deduplication decision.
Work from the repository root. Read the user's import instructions and inspect the current tree before changing anything:
pwd
git status --short
find -L disks -type f | rg -i '(^|/)<overlay>(\\.bin)?$|/<overlay>/'
The repository may contain unrelated modified and untracked files. Record them in the log. Never clean, overwrite, stage, unstage, or commit them.
Resolve:
- PSX version and binary path.
- PSP version and binary path.
- Extractor overlay argument and generated basename.
- Required platform order. Default to PSX first, then PSP.
The current extractor is:
tools/sotn_utils/extract_overlay/extract_overlay.py
Prepare the repository virtual environment and ensure extractor subprocesses also use it:
.venv/bin/pip install -r tools/requirements-python.txt
.venv/bin/pip install mako
export PATH=/sotn/.venv/bin:$PATH
Rebuild the checked-out Go asset helper so the binary cannot lag its source:
go build -o bin/sotn-assets ./tools/sotn-assets
The repository asset wrapper is:
./sotn.sh
It invokes go run ./tools/sotn-assets "$@". Asset extraction is mandatory;
do not treat an overlay link alone as full validation.
Run:
PATH=/sotn/.venv/bin:$PATH \
.venv/bin/python \
tools/sotn_utils/extract_overlay/extract_overlay.py \
<overlay> -v <psx-version>
Use --clean only when a failed attempt produced partial artifacts for the
exact overlay currently being imported:
PATH=/sotn/.venv/bin:$PATH \
.venv/bin/python \
tools/sotn_utils/extract_overlay/extract_overlay.py \
<overlay> -v <psx-version> --clean
--clean is destructive for generated artifacts. Confirm the target first.
Resolve generated symbol/link failures by comparing analogous overlays and the extractor implementation. Record every manual correction in the log.
Build the overlay target and compare it with the disc binary:
ninja -f build/<psx-version>/build.ninja <generated-overlay-target>
sha1sum <generated-overlay-binary> <psx-disc-binary>
Do not begin PSP until the PSX SHA-1 values match exactly.
Run PSP extraction only after PSX is stable:
PATH=/sotn/.venv/bin:$PATH \
.venv/bin/python \
tools/sotn_utils/extract_overlay/extract_overlay.py \
<overlay> -v <psp-version>
The first PSP extraction may build many reference ELFs and spend several minutes comparing functions. Record ambiguous or unhandled matches in the log; distinguish warnings from actual build/hash failures.
Validate:
sha1sum <generated-psp-binary> <psp-disc-binary>
Use tools/dups, objdiff, or asm-differ. PR #3380 is the model for moving
overlay-specific source toward shared implementations:
https://github.com/Xeeynamo/sotn-decomp/pull/3380
Required order:
- Compare PSX overlay X only against other PSX overlays.
- Deduplicate PSX functions/data and prove the PSX binary still matches.
- Compare PSP overlay X only against other PSP overlays.
- Deduplicate PSP functions/data and prove the PSP binary still matches.
- Only then combine the independently validated PSX and PSP sources.
Do not use direct PSX-versus-PSP dups scores as primary evidence. Different
compilers and ABIs make those assembly similarity scores misleading.
Example same-platform comparison:
tools/dups/target/debug/dups \
--dir asm/<version>/st/<overlay>/nonmatchings \
--dir asm/<version>/st/<reference>/nonmatchings \
--threshold .94
Build tools/dups first if needed:
cargo build --manifest-path tools/dups/Cargo.toml
After importing a new PSX overlay, add it to the SrcAsmPair registry in
tools/dups/src/main.rs. Include its matching assembly directory, source
directory, overlay name, get_all_include_asm directory, and path matcher.
This is required even if pairwise checks have already been run; otherwise the
new overlay is absent from the standard codebase-wide clustering report.
Rebuild and run the global report from the tool directory, because its current
registry paths are relative to tools/dups:
cd tools/dups
cargo build --release --manifest-path Cargo.toml
target/release/dups \
--threshold .94 \
--output-file /tmp/<overlay>-dups-report.txt
Verify the report actually contains paths for the new overlay. The current
global MIPS registry targets asm/us; keep PSP comparisons same-platform with
explicit two-directory invocations unless/until the tool gains a PSP registry.
Do not substitute PSX-versus-PSP clustering.
For a candidate:
- Confirm function identity with same-platform assembly/object diffs.
- Find an existing shared header or implementation.
- Split the candidate's owned
.data,.rodata, or.bssin its splat config before replacing raw assembly with shared C. - Check section sizes, alignments, symbol ownership, and function addresses.
- Rebuild immediately and compare SHA-1 after each small batch.
Small final PSX/PSP differences may use focused #ifdef VERSION_PSP
conditionals. If code or data differs substantially, add a concise source
comment explaining why it remains overlay-local and move on. Do not force a
merge merely because function names match.
When code matches but data appears different, diff the arrays and tables directly. Keep stage-specific data in the overlay file and include the shared function body beneath it where the existing source pattern supports that.
For PSP entity families, inspect how the shared header composes PSP-only helpers before assuming adjacent splat objects are independent. A header may include another PSP header in the middle of its normal function order (for example, Maria particle helpers before the remaining particle entities). When the original overlay has that same interleaving, replace all contiguous text entries with one shared source entry so the compiler emits the intended order.
If only the middle function in a family differs, preserve original function
order rather than abandoning the whole family. Add narrow selection guards to
the shared header, then let the PSP wrapper emit matching leading shared
functions, directly INCLUDE_ASM the differing function, and include the
header again in a mode that emits only matching trailing functions.
PSP INCLUDE_ASM expansion is performed on direct .c input by
sotn_str; an INCLUDE_ASM hidden inside an included header can preprocess
to nothing. Keep the actual PSP assembly include directly in the wrapper.
Verify every interleaved function start address and size in the linker map.
When diagnosing a small entity-family mismatch, compare only the differing function and distinguish semantic constants from compiler noise. Check byte size, instruction immediates, branches, and arithmetic signs. A corrected palette or coordinate can still leave widespread register-allocation or expression-scheduling differences. If behavior constants match but the function remains materially different, comment the measured sizes and retain that function alone in assembly.
Treat tools/dups similarity 1.0 as structural evidence, not a checksum.
The tool can normalize addresses and immediate operands, so two functions
with identical instruction shape may still differ by stage constants. After
selecting a duplicate, perform a raw instruction or binary diff and inspect
every remaining word before accepting it. Express small verified constant
differences with the narrowest stage guard available.
Raw graphics, compressed streams, and other opaque data can coincidentally
contain a word near a known function address. If splat emits a suspicious
reference such as FunctionName + small_addend from otherwise arbitrary data,
inspect the surrounding words before adding the function to an undefined
symbol file. Mark known functions with // type:func in the input symbol map
so spimdisasm does not create addend references to them, regenerate the split,
and confirm the word remains numeric and the overlay still matches.
When C-owned data ends at an address that is not aligned for the following
raw assembler input section, inspect the original boundary bytes. If they are
alignment padding, begin the next raw splat segment at the aligned offset and
let the linker reproduce the padding; starting the raw object at the
unaligned offset can add padding and shift its contents. Also inspect the map
for raw sections discarded by --gc-sections. If an otherwise unreferenced
raw segment is part of the original binary, retain it by exporting a symbol
at its true aligned start in the overlay's symexport file. Rebuild and
checksum after both adjustments.
Shared PSP headers can own BSS that an imported assembly object previously accessed through an autogenerated address name. Split the exact BSS prefix to the shared C object, resume named raw BSS at the next boundary, and preserve the old address symbol for unconverted callers. Verify both the storage symbol and legacy address resolve to the same location; do not leave a duplicate raw BSS allocation.
Do not use an overlay binary checksum to validate BSS ownership. BSS is not stored in the output binary, so a build can hash exactly while allocating variables at incorrect addresses or shifting the raw BSS that follows them. Inspect the linker map after every BSS conversion. Account for the compiler's input-subsection alignment and verify each named variable, fill region, object end, and raw-BSS resume address against the original layout.
An extractor may incorrectly split a short function prefix from the rest of
an object even though a shared header emits the complete function sequence.
Before adding a helper-only source file or a new header-selection macro, test
whether one C object can own the complete original text range together with
its .data, .rodata, and .bss ranges. Data that initially appears
stage-specific may still be emitted byte-for-byte by existing stage guards
when the wrapper defines the same stage constants as its PSX counterpart.
Regenerate the splat and build files after changing the first text boundary;
stale build rules can compile both the old and new object split.
Some PSP loader/entity-ID variables are fixed symbols in an existing BSS
range rather than storage the new C object should allocate. If ordinary
definitions shift many unrelated references, inspect the original stores and
linker map. Declare fixed variables extern and name every exact address in
the overlay symbol config; do not let tentative common symbols consume a
neighboring raw BSS block.
When shared C emits a table used only by interleaved assembly, that assembly
may reference a legacy raw-address symbol rather than the C symbol.
--gc-sections then treats the C table as unreferenced. Give the table
external linkage where necessary and add EXTERN(table_name) to the overlay
symexport file. Confirm the table address and raw-data resume boundary after
linker input-section alignment.
Reassess fixed symbol-map and symexport entries after converting an object to shared C. Remove entries for storage now defined and referenced entirely by C; the compiled definition establishes its address through the splat section assignment. Keep a fixed symbol-map entry when a neighboring raw segment still owns the definition but shared C references it, such as a stage-specific table inside a larger raw data block. Prove removals with a clean splat regeneration, not an incremental link that may retain previously generated symbolic labels.
When several shared objects own adjacent PSP data, assign every subsection in
address order rather than letting source data collect at the first available
boundary. Use the object section table and linker map to calculate each
retained object's complete size. A successful chain should end exactly at the
next known object's start without increasing DATA_END or changing the
aligned rodata/BSS starts.
Do not force a shared header's complete data block into an overlay when the original code aliases or interleaves data owned by neighboring objects. In that case, keep the text trial separate from the data decision. The eventual shared conversion needs guarded external declarations and exact legacy aliases for the original layout; simply reordering splat entries can overlap other verified source objects and silently shift every later section.
For a PSX/PSP pair, the final combined wrapper can select platform headers or
data with narrowly scoped VERSION_US / VERSION_PSP conditionals. PSP
assembly may reference data that remains in a raw PSP data segment; in that
case name the exact address in the PSP symbol file and exclude the PSX-owned
definition under VERSION_PSP. Validate this with a PSP checksum before
deleting the PSP wrapper and redirecting its splat entry to the shared source.
If a proposed shared implementation compiles but changes the checksum, revert that candidate immediately, document the measured difference in the source and overlay log, and continue. A matching function name is insufficient evidence across platforms or stage variants.
Run both requested full builds:
PATH=/sotn/.venv/bin:$PATH VERSION=<psx-version> make build
PATH=/sotn/.venv/bin:$PATH VERSION=<psp-version> make build
Then run the explicit asset/build wrapper:
./sotn.sh build
Wait for the wrapper's actual exit code; it can continue post-build work after printing success. Recheck both overlay SHA-1 pairs afterward.
Inspect both staged and unstaged state:
git status --short
git diff --stat
git diff --cached --stat
Expected work normally includes:
- Version checksum manifests.
- PSX/PSP splat and symbol configs.
- Any required undefined-symbol entries.
- Shared and platform-specific overlay source.
-
overlay_import_log.txt. -
overlay_import_runbook.txtwhen the reusable process changed.
Stage only files belonging to overlay X. Exclude random, generated, pre-existing, or unrelated repository files. Review the staged diff, then commit the import and deduplication with a specific message.
Do not introduce or retain OVL_EXPORT in newly imported overlay source; it
is a legacy naming mechanism. Replace each symbol that was expressed through
OVL_EXPORT(name) with its canonical unprefixed name, and update the
overlay's PSX/PSP symbol files, PSP symexport entry, declarations, assembly
labels, and INCLUDE_ASM names together. This rule applies only to symbols
that used OVL_EXPORT; do not mechanically remove legitimate overlay
identifiers from unrelated names, paths, header guards, or comments.
Some canonical names collide in the PSP compiler's ordinary identifier
namespace. In particular, an object named Overlay conflicts with the
Overlay typedef, and a function named Load conflicts with the SDK
declaration. Use collision-safe descriptive names such as g_StageOverlay
and StageLoad while retaining their original fixed addresses.
If a shared header still serves legacy overlays, make its symbol reference
select the old OVL_EXPORT(name) form when that macro is defined and the
canonical name otherwise. This lets the imported overlay stop defining the
macro without forcing a repository-wide migration. Rebuild both full
platform targets and compare their overlay binaries to the disc inputs.
Update overlay_import_log.txt after every completed phase and before long or
risky work. It must contain enough detail for a new instance with zero chat
context to resume, including:
- Exact input/output paths and hashes.
- Commands run and their exit status.
- Extractor warnings.
- Every failure, root cause, and fix.
- Splat section offsets and symbol corrections.
- Deduplication candidates accepted or intentionally deferred.
- Current build status.
- Exact files intended for the final commit.