Rust + KID pivot — M2 Part 2: mora-esp#43
Merged
Conversation
Delivers mora-esp end-to-end. Covers: byte-reader primitives, TES4 header parsing, record/group/subrecord iteration, LZ4 Frame decompression (Task 8 Step 0 verifies against xEdit/CommonLibSSE-NG), plugins.txt parser, load-order resolver (full + light slot pool), master-index remapping, EspPlugin (mmap), EspWorld (cross-plugin indexed view), EDID/KWDA subrecord parsers, WEAP/ARMO record types. 22 tasks across 13 phases. mora-core::Distributor becomes generic over World to avoid a mora-core -> mora-esp cycle. Synthetic ESP fixture builders in tests/fixtures.rs drive end-to-end integration tests covering the full parser stack.
ESP binary format (SSE/AE), plugins.txt spec, load-order rules (including ESL light-slot pool + sub-indexing), master-index remapping algorithm, LZ4 Frame compression notes, synthetic-fixture test strategy. Cited by every task in Plan 5.
lz4_flex = 0.11 added to workspace deps (pure-Rust LZ4 for SSE/AE compressed records). mora-esp src/: module tree for reader, signature, tes4, record, group, subrecord, compression, plugins_txt, load_order, plugin, world, records/, subrecords/. All stubs minimal.
4-byte ASCII signatures (TES4/GRUP/WEAP/ARMO/HEDR/MAST/EDID/KWDA/XXXX). Display / as_str / as_bytes accessors. 3 unit tests.
le_u8/u16/u32/f32 + read_signature + read_bytes + read_cstr against a &[u8] cursor. ReadError::Truncated for short input. 6 unit tests covering happy path, truncation, and NUL handling.
24-byte header parse → Record { signature, flags, form_id,
record_version, body: &[u8] }. is_compressed() / is_deleted() flag
accessors. 5 unit tests covering minimal header, body, compressed
flag, deleted flag, truncation.
24-byte group header → Group { label, group_type, contents: &[u8] }.
Top-level test (group_type == 0) + label_signature accessor. 2 unit
tests: empty WEAP group, group with payload.
Implements subrecord.rs — SubrecordIter that iterates over a record body, handling the XXXX subrecord whose 4-byte u32 payload overrides the next subrecord's u16 size field. Three tests: happy path, 100k XXXX override, and empty body.
decompress(body) reads a 4-byte LE u32 decompressed_size prefix then inflates the LZ4 Frame payload via lz4_flex::frame::FrameDecoder. Errors: Read (truncation), Lz4 (frame error), SizeMismatch (prefix disagrees with actual inflated length). 3 round-trip tests: small payload, 10k-element payload, size-mismatch detection. Step 0 findings: plan's assumption confirmed correct. xEdit lz4io.pas selectDecoder dispatches on LZ4S_MAGICNUMBER ($184D2204) = LZ4 Frame. wbImplementation.pas TwbMainRecord.DecompressIfNeeded reads the first 4 bytes as PCardinal (decompressed size), then passes the remainder starting at dcDataBasePtr+SizeOf(Cardinal) to lz4DecompressToUserBuf. Format is Frame (not Block) with 4-byte LE u32 prefix — exactly as the plan assumed. No adjustments to code or tests were needed.
parse_tes4(bytes) -> Tes4Header { flags, version, num_records,
next_object_id, masters, author, description }. is_esm() / is_esl()
flag accessors. CNAM/SNAM signatures added. 2 smoke tests: TES4
with two masters, TES4 with no masters.
parse(content) -> Vec<PluginEntry { name, active }>, preserving file
order. active_plugins filter. Handles CRLF/LF, comments, blank lines,
leading/trailing whitespace. 4 unit tests.
LoadSlot::Full(u8) | Light(u12) with compose_form_id helper. LoadOrder indexes plugins by lowercased name. build() assigns slots in (implicit first, user-active second) order; ESL-flagged plugins go to the 0xFE light pool without consuming a full slot. 5 unit tests.
Adds master-index remapping helper that converts plugin-local FormIDs into fully-resolved FormIDs using the plugin's master list and live load order. Includes 5 tests covering master references, self-reference, ESL remap, and out-of-range validation.
EspPlugin::open(path) mmaps the file and parses the TES4 header. Exposes filename, masters, is_esm / is_esl, and body() — the slice after the TES4 record. Arc<Mmap> for multi-threaded access. 1 smoke test: open a synthesized ESM file, confirm flags + masters.
EspWorld::open(data_dir, plugins_txt) -> World. records(signature) iterates all records of that type across plugins in load order, with resolved FormIDs. Nested groups are skipped. Integration tests land in tests/esp_format.rs (Task 21).
Plan 5's EspWorld placeholder becomes real in mora-esp, but adding a mora-core -> mora-esp dep would cycle. Generic World param lets frontends bind to mora_esp::EspWorld without the cycle. Old EspWorld type retained with #[deprecated]; removed from crate re-exports.
parse(data) -> Result<String, ReadError> reads a NUL-terminated ASCII string. 3 unit tests: happy, empty, missing NUL errors.
parse(data) -> Vec<u32> reads LE u32 FormIDs. Rejects payloads not a multiple of 4 bytes. 3 unit tests: 3-id happy, empty, unaligned error. IDs are local — callers remap via the plugin's master list.
parse(record) -> WeaponRecord { editor_id, keywords }. Handles
compressed records (LZ4 inflation). Keywords are local FormIDs —
caller remaps via plugin's master list. Full-stack tests in
tests/esp_format.rs.
parse(record) -> ArmorRecord { editor_id, keywords }. Mirrors
weapon.rs; the two share no code to let per-type struct evolve
independently.
PluginBuilder / GroupBuilder / RecordBuilder / SubrecordBuilder + edid_payload / kwda_payload helpers. Composes ESP byte buffers inline — reused by esp_format.rs integration tests.
4 tests using synthetic plugins written to tmp: open single plugin, EspWorld iterates across A.esm + B.esp, WEAP parse extracts EDID + KWDA, ARMO smoke. Exercises full stack: mmap -> TES4 parse -> plugins.txt parse -> load order -> group scanning -> record iteration -> subrecord parsing.
Applied cargo clippy --fix (for-loops over while-let, .is_multiple_of, repeat_n, lifetime elisions) plus cargo fmt. Also added crate-level #[allow(dead_code, clippy::should_implement_trait)] to tests/fixtures.rs since builder methods are test-only and some (esl, flag) aren't called yet by every integration test.
KWDA FormIDs in records are plugin-local (high byte = index into the source plugin's master list), not yet resolved runtime FormIDs. Plan 5 originally returned raw Vec<u32>; this commit resolves them at parse time using the enclosing EspWorld's load order, so callers receive Vec<FormId> (the nominal type from mora-core). - WeaponRecord::keywords: Vec<u32> -> Vec<FormId> - ArmorRecord::keywords: Vec<u32> -> Vec<FormId> - weapon::parse(record, plugin_index, world) -> adds context - armor::parse(record, plugin_index, world) -> adds context - WorldRecord::resolved_form_id: u32 -> FormId - New EspWorld::resolve_in_plugin(plugin_index, raw_local) helper - New EspWorld::weapons() / armors() typed iterators yielding Result<(FormId, TypedRecord), _> - Unresolvable keywords (referenced plugin not in load order) are silently dropped to match KID behavior Integration tests extended to cover silent-drop semantics and the typed weapons() iterator. 126 tests pass workspace-wide (124 + 2).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Delivers
mora-espend-to-end perdocs/superpowers/plans/2026-04-21-rust-kid-pivot-plan-5-mora-esp.md:lz4_flex::frame::FrameDecoder.*EspPlugin— mmap a plugin file, parse TES4 headerEspWorld— iterate records of a given signature across all active plugins in load order, with resolved FormIDsWeaponRecord,ArmorRecord(editor_id + keywords; handles LZ4 compression)mora-corebreaking change —Distributortrait is now generic overWorldto avoid a mora-core ↔ mora-esp cycledocs/src/mora-esp-reference.md— binary-format spec + plugins.txt + load-orderTest plan
cargo test --workspace— 124 tests pass (72 from M2 Part 1 + 52 new)cargo clippy --all-targets -- -D warningscleancargo fmt --checkcleancargo xwin check --target x86_64-pc-windows-msvc --workspacecleanNotable findings
lz4/lz4io.pas+wbImplementation.pas: LZ4 Frame (magic0x184D2204) with 4-byte LE u32 decompressed-size prefix, read vialz4_flex::frame::FrameDecoder. Thank you for the correction on Frame vs Block.Scope discipline
Next up
Plan 6: mora-kid MVP — KID INI parser, filter AST, distributor impl for Weapon + Armor, first end-to-end
mora compileproducingmora_patches.bin.