Skip to content

pipeline_max_design

Jan Boon edited this page Jul 6, 2026 · 36 revisions

title: Pipeline Max — Design and Coherency Contract description: Codified design of the headless .max parser/writer (nel/tools/3d/pipeline_max) — the storage model, the parse/build/disown lifecycle, roundtrip invariants, known defects, and the rules any extension work must follow published: true date: 2026-07-06T00:00:00.000Z tags: editor: markdown dateCreated: 2026-07-05T00:00:00.000Z

This document codifies the design of the headless 3ds Max file library at nel/tools/3d/pipeline_max (namespace PIPELINE::MAX, "RYZOM CORE PIPELINE", 2012), together with its driver tools pipeline_max_dump and pipeline_max_rewrite_assets. It exists so that the session extending this library into the full headless export pipeline (see the Tutorial Tech Tree, side-quest 5) preserves the two properties the whole design is built around, rather than re-deriving them halfway and getting them subtly wrong:

  1. Whole-library coherency — every class participates in one shared storage lifecycle with one ownership discipline; a new class that improvises its own variant breaks the guarantees for everything beneath it.
  2. Roundtrip coherency — a file that is read and written back must be byte-identical per stream, whether or not any of it was parsed into typed form, except where a normalization is explicitly documented and whitelisted. Unknown data is never guessed at, never dropped, never reordered.

Everything below was verified against the actual source (every .cpp and .h in the library was read for this document), not reconstructed from memory. File/line references are as of commit d8deff3f2-era develop.

1. Scope and dependencies

  • Reads and writes 3ds Max .max files without 3ds Max. The container is an OLE2 compound document, accessed through libgsf (gsf_infile_msole_* / gsf_outfile_msole_*). NeL's NLMISC::IStream is bridged onto gsf streams by CStorageStream (storage_stream.h/.cpp) — a thin seek/read/write adapter.
  • Target versions: Max 3 through Max 2010. typedefs.h defines the version constants (Version3 = 0x2004 … Version2010 = 0x2012). The corpus in ryzomcore_graphics spans this whole range; the format is essentially stable across them, with per-version variations inside individual classes (e.g. CSceneImpl gains a 12th reference after Max 3).
  • Compressed .max files exist (gsf_input_uncompress is noted in a comment in storage_stream.h) and are not handled. If the corpus contains any, they must be detected and either supported or explicitly skipped with a count — not silently failed.
  • derived_object.*, wsm_derived_object.*, storage_file.* are empty stubs (constructor/destructor only). Placeholders, no design in them.

2. Anatomy of a .max file

An OLE compound document with these streams, all of which must be preserved on rewrite (this is exactly the set pipeline_max_rewrite_assets reads and writes back):

Stream Handling today
Scene CScene — the real content; one version-numbered root chunk containing the scene class container
DllDirectory CDllDirectory — fully parsed (list of plugin DLL description/filename pairs)
ClassDirectory3 CClassDirectory3 — fully parsed (per-class: dll index, ClassId, SuperClassId, display name)
ClassData CClassData — container structure known (entries 0x2100, header 0x2110 with ClassId/SuperClassId); entry body 0x2120 depends on the header and is an open TODO, kept raw
Config CConfig — partially typed (0x20a0 entry table, 0x2180 ConfigScript with its recursive "meta container" value format); many leaf ids known only as unlabeled ints/strings
VideoPostQueue generic CStorageContainer (raw pass-through)
\05SummaryInformation, \05DocumentSummaryInformation raw byte blobs, copied verbatim
OLE class id (16 bytes) read via gsf_infile_msole_get_class_id, must be written back via gsf_outfile_msole_set_class_id

The roundtrip unit is the stream, not the file. libgsf will not reproduce Max's OLE sector layout, so whole-file byte comparison is meaningless; per-stream byte comparison is the invariant. The Scene stream's single top-level chunk id doubles as the file version (CScene::version() asserts exactly one chunk and returns its id).

3. Chunk layer — CStorageChunks

storage_chunks.h/.cpp. Max chunk format:

  • 6-byte header: uint16 id, uint32 size. size includes the header; bit 31 = "container" flag (chunk contains child chunks rather than leaf data).
  • 64-bit extension: size == 0 means a uint64 follows (14-byte header total), same bit-63 container flag. The reader accepts these and downgrades to 32-bit bookkeeping (throws EStream if ≥ 2 GiB). The writer round-trips the header width per chunk (2026-07-05, see §11 "Fixed"): each IStorageObject records whether its own chunk was read with a 64-bit header (tri-state — chunks freshly authored by typed build() have no original width and fall back to their containing container's aggregate m_Was64Bit), and enterChunk(id, container, as64Bit) takes the width explicitly on write. Streams genuinely mix 32- and 64-bit headers in the corpus (~15% of files), so any coarser granularity (per-stream, per-container) corrupts on rewrite — that mistake was made once and is documented in §11.
  • Write path: enterChunk(id, container) writes the id and a 0xFFFFFFFF size placeholder; leaveChunk() seeks back and patches the real size with the container bit. Sizes are therefore never computed from getSize() — see the defect list (§11) before ever relying on getSize().
  • Read path: leaveChunk() returns the number of bytes skipped (unconsumed) in the chunk. CStorageContainer::serial throws EStorage if this is non-zero (storage_object.cpp:268). This is the load-bearing read-side guarantee: every byte of every chunk must be consumed by whatever object type was instantiated for it. A typed reader that under-reads cannot silently pass; it either matches the format exactly or the file fails loudly.

4. Storage object layer

storage_object.h/.cpp, storage_value.h/.cpp, storage_array.h.

  • IStorageObject (abstract, extends NLMISC::IStreamable): serial(IStream&), toString(ostream&, pad) (a readable dump — diagnostics only, never a test criterion), setSize(sint32) (called before reading a leaf so it knows how much to consume), getSize(sint32&), isContainer().
  • CStorageContainer: an ordered std::list of (uint16 id, IStorageObject*) — order is significant and duplicate ids are normal (that's why it's a list, not a map). Reading: for each child chunk, createChunkById(id, isContainer) manufactures the object, setSize primes it, then it serials itself. Writing: iterate the list in order, wrap each in a chunk.
  • createChunkById is the single extension point for typing a chunk: subclasses switch on the id and return a typed object; the default (and mandatory fallback for every unrecognized id) is CStorageContainer for containers and CStorageRaw for leaves. CStorageRaw is an exact byte vector. This is how unknown data survives byte-perfectly: not by special-casing, but because raw is the default and typing is the exception.
  • Typed leaves: CStorageValue<T> (single value; std::string/ucstring specializations sized by the chunk), CStorageArray<T> (chunk size / sizeof(T) elements), CStorageArraySizePre<T> (uint32 count prefix), CStorageArrayDynSize<T> (count prefix, variable-size elements — used for poly faces).
  • A typed leaf must serialize to exactly the same bytes it consumed. CStorageValue<ucstring> reads/writes raw UTF-16 (size/2 chars); CConfigScriptMetaString shows the pattern for a length-prefixed, null-terminated string. When a format has optional fields, the bitfield discipline in CGeomPolyFaceInfo::serial (geom_buffers.cpp:155) is the model: parse each known bit, clear it, and nlerror on any residue — unknown variants abort rather than desync.

5. The four-phase lifecycle — the roundtrip mechanism

This is the heart of the library and the thing an implementing session must not gloss. Declared on CStorageContainer (storage_object.h:116-124), with the class-level protocol implemented in CSceneClass (scene_class.cpp):

serial(in) → parse(version[, filter]) → [use / modify typed data]
          → clean()                     (optional: drop raw source, forces full rebuild)
          → build(version[, filter])    (reconstruct m_Chunks from typed data)
          → disown()                    (return ownership to m_Chunks, drop typed refs)
          → serial(out)

Ownership is tracked by one flag, m_ChunksOwnsPointers:

  • After serial(in): m_Chunks owns everything; the object is pure storage. true.
  • parse (CSceneClass): parent's parse runs first; then all chunks are moved onto m_OrphanedChunks and the flag flips to false. Subclasses then claim their chunks with getChunk(id), which removes them from the orphan list — in order: taking a chunk that is not at the head of the orphan list logs a warning ("chunks out of order"), because it means the class's read order no longer mirrors the file's chunk order. Chunks the class keeps unmodified are pushed to m_ArchivedChunks (deleted on clean, forgotten on disown). Chunks the class never claims stay orphaned and are re-emitted verbatim by build — that's per-chunk forward compatibility inside otherwise-typed classes (see CEditableMesh::parse, which drains a dozen known-unknown ids in their observed sequence purely to keep them in position).
  • Failure discipline: if a class cannot make sense of its data, it calls disown() and aborts its own parse — the object degrades to raw pass-through for this file, without failing the file. Every subclass parse body is guarded by if (!m_ChunksOwnsPointers) so that a parent's disown short-circuits the children. Codified rule: graceful degradation is per-object, and "degrade" always means "keep the original bytes", never "drop".
  • clean: only legal after a successful parse. Deletes the raw chunks that the typed representation fully supersedes (and the archived ones). After clean, the original bytes are gone — build must regenerate them bit-perfectly from typed state. A class whose build cannot yet do that must not clean (compare CAppData, which cleans its raw entry values because CAppDataEntry::build re-serializes them, with CDllEntry::clean, which deliberately does nothing because m_Chunks retains ownership of the two value chunks it aliases).
  • build: CSceneClass re-inserts the remaining orphans into m_Chunks, then sets an insertion cursor before the orphans; putChunk(id, obj) inserts at the cursor, so class-written chunks precede leftover orphans while both keep their internal order. putChunk calls build on any container passed through it. The putChunkValue<T>/getChunkValue<T> pair handles single-value chunks and archives them automatically.
  • disown: nulls every typed reference, clears orphan/archive bookkeeping, flips the flag back to true, recurses. After disown, the object is again pure storage and can be serialized out. CSceneClass::disown sanity-checks that build actually ran (orphan count vs chunk count) — see the defect list for a caveat on that check.

The null-rewrite property (codify as the primary test): for any file in the corpus, serial(in) → serial(out) unparsed, and serial(in) → parse → clean → build → disown → serial(out) parsed-but-unmodified (clean is mandatory — build asserts it; see the T2 note in §9), must both reproduce every stream byte-identically. The lifecycle above is precisely the machinery that makes the second path equal the first; any new typed class is a new way to break it, which is why the corpus gate (§9) reruns it wholesale.

Known deliberate normalization (whitelist, currently the only one): CAnimatable::build discards an AppData container that has zero entries (animatable.cpp:100-112). A source file carrying an empty AppData chunk will lose it on a parsed rewrite. Keep the whitelist explicit in the harness; do not let new normalizations in without listing them here.

6. Scene layer — directories, registry, unknown classes

  • CSceneCSceneClassContainer (scene.h/.cpp): the Scene stream's root chunk contains one child chunk per scene object. The child chunk's id is the object's index into ClassDirectory3, which in turn names the ClassId/SuperClassId/DLL. Object identity in references is the position (storage index) of the object chunk within this container.
  • Special ids 0x2032 ("OSM Derived", ClassId (0x29263a68, 0x405f22f5)) and 0x2033 ("WSM Derived", (0x4ec13906, 0x5578130e)) — derived-object wrappers that hold modifier stacks — are hardcoded in CSceneClassContainer::createChunkById as unknown-class instantiations rather than directory lookups.
  • CSceneClassRegistry maps (SuperClassId, ClassId)ISceneClassDesc, and SuperClassIdISuperClassDesc. Resolution order in createChunkById: (1) exact registered class; (2) createUnknown on the registered superclass — this instantiates CSceneClassUnknown<TSuperClass>, which inherits the typed superclass, so e.g. an unimplemented material still runs CReferenceMaker::parse and gets its reference wiring parsed and rebuilt while its own chunks stay orphaned/verbatim; (3) an invalid CSceneClassUnknown<CSceneClass> fallback. This three-tier scheme is why the library can parse every Ryzom file today with only ~15 typed classes: parse coverage and byte fidelity are decoupled.
  • CBuiltin::registerClasses (builtin.cpp) registers the typed classes and ~40 CSuperClassDescUnknown<CReferenceTarget, sclassid> entries covering every superclass observed in the corpus (controllers, materials, texmaps, param blocks, modifiers, lights, cameras, etc. — all currently routed to reference-target-typed unknowns). CUpdate1::registerClasses adds EditableMesh; CEPoly::registerClasses adds EditablePoly; BIPED::CBiped::registerClasses adds BipedDriven. A new typed class = write the class + add one registry->add line; nothing else changes. Plugin-DLL grouping is mirrored in namespaces (BUILTIN, UPDATE1, EPOLY, BIPED) with a IDllPluginDesc per DLL (update1.dlo, EPoly.dlo, biped.dlc); follow that pattern for new plugin classes (e.g. nelexport.dlu would get its own namespace if it ever needs scene classes — though NeL data itself doesn't; see §8). CBipedDriven originally landed under BUILTIN — corrected 2026-07-05 after confirming against the corpus that its ClassDirectory3 entry resolves through a real DLL entry (not the internal "Builtin" pseudo-entry), same as EditableMesh/EditablePoly's. Note the IDllPluginDesc::displayName() strings for these three deliberately don't reproduce the real per-vendor description text found in the corpus — see pipeline-max-vendor-naming (session memory) for why and how far that convention extends.
  • CDllDirectory / CClassDirectory3 (marked "Parallel to" each other — keep them symmetrical): fully parsed into entry vectors plus a m_ChunkCache that remembers where the entry run sat among any non-entry chunks, so build re-emits entries at the original position. Both throw EStorageParse if entries are interleaved with other chunks (never observed). Internal pseudo-indices: dll index −1 = builtin, −2 = maxscript script. getOrCreateIndex appends new entries when writing new classes into a file. reset() exists to blank the directory for from-scratch writes.
  • Storage-index mapping is currently read-only: CSceneClassContainer::parse assigns indices by position, build's getOrCreateStorageIndex just looks up the frozen map ("Temporary 'readonly' implementation" — all four lifecycle methods). The scene-graph TODO in scene.h ("Evaluate the references tree to build new indexes") is the missing piece for adding/removing objects; until it's done, the pipeline can modify scenes but not restructure them. The export direction (max → NeL) doesn't need it; a TOML-sidecar → .max authoring direction eventually would.

7. Object model — references, nodes, geometry

Typed class hierarchy as implemented (✓ = has real parse/build logic; ✗ = registered but pass-through; stub = source file exists, empty 59-line skeleton, not registered):

CSceneClass
└─ CAnimatable ✓ (AppData 0x2150, unknown 0x2140)
   └─ CReferenceMaker ✓ (references 0x2034/0x2035, 0x204B, 0x2045/0x2047/0x21B0 raw)
      ├─ CSceneImpl ✓ (fixed 11/12 reference slots; the builtin scene singleton)
      └─ CReferenceTarget ✓ (no own chunks)
         ├─ INode ✓ (implicit children set)
         │  ├─ CNodeImpl ✓ (0x09ce version, 0x0960 parent+flags, 0x0962 name)
         │  └─ CRootNode ✓
         ├─ CTrackViewNode ✓ (0x0110/0x0120/0x0130 per child, empties 0x0140/0x0150)
         ├─ CBaseObject ✓ (no own chunks)
         │  └─ CObject → CGeomObject ✓ (0x08fe GeomBuffers, 0x0900)
         │     ├─ CTriObject ✓ (ids 0x0901-0x0904 noted, not yet claimed)
         │     │  └─ UPDATE1::CEditableMesh ✓ (known-unknown sequence kept in order)
         │     ├─ CPolyObject ✓ (ids 0x0906-0x090c noted, not yet claimed)
         │     │  └─ EPOLY::CEditablePoly ✓ (known-unknown sequence kept in order)
         │     └─ CPatchObject → CEditablePatch (✓ registered, pass-through bodies)
         ├─ CModifier, CMtlBase, CMtl, CStdMat, CStdMat2, CMultiMtl,
         │  CTexmap, CBitmapTex, CParamBlock, CParamBlock2   ← stubs (empty, unregistered)
         ├─ BIPED::CBipedDriven ✓ (0x0200 bone_id+link_index; ClassId (0x9154,0), exact match
         │  under the 0x9008 ControlTransform unknown-superclass fallback — "BipSlave Control" /
         │  "BipDriven Control")
         └─ (everything else: CSceneClassUnknown<CReferenceTarget> via superclass descs)
  • References (CReferenceMaker): two storage forms, mutually exclusive — 0x2034 (flat index array, position = reference slot, −1 = empty) and 0x2035 (leading unknown uint32, then (slot, index) pairs). m_ReferenceMap records which form was parsed so build re-emits the same form; the leading 0x2035 value is preserved. Indices resolve through container()->getByStorageIndex, live pointers are CRefPtr (non-owning, self-nulling — the scene container owns objects). Subclasses override getReference/setReference/nbReferences to give slots semantic names (CSceneImpl's 12 slots; CNodeImpl does not — node parent/name live in node chunks, and slot 1 of a node is its object, per INode::dumpNodes).
  • Version-dependent shape: CSceneImpl::parse asserts 11 references, 12 (m_TrackSetList) only when version > Version3. This is the template for all version forks: assert what the version implies, don't silently accept both.
  • Nodes: parent linkage is stored child→parent (0x0960: parent storage index + flags); setParent maintains the in-memory child sets. INode::find resolves by case-insensitive user name — that is the instance-name hook the TOML sidecar route will use.
  • Geometry (STORAGE::CGeomBuffers, chunk 0x08fe): typed serializers exist for the tri buffers (0x0914 vertices / 0x0912 CGeomTriIndexInfo faces incl. smoothing groups / 0x0916+0x0918 and 0x0938+0x0942 secondary vertex/index pairs) and the poly buffers (0x0100 vertices / 0x010a edges / 0x011a faces with material, smoothing, and triangulation cuts), but the id→type mapping is compiled out: PMBS_GEOM_BUFFERS_PARSE is 0 (geom_buffers.cpp:52), so geometry currently rides through as raw. Enabling it is the first real corpus test of the typed leaf serializers. CGeomObject::triangulatePolyFace (geom_object.cpp:162) converts poly faces + cuts into triangles via an edge-travel matrix, asserting triangles == cuts + 1; pipeline_max_dump::exportObj already produces valid .obj output from an EditablePoly through it. The old map-extender surgery in the rewrite tool shows the tri-object counterpart chunk pairs (0x2500→0x2512→0x03e9/0x03eb on derived objects).
  • Two-pass parse via filters: parse(version) with filter 0 handles the common object chunks; geometry is deferred until after the subclass has drained its own known chunks, then invoked explicitly as CTriObject::parse(version, PMB_TRI_OBJECT_PARSE_FILTER)CGeomObject::parse(version, PMB_GEOM_OBJECT_PARSE_FILTER). The filter constants are magic per-class tokens (0x432a4da6 etc.), not flags. Separately, TParseLevel (PARSE_INTERNAL, PARSE_BUILTIN, and the commented-out PARSE_NELDATA, PARSE_NEL3D in storage_object.h:59) sketches the intended depth levels — NeL-data interpretation was always planned as a higher parse level on top of builtin parsing. The export pipeline should revive that enum rather than invent a new switch.

8. AppData — where the NeL data lives

STORAGE::CAppData (chunk 0x2150 on any CAnimatable): header 0x0100 = entry count; each entry 0x0110 = key 0x0120 (ClassId, SuperClassId, SubId, Size) + value 0x0130 (raw bytes). Entries live in a map keyed (ClassId, SuperClassId, SubId); parse fails safe (disown) on any structural surprise, and validates count against the header.

CAppDataEntry::value<T>() lazily reinterprets the raw bytes as a typed storage object by round-tripping through a CMemStream (and can re-cast, invalidating the previous pointer — one live typed view per entry). clean empties the raw copy; build re-serializes the typed view back into the raw chunk and fixes the key's Size. getOrCreate + init support authoring new entries.

All NeL export properties are AppData entries, not scene classes. The 3ds Max plugin stores them keyed by MAXSCRIPT_UTILITY_CLASS_ID (0x04d64858, 0x16d1751d) / superclass 4128 with NEL3D_APPDATA_* sub-ids defined in plugin_max/nel_mesh_lib/export_appdata.h (LOD config, accelerator/cluster flags, instance name/group, don't-export, collision, realtime light, water params, vegetable, lightmap settings, …). So the "parse the NeL data" milestone (PARSE_NELDATA) is: typed decoders for those AppData blobs on CAnimatable — zero new scene classes required. The dump tool's commented-out AppData walk (pipeline_max_dump/main.cpp:289) is the reference snippet.

9. Existing tools, and the test harness to grow out of them

  • pipeline_max_dump — single hardcoded file; parses directories + scene, and already performs the canonical smoke sequence: serial in → serial out to temp.bin → serial back in → parse → clean → build → disown → parse → dump, plus root-node tree dump and the .obj export prototype. This is the embryo of the per-file roundtrip check; it needs argv, per-stream byte comparison, and exception-based failure reporting instead of hardcoded paths and nlerror aborts.
  • pipeline_max_corpus_test (2026-07-05) — the per-file corpus tester grown out of the dump tool. Takes <input.max> + optional --parse, runs T1 (structural roundtrip via CStorageContainer default) and T2 (typed parse→clean→build→disown → serialize back) per chunk-formatted stream (DllDirectory, ClassDirectory3, ClassData, Config, Scene, VideoPostQueue), byte-compares each. Writes each round-tripped stream to a PID-suffixed temp file (/tmp/pipeline_max_corpus_test.<pid>.tmp, 2026-07-05 — was a fixed shared path, which a stray concurrent invocation corrupted during development, producing a spuriously high failure rate that had nothing to do with the code under test) via NLMISC::COFile and reads it back — going through CMemStream doesn't work because its length() returns the current write position and CStorageChunks::leaveChunk needs to seek past that position to restore after patching the size field. On failure prints a per-stream one-line diff summary (first mismatch offset + 8-byte hex both sides). \05SummaryInformation / \05DocumentSummaryInformation / the OLE class-id are raw byte blobs not chunk-formatted, so byte identity is trivial and they aren't retested. Driver skel_corpus.py enumerates the corpus from the workspace directories.py files, classifies biped vs non-biped by scanning for the Biped Object UTF-16 marker, skips git-lfs stubs (files whose first 8 bytes aren't the OLE magic), and reports per-bucket totals — invoke python3 skel_corpus.py --all for the full T1+T2+T3 sweep. Wired into ctest as pipeline_max_skel_corpus (pipeline_max_corpus_test/CMakeLists.txt): self-skips (exit 77, SKIP_RETURN_CODE) when the private ryzomcore_graphics/ryzomcore_leveldesign asset checkouts aren't present, so it's harmless on any machine other than Kaetemi's; fails on any T1/T2 byte mismatch, informational-only on T3.
  • pipeline_max_rewrite_assets — the 2012 database migration tool and the proof that parse-free structural rewriting works at corpus scale: it read every stream, optionally rewrote texture paths inside raw/string chunks (with a huge accumulated table of database renames — historical, not design), retargeted a modifier ClassId across the class directory, and wrote the OLE file back with the class id preserved. Its config is all hardcoded globals (w:\database\ mapping, WriteDummy, HaltOnIssue, overrideFF for ligo quirks). Treat it as reference code for the OLE read/write envelope (handleFile, main.cpp:1085) and the fix-up patterns; do not extend it in place.
  • Codified test protocol for the extension work (the tutorial-roadmap side-quest's "exhaustive unit test suite"):
    • T1 — structural roundtrip, no parse. Every stream of every corpus file: serial in → serial out → byte-compare per stream. Must pass 100%; failures are chunk-layer bugs (or 64-bit/compressed files, which must be counted and reported, not skipped silently). Status 2026-07-05 (skel corpus, pipeline_max_corpus_test over 179 .max files, 169 biped + 9 non-biped + 1 git-lfs stub): T1 green on all six chunk-formatted streams for all 169 biped + 9/9 non-biped files (169/169, up from 168/169) — the mixed-64/32-bit-header defect below is fixed. The single git-lfs pointer file (tr_mo_kami_warlord.max) is filtered out by the driver via OLE-magic check.
    • T2 — parse/build roundtrip. serial → parse → clean → build → disown → serial → byte-compare, whitelisted normalizations only (§5, currently: dropped empty AppData). Rerun over the full corpus after every newly typed chunk or class — this is the non-negotiable gate that keeps typed coverage from eroding fidelity. Note: the code enforces clean() between parse and build (CDllDirectory::build/CSceneClass::build assert m_Chunks.empty()); the phrase "parse → build → disown" without clean is misleading. Status 2026-07-05: T2 green on 169/169 biped + 9/9 non-biped skel-corpus files (up from 2/169 biped on 2026-07-05, then 168/169 after the AppData-order/container-bit fixes, now 169/169 after the per-chunk 64-bit width fix — see §11). Verified against the wider ~8.6k-file corpus too: two independent random samples (400 files seed 42, 800 files seed 7 — 1200 distinct files, ~14% of the corpus) are fully clean, 0 T1/T2 failures, against a measured pre-fix baseline of ~15% failing on the same population (the mixed-header defect turned out to be far more widespread outside the skel-only corpus than the skel baseline suggested). Also fixed this session: 4 unrelated pre-existing crashes from an unregistered superclass (0x1190, "Depth of Field (mental ray)", on *_fp.max first-person-hand files) — see §11.
    • T3 — export snapshot. Once export logic exists: pipeline output diffed against the prebuilt Ryzom Core data set (~/core4_data) as reference — structural fields exact, floats within epsilon, mismatches triaged (plugin-version differences do exist in the reference data; the triage bucket is part of the design, decided per output type, not ad hoc). Status 2026-07-05 (second session): full-coverage skel T3 — the driver finds references for biped fauna rigs under fauna_skeletons too (previously only characters_skeletons was checked for biped-kind files, silently skipping 114 of 169) — reports size-match 169/169, and a large reference-era triage class was identified (see §10 "Reference-era mismatches"): _big/_small variants rescaled ±20% and per-race humanoid re-proportioning postdate the reference exports, so those refs are not reproducible from the current max sources by design. Known reference-data caveats, pre-triaged (Kaetemi, 2026-07-05):
      • Flare sizing flag (_FirstFlareKeepSize). Some flare assets carried this flag mis-set in the max sources; an export/build-side hack once flipped it based on file version to make the live data look right, and was later removed because it broke assets that expected the reverse. The prebuilt reference data predates parts of this cleanup, and the max files may since have been fixed. So flare-shape sizing-flag mismatches against the reference are expected: resolve per-asset from the max source (the max file is authoritative), never by teaching the pipeline to reproduce the reference. Note the engine loader still has its own version clamp — CFlareShape::serial forces the flag false for shape versions ≤ 4 (nel/src/3d/flare_shape.cpp:77) — so what the game renders is not always what the shape file stores; compare stored values, and account for the loader clamp when judging runtime-visible differences.
      • Non-standard UVW mapping modifier. Some assets use a custom UVW-mapping extension written by a Ubisoft employee (a non-standard plugin modifier — it announces itself in the file's DllDirectory, so detection is trivial and unambiguous). The reference .shape outputs of those assets contain garbage UVs; UV equality against the reference is meaningless there. The harness must auto-flag these files by DLL/class entry and route them to a dedicated bucket — either the modifier gets implemented properly (making our output better than the reference) or the assets are recorded as excluded-with-reason; silent inclusion in pass/fail statistics is not acceptable. The flagged set must be emitted as an explicit asset list report (file path, node names, modifier DLL/class entry): Kaetemi may be able to locate original exports from older live clients to serve as known-good UV references for these assets. Caveat on that caveat: such old exports may themselves derive from older revisions of the max files, so treat them as corroborating evidence for the UV channel, not as byte-exact snapshot references.
      • Lightmapped scenes. The headless pipeline exports these unmapped; the lightmapping phase (currently calc_lm* inside nel_mesh_lib, run at export time inside Max) is replicated as a separate standalone tool downstream. Consequently T3 compares unmapped exports structurally, and the lightmapper is validated in aggregate — scene-level comparison of its output against reference lightmaps (statistical/image-level tolerance), not per-shape byte equality.
    • Bucket all results per file version (Max 3 … 2010) — regressions cluster by version.
    • toString dumps are diagnostics for triage; never assert on them.

10. Reference: how the original 3ds Max exporter is actually run

The authoritative reference for export sequence, options, and outputs is the build_gamedata pipeline (nel/tools/build_gamedata, Python — Kaetemi's 2010 port), configured by the workspace in the ryzomcore_leveldesign repository (cloned locally at ~/ryzomcore_leveldesign; the workspace is workspace/). The headless pipeline's integration target is to drop in at the 1_export stage with the same workspace contract, replacing the 3ds Max invocation while honoring the same inputs, options, outputs, and tag protocol.

  • Structure: four phases per process — 0_setup.py / 1_export.py / 2_build.py / 3_install.py — under processes/<name>/. Eleven processes are Max-driven (have a maxscript/ directory): shape, anim, ig, ligo, zone, skel, swt, veget, clodbank, pacs_prim, rbank.
  • Workspace config: workspace/projects.py lists projects (common/objects, common/sfx, common/characters, per-continent, per-ecosystem, …). Each project directory holds directories.py (source directories inside the graphics database, export/build directory names, tag directories, lookup paths) and process.py (which processes run, plus per-project export options — e.g. ShapeExportOptExportLighting, ShapeExportOptShadow, ShapeExportOptLumelSize, ShapeExportOptOversampling; BuildQuality == 0 globally downgrades these for fast builds). The headless exporter must consume these same options with these same names.
  • Max invocation (see processes/shape/1_export.py): the Python driver template-substitutes %TOKENS% (source dir, output dirs, options, log path) into maxscript/shape_export.ms, installs it into the Max user scripts directory, and runs 3dsmax.exe -U MAXScript shape_export.ms -q -mi -mip — Max then iterates every .max in the source directory itself. Incrementality and crash-resilience live in the tag protocol: a .max.tag per successfully exported source file (compared by needUpdateDirByTagLog), a max_running.tag sentinel written before each Max launch (still present afterwards ⇒ Max crashed), and a retry loop that measures progress by tag-count delta with a zero-progress retry limit of 3. The headless pipeline keeps the tag files (other stages and the incremental logic depend on them) and can discard the sentinel/retry machinery — that exists purely because Max crashes.
  • Per-node export policy is in the maxscript, driven by AppData: shape_export.ms::isToBeExported skips nodes with NEL3D_APPDATA_DONOTEXPORT, collision flags, etc., and calls the NelExport plugin per eligible node. This logic moves into the headless exporter's own node filter, reading the same AppData keys (§8).
  • Shape export outputs (per project): shape_not_optimized/, shape_with_coarse_mesh/, shape_lightmap_not_optimized/, plus the anim directory — note that today lightmaps are computed at export time inside Max and then post-processed by lightmap_optimizer in 2_build.py. Under the unmapped-export design (§9 T3 caveats), the export stage stops producing shape_lightmap_not_optimized content and the standalone lightmapper takes over that slot in the flow, feeding the same 2_build optimizer.
  • Precedent for a non-Max route already exists in-tree: processes/shape/1_export.py first runs mesh_export (nel/tools/3d/mesh_export, assimp-based: .blend/.obj/.dae/.gltf) over the same source directories with the same tag protocol before falling through to the Max branch. The headless .max exporter should be wired in exactly the same shape as that branch — and mesh_export is also the natural home or template for the TOML-sidecar import route (tech tree A2).

Pilot done: pipeline_max_export_skel (2026-07-05). nel/tools/3d/pipeline_max_export_skel/main.cpp produces .skel files from .max for the non-biped subset of the skeleton corpus (9 of the 178 usable files — the harness counts 179 enumerated minus 1 git-lfs stub, 169 biped + 9 non-biped; the "182" in earlier notes was a pre-harness estimate: ge_mission_*, fo_carnitree, pr_mo_phytopsy, tr_mo_electroalg, fy_mo_swarmplant, ju_mo_endrobouchea, ju_mo_sapenslaver). Traverses Bip01 in scene-order, reads sub-controller default values from raw chunks (0x2503 CVector position, 0x2504 CQuat rotation, 0x2505 first-12B CVector scale — no new typed classes required), accumulates world matrices, emits .skel matching NeL's CShapeStream + CSkeletonShape + CBoneBase v2. Structural correctness: 9/9 produce identical size to ~/core4_data reference, matching bone count/names/hierarchy and CMatrix state bits. Byte-level match: 61–96% (T3-epsilon float noise in InvBindPos rotation cells; not reproducible without replaying Max's exact arithmetic operation order). --double flag is available for depth-accumulated precision but doesn't help byte-match since Max is float throughout. Corpus tester (2026-07-05) confirms: 6/6 non-biped files with an available ~/core4_data/fauna_skeletons reference produce size-identical .skel at avg 75.8% byte-match (per-file pipeline_max_corpus_test/skel_corpus.py --t3 bucket report). The 3 fauna files without local reference are held out until the reference data catches up.

Biped detection + degraded fallback (2026-07-05). looksLikeBipedFile() in pipeline_max_export_skel/main.cpp scans the ClassDirectory3 for any of the four confirmed Biped ClassIds (below). DllDirectory-based detection is a false positive because Max loads biped.dlc even when no biped exists in the scene; ClassId-based detection also survives the display-name rename (BipSlave_ControlBipDriven_Control in Max 2022). (Historically biped files refused to export unless --allow-biped-degraded was passed, which wrote identity local transforms; as of commit d89f0a7b8 the figure-mode bind pose is reconstructed and the flag is a no-op — see "Biped figure-mode reconstruction" below.)

Confirmed Biped class identities (Autodesk character-studio MAXScript reference, 3ds Max 2023):

Display name in .max ClassId SuperClassId Role
Biped {0x9155, 0} 0x60 (Object) The biped system root — holds structure params, figure-mode state, per-bone table
Vertical/Horizontal/Turn {0x9156, 0} 0x9008 (ControlTransform) COM/body controller attached to Bip01's transform track
BipSlave Control (pre-2022) / BipDriven Control (2022+) {0x9154, 0} 0x9008 (ControlTransform) Per-body-part TM controller; every non-COM biped bone has one as its getReference(0)
Biped SubAnim {0x6b147369, 0x078c6b2a} 0x9003 (ControlFloat) Sub-anim override; referenced as slot 2 of each BipDriven Control
Biped Object {0x9125, 0} 0x10 (GeomObject) Visual per-body-part geometry, attached as getReference(1) of each biped INode

Each BipDriven Control's chunk 0x0200 (8 bytes = 2 uint32s) is (bone_id, link_index) per Autodesk's biped.getIdLink(node) return. The MaxScript-facing IDs in the SDK docs are 1-based (1=#larm .. 22=#prop3); the internal IDs stored in 0x0200 are the same enum shifted to 0-based. Confirmed by dumping each biped node's controller in fy_hom_skel.max:

Internal ID (0x0200) MaxScript ID Name Observed bones
0 1 #larm L Clavicle (link 0), L UpperArm (1), L Forearm (2), L Hand (3)
1 2 #rarm R Clavicle/UpperArm/Forearm/Hand
2 3 #lfingers L Finger0..L Finger42
3 4 #rfingers R Finger0..R Finger42
4 5 #lleg L Thigh, L Calf, L Foot
5 6 #rleg R Thigh, R Calf, R Foot
6 7 #ltoes L Toe0..
7 8 #rtoes R Toe0..
8 9 #spine Bip01 Spine, Spine1, Spine2
10 11 #head Bip01 Head
11 12 #pelvis Bip01 Pelvis
15 16 #footprints Bip01 Footsteps
16 17 #neck Bip01 Neck
17 18 #pony1 Bip01 Ponytail1/11/12

Empirically observed IDs in the corpus reach up to 28, suggesting the internal biped has a few more IDs than the 22 documented (likely twist bones or extension ids); worth cataloguing in a full sweep when finger/toe reindexing gets addressed.

Biped bone rotation/position inheritance is not INode-parent inheritance. From the character-studio SDK docs, biped bones inherit differently from what the scene graph says:

  • Rotation source: base of spine, base of neck, upper arms, upper legs, feet → COM. Clavicle → last spine link. All others → their INode parent.
  • Position source: base of spine → COM. Clavicle → last spine link. Upper leg → pelvis. All others → their INode parent.

That means the current walkNode(bip01, -1, parentWorld, …) in pipeline_max_export_skel — which just multiplies each node's local TM by its INode-parent's world TM — is structurally wrong for biped bones the moment the biped rig is non-trivial (e.g. clavicles skip a level of inheritance vs Max's INode chain). The right shape is:

  1. Walk the INode hierarchy for name/order (as today).
  2. For every biped bone (has a BipDriven Control as its TM controller), read chunk 0x0200 to identify (bone_id, link_index).
  3. Look up the local transform for that (id, link) in the root Biped (0x9155) controller's per-bone table — the transforms Autodesk stores are what INode::GetNodeTM(time) returns in figure mode, and each is already the world transform; the .skel bind pose derives from that plus the biped inheritance rules above (used to reset the root and compute InvBindPos).
  4. UnheritScale on the .skel bone must be set to true for biped bones whose INode parent is also a biped bone (from getNELUnHeritFatherScale in plugin_max/nel_mesh_lib/export_skinning.cpp).

The per-bone table lives in the root Biped controller across chunks 0x000b (80B), 0x000d (72B), 0x000f (232B, dual snapshot), 0x0010 (936B, dual snapshot) — decoding the (id, link) → transform mapping in 0x0010 is the actual open work.

Biped figure-mode reconstruction (2026-07-05, pipeline_max_export_skel commit d89f0a7b8). The biped bind pose is now reconstructed — arms and legs point correctly, not identity. --allow-biped-degraded is a no-op (the tool always reconstructs). The exporter reads the root Biped (0x9155) object's per-limb record chunks and supplies correct local transforms for the decoded roles, falling back to straight-chain inheritance for the not-yet-decoded ones. Corpus accuracy over the full 169-biped set (55 files with a local reference, 4014 bones):

Metric Reconstruction First-order approx Identity baseline
dpos exact+close (< 0.02) 78.2% (647+2614 / 4171) 67.3% ~1.2%
drot exact+close (< 0.02) 63.3% (2242+398 / 4171) 0% (all identity) 0%
Avg dpos error 0.039 0.045 ~0.16
Files size-matching ref 29 0 0

(Bone-count and percentages as of 2026-07-05 after the clavicle-position and finger/toe-nub decodes; see below. T1/T2 stood at 168/169 when this table was made — the one fail was the mixed-64/32-bit-header file, since fixed; current status is 169/169, see §9.) Non-biped T3 unchanged (9/9), no crashes across the biped corpus — including the *_fp.max first-person-hand files, which previously aborted the whole process on an unregistered superclass (0x1190, "Depth of Field (mental ray)"); fixed by adding it to CBuiltin::registerClasses alongside the other ~40 unknown-superclass pass-throughs (builtin.cpp), same pattern as every other unimplemented superclass in the corpus. pipeline_max_corpus_test/skel_corpus.py now reports drot accuracy alongside dpos.

The decode — how it works. Everything the reference exporter does reduces to one input: node.GetNodeTM(time), the figure-mode world matrix per bone (see §10 buildSkeleton). Ground truth for it is the reference .skel's InvBindPos inverted. The reconstruction rebuilds those world transforms:

  1. Coordinate frame. The biped's internal representation is Y-up (Max/NeL world is Z-up). Position conversion (x,y,z) → (x,-z,y). Rotation basis change C = [[1,0,0],[0,0,-1],[0,1,0]]; a stored Max-row-vector 3×3 M becomes the NeL local rotation R_nel = C · Mᵀ (columns I=(m0,-m2,m1) J=(m4,-m6,m5) K=(m8,-m10,m9)), then CMatrix::getRot(). Verified bit-exact on the finger matrices.
  2. Aim==exact. Each bone's world local-X axis points exactly at its child (I·(child_wp−self_wp)/|…| = +1.000 for every single-child bone, all templates). So straight-continuation bones (Spine1/2, Neck, Calf, Forearm, finger/toe/ponytail sub-links) have identity local rotation — matching the reference drot ≈ (0,0,0,±1) — and only direction-changing joints carry real rotation. This is why "read the joints, inherit the rest" works.
  3. Base frame. Bip01 (COM) world = chunk 0x0104 (a Y-up 4×4; its translation converted gives Bip01's world position ≈ (0,0,height)). Bip01's world rotation is the canonical CQuat(0,0,-√½,√½) (−90° about Z), constant across the corpus — note this differs from the .skel DefaultRotQuat, which is identity for the root (reset) and uses the decompMatrix convention, not the getRot()/world convention the joint decodes are validated against. Pelvis local rotation is the constant (0.5,0.5,0.5,-0.5) (COM→pelvis frame reorientation), so pelvisWorld = Bip01World · (0.5,0.5,0.5,-0.5).
  4. Direction-changing joint rotations are stored as quaternions at fixed float offsets in per-limb record chunks. (SUPERSEDED 2026-07-06 — see "Differential-dataset decode round" below: the offsets were validated but the perm/sign forms for Foot and UpperArm were only approximately right, the halves are right-side-first, and each side reads its own half. The original table is kept for history:)
Joint id.link Chunk[off] perm sign Frame
Thigh 4/5 .0 0x0069[2] (2,3,0,1) (+,+,−,+) pelvis-relative
Foot 4/5 .2 0x0069[28] (2,0,3,1) (+,−,−,+) world COM-relative
UpperArm 0/1 .1 0x006a[2] (1,2,3,0) (+,−,+,−) pelvis-relative (~4° off — superseded)
Head 10 .0 0x0064[0] (0,1,2,3) (+,+,+,+) pelvis-relative

World rotation = pelvisWorld · q (pelvis-relative) or q (world). Local rotation = parentWorld⁻¹ · worldRot, fed to walkNode so InvBindPos and default tracks derive exactly as for non-biped bones. 5. Right side = left mirror, applied in the pelvis-relative frame only (the one frame where the mirror is the clean (x,y,-z,-w) rule). (2026-07-06: refined — the mirror is real but each side reads its own half, whose right-side values are mirror-encoded; the foot needs no mirror at all. See below.) 6. Finger/toe bases store a full 4×4 local matrix (relative to Hand/Foot). Arm record 0x0010 layout: [0..15] arm params (incl. [10] = clavicle Z-offset), [16]=nFingers, then per finger [nLinks][4×4 matrix (16 floats, Y-up)][nLinks length floats]; consumes exactly 117 floats per half (L then R). Generalizes across all templates (5 fingers × 3 links); simpler bipeds without fingers correctly lack the chunk. Decoded via the §1 conversion → exact local pos+rot.

SDK-confirmed hierarchy (Autodesk character-studio Biped Hierarchy page, matches the decode): rotation inherits from COM for spine-base/neck-base/upper-arms/upper-legs/feet, from last spine link for clavicle, else from parent INode; position from COM for spine-base, from pelvis for upper-leg, from last spine link for clavicle, else parent INode. DOF: pelvis 2, clavicle 2, knee/elbow 1, finger/toe non-base 1 — the 2-DOF joints (getClavicleVals/getPelvisVal) and hinge joints (getHingeVal) are the still-undecoded ones. Leg link order is Thigh, Calf, HorseLink, Foot — 4-link legs (legLinks=4) put Foot at link 3, not 2 (the link==2 Foot check must generalize to "last leg link" for horse/mount rigs).

Assimp roundtrip validation (2026-07-05). pipeline_max_export_skel --gltf <path> also writes a glTF 2.0 skeleton-only file (JSON, no meshes or skinning) storing the REAL local transforms (not the root-reset values, so the InvBindPos noise pattern is preserved). mesh_utils/assimp_skel.{h,cpp} adds exportSkels(): finds the bone root via scene_meta TBoneRoot or a case-insensitive "Bip01" descendant, walks the aiNode tree in mChildren order, decomposes aiNode.mTransformation into T/R/S, and emits .skel via the same identity+setRot+setPos+invert chain as the direct path. Cross-path parity over the 9-file non-biped corpus: 83.7–95.0% byte match A-vs-B, with the ~5–16% delta from float precision loss through the glTF → aiMatrix4x4 → Decompose cycle. Notably B is often closer to the reference than A for large fauna files (68% vs 62%) — assimp's Decompose renormalization happens to track Max's arithmetic better than direct CMatrix multiplication. The glTF is loadable in Blender as an armature/empty hierarchy, making this the artist path off Max as well.

Method note (how the decodes were found). The chunk offsets/perms/signs above were located by a family of scratchpad tools driven off ground truth (biped_analyze.cpp dumps every biped chunk's floats + emits per-bone #GT world transforms from the reference .skel; Python matchers scan for 4×4 windows / quaternions matching a bone's world or pelvis-relative rotation under permutation+sign, then verify generalization across ≥4 templates before trusting a decode). The generalization gate is essential: an earlier attempt found chain-root Z-offsets at fixed indices in 0x000f/0x0010 that were bit-exact on fy_hom_skel but wrong on ma_hom_skel/tr_hof_skel — the record layout shifts per-rig, so fixed indices into variable-length sections do not hold. The decodes that shipped are all at offsets before any variable-length section (the limb-record headers), which is why they generalize.

Clavicle position decoded (2026-07-05, commit a5e3e848f). Clavicle attaches to the last spine link at a pure Z (side) offset stored in the arm-record header at 0x0010[10] (exact on fy_hom/fy_hof, within the 0.02 "close" tolerance on ma/tr). Set dpos = (0,0,±0x0010[10]) (mirror z for R). Corpus biped dpos 75.1% → 77.6% exact+close.

Finger/toe end-effector nub ids catalogued and decoded (2026-07-05). The "empirically observed IDs reach up to 28" note above is now resolved for the finger/toe case: scanning chunk 0x0200 across all 9 templates with a local reference (fy/tr/zo/ma male, fy/tr/zo/ca female, ca male) found four stable extension ids beyond the 22 MaxScript-documented groups, all end-effector marker dummies with no children (so the "aim at child" rule that gives straight-chain bones identity rotation doesn't apply to them):

Extension id Role Required local drot
22 R finger-tip nub (link = which finger, 0-4) (0,0,-1,0) (180° about Z)
23 L finger-tip nub (link = which finger, 0-4) identity — already exact via the default
24 L Toe0-tip nub (0,0,-1,0) (180° about Z)
25 R Toe0-tip nub identity — already exact via the default

Ids 26-29 were also observed (26 = Tail on ca_hom/ca_hof, 27 = a second head-attach dummy, 28 = Ponytail1 on a second attach point, 29 = a neck accessory dummy on ca_hom) but aren't decoded — no reference mismatch currently traces to them, so there's nothing to validate a decode against yet (see the "type only what you can prove" rule, §12.2). pipeline_max_export_skel now special-cases ids 22/24 to the constant 180°-about-Z local rotation; corpus drot exact+close rose 55.6% → 63.3% (2242+398 of 4171) with no dpos/T1/T2 regression.

UnheritScale controller-flag reader: confirmed not exercised, deferred by design. Checked every bone in the 6 non-biped corpus files that have a local reference: UnheritScale is false for all of them (never true). Building a controller-inheritance-flag reader (INHERIT_SCL_X|Y|Z, Control::GetInheritanceFlags()) would therefore be pure speculation with zero corpus signal to validate against — per §12.2 ("type only what you can prove"), left as the current default (false for non-biped bones) rather than guessed at.

4-link legs (HorseLink) generalized (2026-07-05). The link==2 Foot check is now link == g_maxLegLink, where g_maxLegLink is computed once per file by scanning every BID_LLEG/BID_RLEG bone's link index and taking the max (falling back to 2 if no leg bones are found at all). This is a strict generalization — byte-identical output on every 3-link rig in the corpus today (verified) — that also handles a hypothetical 4-link legLinks=4 mount/horse rig correctly, should one enter the corpus later; no such rig exists today so this specific path stays unverified by a reference file.

Biped structure records fully decoded (2026-07-05, skel-corpus continuation session). The previous session's conclusion that thigh/toe positions are "procedurally computed, not stored" was wrong — they are stored, in the per-limb structure records, and the earlier failure to find them was a reference-era artifact (below). The full record family on the 0x9155 Biped system object:

Chunk Record Layout
0x000b spine struct [0]=?, [1..n] per-link lengths, 4×4 base-attach matrix; n = floats − 17
0x000d pelvis struct [0..1], identity matrix (pelvis fwd-offset not here — still open)
0x000e tail struct same shape as spine
0x0013 / 0x0014 ponytail1 / ponytail2 struct same shape as spine
0x000f leg struct per side half: [0..9] params ([1] = thigh side-offset, pelvis frame (0,0,±v)), [10]=nToes, per toe [nLinks][4×4 matrix][nLinks lengths]
0x0010 arm struct [0..15] params ([7] = clavicle angle, [10] = clavicle Z-offset), [16]=nFingers, per finger [nLinks][4×4][lengths]
0x0067/0x0068/0x006d/0x006e spine/tail/pony1/pony2 per-link angles [0]=int count, then (a1,a2,a3) per link
0x0064 head record [0..3] head quat (known), [7]=int count, [8..] neck per-link angle triples
0x006c COM record [4..6] COM position (Y-up), [8..11] COM world quat (Y-up; NeL = (−x, z, −y, w)) — not always the canonical −90°Z (e.g. tr_mo_balduse), so this replaces the constant
0x0258..0x0261 snapshot bank byte-mirrors of 0x0064..0x006d

Conversions (empirical, corpus-validated): chain-base matrices (spine/tail/pony) use rows-as-NeL-IJK directly with local pos (m13, m12, −m14); toe-base matrices additionally z-flip the quat (x,y,−z,−w) with pos (m12, m13, −m14); right-side toes = mirror of the left half's data (the R half's own matrices use yet another basis — don't parse them). Per-link angle composition: R = Rx(a3) · Rz(−a1) · Ry(a2); chain base local = matrixRot · R(angles[0]). Chain link positions come from the record lengths (the links' own 0x096c dims don't track figure spacing — ca_hom ponytails, every tail); finger/toe link positions keep 0x096c (which does track figure rubber-band scaling that the record lengths don't — tr_mo_kitin_queen's ~24× scaled fingers). Chain end-effector dummies: id 26 (tail nub) / 28 (pony1 nub) / 29 (neck dummy) have the constant local rotation (−√½,−√½,0,0) on every corpus instance; id 27 (head dummy) identity. Neck base sits at the last spine link's stored length.

Multi-biped files. tr_mo_kitin_queen carries two bipeds (Bip01 + Bip02 nested under Bip01 Spine2). Every BipDriven Control and every COM Vertical/Horizontal/Turn controller (ClassId {0x9156,0}) references its own 0x9155 system object as getReference(0) — the exporter now keeps all figure state per-system (SBipedRig), and COM nodes (Bip01/Bip02) take their world transform from their rig's COM record.

Skeleton LODs (byte-structure fix). The reference .skels carry multiple LODs built from NEL3D_APPDATA_BONE_LOD_DISTANCE AppData on the nodes (e.g. finger bones disabled at distance). The exporter now reads that AppData (string-valued script entries, matched by SubId 1423062615) and reproduces CSkeletonShape::build exactly: father-clamped distances, one LOD per distinct distance. This was the missing 88 bytes on otherwise size-matching files — size-match is now 169/169. Also fixed: the reference exporter sets UnheritScale=true on the COM node (checked against ref bytes).

Reference-era mismatches (the trap that hid the stored thigh). 0x000f[1] bit-matches the reference thigh offset on 91/169 files — including every base monster rig. The 78 mismatches are exactly: the _big/_small monster variants (stored values differ from ref by precisely ×1.2 / ×0.8 — the max rigs were rescaled ±20% after the refs were exported; the refs equal the base rig), the humanoid races (ma/tr/zo/ca refs all carry fy-era figure params bit-identically; the per-race max files were re-proportioned afterwards), and tr_mo_kitin_queen (×~2). Per the §9 T3 triage rules, the max file is authoritative; the corpus report should learn to bucket these separately (an "era-matched" filter keyed on the thigh bit-match is what the analysis tooling uses today).

Corpus accuracy after this round (full T3 coverage — the harness now finds references for biped fauna rigs under fauna_skeletons, so all 169 biped + 9 non-biped files are measured, up from 55): biped size-match 169/169, dpos 4324 exact + 4107 close / 11120 (~39% bit-exact; the era-mismatched files above account for most of the rest), drot 7006+538/11120, avg byte-match 68.1%; non-biped dpos 222/222 all bit-exact, byte-match 81.9%. On era-matched files the remaining error is concentrated in: clavicle/hand rotations, calf/knee (+foot local drot), the generic PRS path (prs:other), Footsteps, and a small pelvis forward-offset.

Two PRS-path defects fixed (2026-07-05, same session — these were most of the remaining drot error):

  • Max rotation-controller values are stored in the inverse convention relative to the node-TM rotation. getLocalTransform now conjugates the TCB quaternion (0x2504). The reference exporter never hit this because it read GetNodeTM() matrices, not controller values; our earlier reads passed on exactly the self-conjugate rotations (identity and any 180°), which is what made the bug look like scattered per-bone noise instead of a convention. After the fix the non-biped corpus is 222/222 bit-exact on both dpos and drot (byte-match 81.9% → 90.9%; the rest is InvBindPos float noise), and prs:other went from 422/508 era-matched bones wrong to 7.
  • PRS children of a biped COM node don't inherit the COM's rotation: their stored rotation value is world-frame (position stays parent-relative). Verified bit-exact on every corpus name tag marker including tilted-COM rigs (tr_mo_c03), and it makes Footsteps' drot exact on all 169 files (its controller value is identity ⇒ local = COM⁻¹ = the constant (0,0,−√½,−√½)).

Also landed: clavicle base rotation = 180° about (cos φ, 0, sin φ), φ = π/4 + 0x0010[7]/2 (right side = (−x,−y,z,w) mirror) — exact on planar rigs, −87% clavicle drot overall. Corpus after this round: biped drot 8368+413/11120 exact+close (err 0.1037), dpos unchanged (4324+4107), non-biped fully exact.

COM V/H/T displacement decoded (2026-07-05, same session). 0x006c[0..2] is the COM node's current Vertical/Horizontal/Turn displacement relative to the figure COM, in the COM's local frame: Bip01World = figureCom + ComRot·(d0, −d2, d1) — verified exact on every era-matched corpus file (the z-component discriminated by tr_mo_kitifly/kitikil, the big cases by ca_spaceship's 1.28 shift). The Pelvis stays at the figure COM, so its local position is −(d0, −d2, d1) — pelvis dpos error went 2.69 → 0.001 over the era-matched set (only the scaled mektoub_selle variants remain).

Knee/elbow/horse-ankle hinges decoded (2026-07-06, from the character-studio SDK reference). The SDK's IBipMaster confirmed the DOF model (GetHingeVal — knee/elbow 1-DOF; GetHorseAnkleVal; GetClavicleVals returning exactly two floats; the V/H/T key layouts matching the COM-displacement decode). The hinge figure values sit at the head of each side's pose-record half (halves = chunk floats / 2, per side L then R):

Hinge Slot Local rotation
Knee (calf, leg link 1) 0x0069[0] rotZ([0] − π) (stored as interior angle)
Horse ankle (leg link 2 on 4-link legs) 0x0069[1] rotZ([1]) (as-is)
Elbow (forearm, arm link 2) 0x006a[0] rotZ([0] − π)

Bit-exact across the era-matched corpus (calf/forearm/horse-link drot now 0.000 on humanoids, kami, mounts; the humanoid foot locals follow for free since foot world was already decoded). Known deviations: the kitin family's calf is off by a constant 5.00° and the asymmetric-edited bird rigs (tr_mo_c01/kazoar/lightbird/yber — the same 4 files as the L-toe position anomaly) have a 0.5° L-forearm deviation; both look like post-export figure edits. Also mapped: 0x0100 on the system object = the SetNumLinks table per keytrack ([8]=spine, [9]=tail, [17]=pony1, [22]=nFingers, …).

Differential-dataset decode round (2026-07-06) — the figure-IK "not stored" list was wrong; nearly everything IS stored. A Max 9 differential dataset (140 single-change .max variants generated by biped_dataset_gen.ms: a canonical reference biped + one structural param or one bone's figure pose changed per file, with biped.getTransform world pos/rot ground truth for every bone logged to manifest.txt — at ~/biped_dataset) pinpointed each edit's exact storage floats by per-chunk float diff against the baseline. The Max 9 chunk layout is byte-size-identical to the corpus era, and every decode below was re-validated against the corpus references (numeric hypothesis search over the 24 signed-permutation rotations × reference frames, then the exporter + T3). Corpus-facing results:

  • Half order is RIGHT-first. All paired records (pose records 0x0069/0x006a, and the paired sections of 0x000f/0x0010) store the right side in the first half, left in the second (R Hand lands at 0x006a[28..31], L Hand at [half+28..]; R Clavicle at 0x0010[6],[7], L at [half+6..7]). The earlier left-first assumption was undetectable on symmetric rigs (L == mirror(R)) and is what broke the asymmetric-edited ones (bird-rig L forearm 0.5°, kami_guide_4's L Toe0 180° flip — both resolved). Each side now reads its OWN half; the right half's values are mirror-encoded (left convention), so right-side decodes wrap in the LR mirror (x,y,-z,-w) — except the foot, whose halves store true per-side absolutes.
  • Foot (solves the free-foot problem): [28..31] of the side's 0x0069 half is the ABSOLUTE foot orientation in the COM-relative Y-up frame: R_foot = R_com · C · qmat(s)ᵀ, both sides direct, any orientation (planted, tilted, yawed — exact on the bird/crab/kitin free-foot rigs). The old "world quat, planted only" perm had silently baked the canonical −90° COM yaw into its fixed signs.
  • Hand/wrist (was "not stored"): the default hand frame is the forearm twisted ∓(π/2 − ε) about its X (ε = 0.0008 rad, a fixed biped-internal constant, invariant across rig structure/pose/scale, confirmed corpus-era); the stored quat at [28..31] of the side's 0x006a half is a local post-multiplied delta: R_hand = R_forearm · Rx(∓(π/2−ε)) · C · qmat(s)ᵀ · Cᵀ (right side: LR-mirror the delta). [39..42] holds a second, redundant quat (absolute-when-edited; not needed).
  • UpperArm (old perm was ~4° off even at dataset baseline): exact per-side rule L: R = R_pelvis · Rx(π) · qmat(s)ᵀ · Ry(π), R: R = R_pelvis · Ry(π) · qmat(s)ᵀ · Rx(π), s at [2..5] of the side's half (+ leg-head shift n/a for arms).
  • Clavicle 2-DOF complete: relative to the LAST SPINE LINK (not the neck): rel = Rx(−a) · Rz(−ε) · B(b) with a = armRecord[6], b = [7] per side, B(b) = the known 180°-about-(cos(π/4+b/2), 0, sin(π/4+b/2)); right side = LR mirror of its own-half decode. The same ε as the hand.
  • Finger/toe pose blocks: inside each 0x0069/0x006a half, 10-float blocks from in-half offset 46: [46+10k..49+10k] = chain k's base-link delta quat (local post-multiply qmat(s)ᵀ on the record-matrix base local; right side mirrors the composed local), [54+10k], [55+10k], … = the non-base links' ABSOLUTE bend angles in radians, local rotation Rz(angle) (this replaces "straight-chain" finger/toe links — the humanoid default curl 0.2333 rad was always there).
  • Leg-record head shift: the 0x0069 half's head grows 2 floats per leg link beyond 3 — 4-link rigs (mounts/birds) put the thigh quat at [4..7], foot at [30..33], toe blocks at [48+10k]. Knee [0] and horse-ankle [1] do NOT shift.
  • Neck record 0x0065 is authoritative: [0] = int count of angle floats (3/link), [1..count] = the (a1,a2,a3) triples (same composition), then a 4×4 base-attach matrix. The head-record copy at 0x0064[8..] is stale on 48/169 corpus files (tr_mo_arma/bul, the ge_ kami pair); the exporter now reads 0x0065 with 0x0064 as fallback.
  • Footsteps Z is derived, not stored: the Footsteps marker's world height equals the (left) toe attach height (exact across the dataset's ankle/height/foot-scale variants) — patched post-walk. Figure-mode footsteps rotations don't persist at all (nothing stored, node snaps back). Pelvis figure rotation (0x0066[0..3], a quat) is stored but also not applied to the pose; identity on all 169 corpus files.
  • Head quat 0x0064[0..3] unchanged (pelvis-relative, identity component order).

Follow-up fixes, same day: the corpus-era Footsteps node is a plain PRS child of the COM (not a BipDriven with BID_FOOTPRINTS like later plugin versions), so the ground patch identifies it by name; the reference footsteps sits at world (Bip01.x, Bip01.y, ground), where ground defaults to the z=0 plane / the toe attach height (mm-equivalent; toe-z is bit-exact on fy_hom and fy_mo_frahar) — EXCEPT ~30 rigs whose marker was moved (ca_spaceship's 7.73, capryni, mektoub, c05): that moved ground level is genuinely not stored anywhere in the 0x9155 records (re-verified with value scans; it likely lives in footstep animation keys), left open. The clavicle position is the full 3-vector (armRecord[9], [8], [10]) per side in the parent frame (x/y are zero on humanoids — which is why [10] alone had looked sufficient — but real on monster rigs: tr_mo_c05 (0.068, 0.103), fy_mo_frahar, kamique; the left half stores (−x, −y, z)). Also fixed: skel_corpus.py --t2 alone silently ran nothing (the --parse flag rides on the T1 invocation), and the pipeline_max_skel_corpus ctest now gates T3 accuracy (--gate-t3: biped size-match 100%, drot exact ≥ 97%, dpos exact+close ≥ 72%, non-biped bit-exact) instead of being informational-only.

Corpus T3 after this round: biped drot 11058+3/11120 exact+close (99.4% exact, err 0.0033) — from 8368+413 (75.3%) before; dpos 4408+4101/11120 exact+close (err 0.0262) from 4386+4053 (0.0401); byte-match 69.0→75.6%; size-match 169/169; T1/T2 stay 169/169 + 9/9; non-biped untouched (222/222 bit-exact both). Remaining drot: the prs:name marker class (28, markers not COM-parented — pre-existing), 13 L finger bases on asymmetric-based rigs, and ~1 outlier file per role (genuinely divergent/era).

Float-level field validation (2026-07-06, follow-up). skel_corpus.py now parses every stored .skel field and compares at float level, not just via whole-file bytes: strict bit-exact counters for dpos/drot (drot double-cover aware)/scale/lod/aux fields, father + UnheritScale equality, and a per-element ULP histogram for InvBindPos (state bits compared first). This immediately surfaced a real defect the aggregate byte-match percentage had hidden: UnheritScale was wrong on 862 corpus bones — the reference exporter's getNELUnHeritFatherScale sets it when the parent is a biped node, so plain PRS bones hanging off biped bones (Footsteps, the name/cheveux/weapon-box markers) unherit too, not only biped bones themselves. Fixed; all field classes now 100% (father, unherit, lod, aux, InvBindPos state bits) and gated in ctest. Calibration facts the numbers established: the epsilon-"exact" dpos/drot counts are NOT bit-exact — only 289/398 of 11120 biped (47/32 of 222 non-biped) bones are bit-identical, because the reference exporter re-derived values through Max's matrix decompose, which injects low-ULP noise even where our inputs are the stored controller values verbatim; the reference dscale on _big/_small rigs carries the old uniform figure scale (~1.034 — reference-era class, max file authoritative), and the rest of the scale deltas are ±1–2 ULP decompose noise around 1.0. The biped InvBindPos ULP histogram (64% of elements >256 ULP) is dominated by near-zero matrix cells where tiny absolute noise is a huge ULP distance — read it together with the byte-match percentage, not alone.

Era-divergence warning — verified the hard way, don't re-try: the toe/finger base matrices do NOT reliably share their encoding between the two halves — decoding each side from its own half regressed 250 drot bones corpus-wide; the shipped code parses both halves but decodes only the right one (left = direct, right = LR mirror), keeping only the pose-block deltas per-side. The Max 9 dataset cannot discriminate this (its symmetric baselines make both schemes coincide).

Encode-direction cross-validation (2026-07-06, pending run). pipeline_max_export_skel --maxscript <out.ms> emits a Max 9 MAXScript that regenerates the file's biped from the decoded reconstruction (biped.createNew with structure detected from the records — height = 0x000c/0.113253, ankleAttach = 0x000f[8]/([8]+[9]), link counts from the id/link table — then two biped.setTransform passes forcing every bone's figure-mode world transform). gen_biped_regen.py assembles the whole-corpus script (147 rigs; the 22 two-biped kitin rigs are skipped in v1). Kaetemi runs it in Max 9 (units = meters): the output manifest.txt (differential-dataset format) validates decode→Max acceptance, and the regenerated .max files' 0x9155 records diff against the originals to confirm each record's decode bidirectionally — also yielding a fresh-format (0x0115 == 0) corpus with known-good legacy twins for settling the L-half base-matrix encoding and the remaining open slots.

The figure-version marker 0x0115 (2026-07-06). The "era" here is per-FILE, not per-plugin, and it is detectable: chunk 0x0115 on the 0x9155 system object is int 3 on 168/169 corpus rigs (authored in Max 3, upgraded to Max 9) and int 0 on figures created fresh in Max 9+ (all 139 differential-dataset files — and exactly one corpus rig, tr_mo_kitin_queen, which is also every "single outlier file per role" in the error tables: the R-arm drot outliers, the 1.44 m finger bases, the Bip02 residues). The plugin build id pair in 0x0107 corroborates ((15780518, 12779458) on every corpus file). On the queen, the half-staleness is even per-record — its ×24-scaled finger bases are current only in the LEFT half while its toe bases are current only in the right — so no selection rule is derivable from one file, and the exporter keeps the legacy right-half read (the flag is parsed and documented as the gate point, SBipedRig::FigureVersion). Reference-data caveat (Kaetemi): the kitin queen was likely fixed up manually at some point after the reference exports (which would be exactly what rewrote its figure records in the fresh format and set 0x0115 = 0), and some kitin-queen assets crash 3ds Max, so its reference .skel may additionally come from a broken export. Either way: every queen mismatch is the reference-era class (max file authoritative), not decode signal. When fresh-format (0x0115 == 0) files matter for real (newly authored content through the headless pipeline), re-derive the fresh conventions from the differential dataset plus purpose-made asymmetric variants rather than from the queen.

Remaining biped work:

  1. dpos residues (err 0.0262): the reference-era mismatch class (_big/_small, humanoid re-proportioning), the ~30 moved-footsteps rigs (ground not stored), figure-scaled tail bases (kitin_queen ×24, kamique), and the ~0.5–5 mm chain-position residues (tail.0/spine links/pony1.0/neck.0/finger bases — suspected per-limb figure scale factor; the dataset's sc_* scale variants localize candidate slots: leg scale at 0x000f[31]-region + 0x01f9, per-chain length tables at 0x01f4..0x01fd).
  2. L-half base matrices for fingers/toes on asymmetric rigs (corpus-era encoding unknown — see the era-divergence warning; 13 L finger bases still off).
  3. prs:name markers not COM-parented (~28).

For the animation-export phase (future): the SDK reference documents the biped key layouts (IBipedKey TCB params; IBipedVertKey.z/IBipedHorzKey.x,y/IBipedTurnKey.q; IBipedBodyKey IK pivots; footstep keys with Matrix3 + duration) and the non-uniform-scale export protocol (RemoveNonUniformScale — biped node TMs carry non-uniform scale pre-2.1, object-offset post-2.1; removal corrupts Physique export so it must only be active for motion export). The Physique export interface (IPhyRigidVertex/IPhyBlendedRigidVertex, offset vectors in bone-local space) is the reference for the .shape skinning decode.

Corpus status after the full session (2026-07-05): T1/T2 169/169 + 9/9; biped size-match 169/169, dpos 4386+4053/11120 exact+close (err 0.0401), drot 8368+413/11120 (err 0.1037), byte-match 69.0%; non-biped bit-exact on all dpos and drot (222/222 each), byte-match 90.9% (rest is InvBindPos float noise). The pipeline_max_skel_corpus ctest gates T1/T2 and reports T3 over the exact SkelSourceDirectories workspace listing.

10b. Animation controllers — typed key tables (2026-07-06, anim-export session)

The keyframe animation controllers are typed scene classes now (builtin/control_keyframer.{h,cpp}, registered in CBuiltin::registerClasses): CControlPosLinear (0x2002), CControlRotLinear (0x2003), CControlScaleLinear (0x2004), CControlFloatBezier (0x2007), CControlPosBezier (0x2008), CControlScaleBezier (0x2010), CControlPosTCB (0x442312), CControlRotTCB (0x442313), CControlScaleTCB (0x442315, added 2026-07-06). A corpus-wide survey over all 4523 anim sources (2026-07-06) confirms these nine cover every controller class appearing in an exportable role (PRS refs 0/1/2, LookAt refs 1/2/3, morph-factor pblock ref 0) — no Linear/TCB Float, Bezier Rotation/Point3 or list controllers carry keys in those roles anywhere in the corpus. A shared base (CControlKeyFramerBase) claims the controller's known chunks head-first in file order and stops at the first unrecognized id (so unknowns stay orphaned pass-through); all claimed chunks are re-emitted verbatim in original order, with the key table, default value and range additionally exposed through typed read accessors. Raw bytes stay authoritative — no authoring direction — so T2 byte-identity holds by construction. Gate status at landing: skel corpus T1/T2 169/169 + 9/9 unchanged (T3 numbers bit-identical to the pre-change baseline), all 75 anim-source targets pass T1+T2, 800-file random sample of the wide corpus clean.

Common chunks on every keyframe controller, in file order: default value (0x2501 float / 0x2503 pos CVector / 0x2504 rot CQuat — stored in the INVERSE convention, see §10 "PRS-path defects" / 0x2505 scale CVector+CQuat 28B), 0x2500 (8B, unknown, zeros), 0x3002 (int, usually 0), 0x3003 = time range (Max Interval, 2×sint32 ticks; sentinel 0x80000000 pairs = NEVER/FOREVER), 0x2532/0x2533/0x2534 (containers, unknown), key table (per-class, below), 0x3005 (int, 390 in the corpus).

Key-table layouts (little-endian dwords; sizes verified to divide every instance across the 75 anim-source files and the corpus samples; ticks = 1/4800 s, 160/frame):

Controller Key chunk Bytes/key Layout
Linear Position 0x2513 20 time, flags, val[3]
Linear Rotation 0x2514 24 time, flags, ABSOLUTE quat x,y,z,w (NeL export negates w)
Linear Scale 0x2515 36 time, flags, s[3], q[4] (Max ScaleValue)
Bezier Float 0x2525 28 time, flags, val, intan, outtan, cache[2]
Bezier Position 0x2526 80 time, flags, val[3], intan[3], outtan[3], cache[9] (in/out lengths ⅓ defaults, −1 sentinels)
Bezier Scale 0x2528 148 time, flags, s[3], q[4], intan[3], outtan[3], cache[22] (tangent offsets provisional — all corpus instances carry zero tangents)
TCB Position 0x2521 64 time, flags, val[3], tens, cont, bias, easeIn, easeOut, cache[6]
TCB Rotation 0x2522 92 time, flags, cumulative ABSOLUTE quat[4], tens/cont/bias/easeIn/easeOut, RELATIVE angle-axis (axis[3] world-space + angle — what Max GetKey returns and the NeL exporter consumes, angle negated), cache[8]
TCB Scale 0x2523 112 time, flags, s[3], q[4] (Max ScaleValue), tens/cont/bias/easeIn/easeOut, cache[14] — decoded 2026-07-06 off the weapon-box scale tracks (fy_hof_a_stun_end Ma_Epee2M et al), 1711 keyed instances corpus-wide

Bezier key flags: tangent types at bits 7..9 (in) / 10..12 (out); out-type 2 = step (CKeyBezier.Step). Decode provenance: located by matching the stored floats bit-for-bit against direct pre-optimizer reference .anim exports (~/characters/anim_export/ge_mission_eolienne_tr_idle.anim for Bezier pos + TCB rot including the −0.0 tangent signs, ge_mission_borne_teleport_kami_idle.anim for Linear rot). Open: the controller ORT (out-of-range type → NeL _LoopMode) storage bit is not yet located — no anim in the non-biped target corpus has loop mode set, so nothing validates a decode yet (candidates: 0x2500, 0x3002).

pipeline_max_export_anim (2026-07-06, same session). nel/tools/3d/pipeline_max_export_anim/main.cpp replicates NelExportAnimation (build_gamedata processes/anim, scene=false) for non-biped rigs, building real NL3D CAnimation/CTrackKeyFramer* objects (links NeL::3d, registerSerial3d) so the output format is exact by construction. Key facts established:

  • Node selection: the anim maxscript selects $Bip01 plus nodes with NEL3D_APPDATA_EXPORT_NODE_ANIMATION == "1". The database convention for non-biped animation rigs is a plain Box literally named "Bip01" as the animated root (every one of the 71 non-biped anim sources) — the AppData flags in the current database are all "0", so $Bip01 is the only selection that reproduces the references. Each selected node exports its own tracks under bare names (scale/rotquat/pos — that insertion order), descendants under flat <nodeName>.<value> names (recursion passes the original prefix down, not nested); first-wins on name collisions; tracks only for supported keyframer sub-controllers with ≥ 1 key on the PRS transform (0x2005) refs 0/1/2.
  • Key conversions (mirroring plugin_max/nel_mesh_lib/export_anim.cpp): time = ticks/4800; Linear rot w negated; TCB rot = relative angle-axis with angle negated, axis renormalized with a double-precision norm (the stored axis is a cache accurate to 1-2 ULP; Max renormalizes on GetKey — surfaced by de/la_sky_dome); Bezier Point3 tangents ×4800, Bezier scale tangents unscaled (reference-exporter inconsistency, reproduced); scale values via the Max Inverse(srtm)*stm*srtm diagonal, which in column convention is srtm·stm·srtm⁻¹ (validated to ~1e-6 against animated non-identity-q scale keys; the other order is off ~3e-4); range from 0x3003 with NEVER/FOREVER falling back to first/last key.
  • Stale-cache TCB keys: key flag 0x10 marks the stored relative angle-axis as invalid (garbage axis + zero angle observed on single-key tracks in pr_mo_phytopsy_attack); Max rederives from the absolute quat — the evaluated result is the stored absolute quat's conjugate bit-exactly, reproduced through axis=xyz/‖xyz‖ (double), angle=2·asin(‖xyz‖) to within 1 ULP. Only key-0 derivation is known (= relative to identity); a flagged non-first key would warn (none exist in the corpus).
  • Validation status: 10/10 direct-reference files byte-identical (4 sky domes vs core4_data/sky, eolienne + borne_teleport + 4 karavan kites vs ~/characters/anim_export); 61/61 fauna (plante_carnivore) via in-tree anim_builder + fauna cfg vs core4_data/fauna_animations: 1 byte-identical, 60 within the optimizer-threshold tolerance (worst reconstructed delta 0.0017; the reference builder ran on 2004-era x87 codegen, so borderline key-drop decisions flip freely — whole drop patterns diverge on slow tracks while the reconstructed animation stays equivalent). The 4 ge_mission_kite_kamique_* files carry real bipeds and are deferred to the biped anim phase (exporter refuses with exit 2).
  • Harness: pipeline_max_corpus_test/anim_corpus.py — T1/T2 over the anim corpus, T3 export with the two-tier reference comparison (direct = byte-identity required; optimized = anim_builder pass + reconstructed-animation comparison with double-cover-aware quat deltas, tolerances derived from the optimizer's own thresholds: quat 0.002, vector 5e-4). Wired into ctest as pipeline_max_anim_corpus (--all --gate-t3 --nonbiped-only, self-skips without the private checkouts). Full-corpus T1/T2 sweep over all 4524 anim sources: clean (one-time validation after the typed controller classes landed).

Remaining anim work: biped anim export (§10c), NoteTrack/SSS, morph, camera, particle-system tracks (§10d — all landed 2026-07-06); still open: the ORT/loop bit (no corpus anim sets loop mode, nothing to validate a decode against) and the CS IK in-between solve (§10c). NEL3D_APPDATA_EXPORT_ANIMATION_PREFIXE_NAME/instance-name prefixes remain implemented-untested (no corpus file sets the appdata; the "trigger_right." prefixes on the PSTrigger files come through this path and match, which is the first real signal it works).

10c. Biped animation keys — decode + headless export (2026-07-06, biped-anim session)

pipeline_max_export_anim handles biped rigs since this session: the figure rig is reconstructed through the new shared pipeline_max_rig library (extracted verbatim from pipeline_max_export_skel/main.cpp, gated byte-identical on the skel corpus), the animation keytracks are decoded off the Biped (0x9155) system object, and every biped node is oversampled once per frame across the union key range into CTrackKeyFramerLinearQuat/LinearVector tracks — replicating CExportNel::addBipedNodeTracks + overSampleBipedAnimation (NL3D_BIPED_OVERSAMPLING=30 at 30 fps ≡ one sample per frame, step 160 ticks, last sample clamped to the range end). Pos tracks are emitted for the COM and the biped.getNode(#larm/#rarm/#spine/#tail) first links (clavicles, spine base, tail base) = the reference's mustExportBipedBonePos set; rot tracks for every biped node; no scale tracks. Track names are flat (parentName + nodeName + ".", bare for the selected root), first-wins on collisions (the L-side nub dummies share their base bones' names and lose). Non-biped children inside the biped subtree (weapon boxes, Footsteps, markers) ride the existing PRS keyframer path.

Range and sampling rules (corrected 2026-07-06, anim-completion session — these closed the 26 key-count mismatches): the reference's BipedRangeMin/Max is the union of the BIPSLAVE nodes' IKeyControl key spans ONLY — the COM (BIPBODY) controller contributes NOTHING (its h/v/t keys are invisible to that enumeration: fy_hof_co_fus_tir's turn/vertical tracks run to tick 13440 while the reference range stops at the limbs' 4800; same shape on the arms-only first-person rigs). Keytracks without a corresponding scene bone still don't count, and pony2 does (fy_hom_cur_heal_hp_end: pony2 to 6400 vs everything else 4800). The oversampling loop places samples at min + k*160 while <= max with NO final clamped sample (the reference loop's inner std::min clamp is dead code — fy_hof_co_a1md_attente2's range ends at tick 5638 and the reference's last sample sits at 5600). Every biped track's unlockRange is the global sampled span — the reference's per-controller GetTimeRange is FOREVER/NEVER on biped controllers so every track falls back to its first/last sampled key; the per-limb spans previously used here were wrong (range == key span on every direct reference). Track names are bare only when the selected node's parent is the scene root (CNelExport::exportAnim: GetParentNode() == GetRootNode()); fy_hof_monture_aquatique_demitour_gauche_attack's Bip01 hangs under a stick_1 node and gets "Bip01."-prefixed tracks.

Keytrack storage — chunk pairs (data, time) on the 0x9155 object, decoded against direct-reference .anim exports (~/characters/anim_export) via a conjugation solver (find constant quats A,B with target = A·g(s)·B per candidate reference frame; crisp constants = decode, non-crisp = wrong frame):

Chunks Track Data layout (per key)
0x012c/d horizontal hdr 1; rec 10: COM position Y-up (x, y↑, z↑) at [0..2] (drives world x,y)
0x012e/f turn hdr 3; rec 13: [1] signed angle, quat Y-up Max-conv at [4..7]
0x0130/1 vertical TWO banks (count + count×13 each); rec: z at [2] (drives world z)
0x0132/3 pelvis hdr 3; rec 7: quat [0..3]
0x0134/5, 0x0136/7 R arm, L arm hdr 4; rec 110 (right track first, like the pose-record halves)
0x0138/9, 0x013a/b R leg, L leg hdr 4; rec 110 (+2 floats head-shift per leg link beyond 3, same as 0x0069)
0x013c/d spine hdr 4; rec = 1+n: [0]=n (int), n angle floats (3 per link)
0x013e/f head hdr 3; rec 12: head quat [0..3], zeros, [7]=count, neck angles [8..]
0x0142/3 tail like spine; the TIME recs are 26 dwords (per-link TCB sets), times still at rec[0], track TCB at rec[7..9]
0x0147/8 pony1 like spine (2-link)
0x0149/a pony2 like pony1 (maps to BID_PONY2 = 18; decoded 2026-07-06 — extends the sampled range on 26 character files and drives the Ponytail2 chain)

Time chunk: hdr 7 dwords (count, ?, ?, ?, ?, trackType≈nLinks, ?), then count × 10 dwords (time_ticks, index, p0..p4, tens, cont, bias) — TCB in Max UI units (25 = default → internal (v−25)/25). Limb record fields: [0] hinge angle (elbow/knee; local = Rz(a−π); horse-ankle at [1], Rz(a)), [1]=0.3 ballistic const, [2..5] upper quat, arms [9]/[10] clavicle (Δa, Δb) added to the figure arm-struct values (rule Rx(−a)·Rz(−ε)·B(b) off the last spine link, R = LR mirror), [11] pivot int, [12] IK blend (float 0/1), [14..16]+1 end-effector pos body(COM-rel Y-up), [18..20]+1 end-effector pos WORLD Y-up (the ankle/wrist — exact vs ground truth), [22..24]+0 unit vector (unidentified; ≈knee-plane-normal-ish), [28..31] foot/hand quat, [46+10k..49+10k] finger/toe base delta quat + [54+10k],[55+10k] absolute bend angles (Rz). The [50+10k..53+10k] and [59+10k]-family windows are caches — they do NOT drive the pose (same stored local across files with different cache values proved it).

Conversions (all corpus-validated to float noise at key times; standard-convention quats, C = Rx(+90°) Y-up basis):

  • COM rot = C·conj(turn)·C⁻¹; COM pos = (h.x, −h.z↑, vertical.z).
  • pelvis world = COM(t)·Rx(π)·conj(s)·(−.5,−.5,.5,−.5); spine base = COM(t)·(.5,.5,.5,−.5)-class figure attach·R(a₀(t)) — attach folded as C_fig = figComRot⁻¹·figSpineWorld·R(a₀_fig)⁻¹; spine/tail/pony/neck links: local = C_fig·R(aₖ(t)), R(a) = Rx(a3)·Rz(−a1)·Ry(a2) (same as figure); tail base COM-relative like spine; neck angles ride in the head track; head world = COM(t)·C·conj(s)·(−√½,−√½,0,0).
  • upper arm = COM(t)·(.5,.5,−.5,.5)·conj(s)·Rx(π) (L: stored is LR-mirrored); thigh = COM(t)·(.5,.5,.5,−.5)·conj(s)·Ry(π) (R mirrored — note arms are right-natural, legs left-natural); foot = COM(t)·C·conj(s) both sides direct; hand local = conj(s)·Rx(∓π/2) off the forearm (R: mirror the stored, Rx(+π/2); NO ε in the anim keys unlike the figure default).
  • finger base local = M_own-half·conj(s), R side mirror-wrapped (the anim deltas reference their OWN half's arm-record matrices — unlike the figure decode's right-half-for-both; residual ~3e-4 ε-class). Toe base = M_right-half·conj(s) both sides, L mirror-wrapped (the L half's toe matrices use another basis — same era caveat as the figure). Bends are absolute Rz(angle), angles unwrapped mod 2π against the previous key (stored values wrap: 0 → 6.22 → 0 across keys).
  • Positions: spine/tail base attach rigid in the COM frame, pelvis offset rigid in the pelvis frame, clavicle offset rigid in the last-spine-link frame — all folded from the figure walk's world transforms; everything else parent-relative with figure-static local offsets.

Interpolation — per-channel TCB in STORED space, then per-frame conversion with the time-varying frames. Scalar/vector: Max TCB (the NeL track_tcb.h formulas) except the boundary tangents, where the biped uses the plain difference: tanFrom(first) = (1−t)(v₁−v₀), tanTo(last) = (1−t)(vlast−vprev) (solved from reference in-betweens; NeL/Max's 3-point boundary formula is measurably wrong here). Quats: squad on the raw stored key quats (chained makeClosest, lnDif/exp A-B control points, same factor math), converted after evaluation — validated exact (1e-7) against reference in-betweens on FK intervals, including the time-varying COM composition.

IK — the open work. Keys with [12] = 1 mark IK intervals (planted feet/pinned hands). At key times the stored channels are the SOLVED pose (exact); between keys Character Studio re-solves: the ball-of-foot stays world-planted through pivot rolls (verified: toe-attach point identical across a roll's frames), the ankle path is NOT the world-TCB of the stored ankle positions (0.005 residual), the foot rotation is NOT a squad of the stored channel (≈0.03), the knee angle is NOT its channel's TCB, and α = 1→0 intervals evaluate pure-FK while 0→1 and 1→1 intervals don't. A minimal-rotation-from-key model following the ankle (quat_between on ball→ankle vectors) reduces the foot error ~4× but nothing closes it; the exporter currently ships pure channel-FK for the in-betweens (an experimental 2-bone align+law-of-cosines solve exists behind PMB_BIPED_IK=1 but corrupts at-key exactness through position-model noise and is off by default). Consequence: biped T3 vs direct references is structural + bounded-error, not byte-identical — per-file worst key delta median ≈0.08 rad-class on the character corpus, concentrated 100% in IK-interval in-between frames (everything else is float-noise exact). Next step when this matters: a Max 9 differential ANIM dataset (the biped_dataset_gen.ms channel) with controlled single-effect IK cases to pin the in-between solver exactly.

Stale-cache TCB rot keys, negative-w case (found via box_arme_gauche in ca_hof_mort): when the flagged (0x10) absolute quat has w < 0, Max normalizes to positive w before rederiving — axis = −xyz/‖·‖, angle = 2·acos(−w) (the w ≥ 0 path keeps the previously validated 2·asin(‖xyz‖) form).

Stale-cache TCB rot keys, third data point (2026-07-06 — representation-only, era-drifted, don't re-litigate): the fy_hof/fy_hom_first_* camera-rig dummies (Dummy06/Dummy15, 10 keys over 5 rigs) carry stale near-identity keys where the reference's rederived AXIS is the conjugate of ours ((axis, angle) vs (−axis, −angle) — the identical rotation) and the angle differs by exactly 1 ULP; fy_hof*_marche's Ma_Wea_Epee1M stale keys (w < 0, ‖q‖ drifted past 1) differ era-class too. Both classes prove the CURRENT stored quats are not the bits the reference exporter read (the reference's exact-2^-23 angle needs an exact-2^-24 component; the stored one is 1 ULP off), so the axis-sign question for the w ≥ 0 path is corpus-undecidable and the validated formula stays. The harness's TCBQuat comparison is double-cover aware and treats rotations with |angle| < 1e-5 as axis-degenerate (identity class) — these keys compare on angle and TCB params only.

Validationanim_corpus.py now runs T3 over biped files too: direct refs (characters, kites) compare at float level (track sets + key counts must match — gated; per-key worst delta reported against --biped-tol, informational), fauna/characters optimized refs via anim_builder + the reconstructed-animation tolerance. The pipeline_max_anim_corpus ctest gates structural correctness on bipeds and byte-identity/optimizer-tolerance on non-bipeds (the --nonbiped-only restriction is gone).

Full-corpus sweep, 2026-07-06 (4524 anim sources: 4452 biped + 71 non-biped + 1 stub): non-biped unchanged green (10/10 direct byte-identical, 61 optimized: 1 identical + 60 within optimizer tolerance). Biped: 3064/3173 direct-ref files structurally exact (same track sets, key counts and ranges; per-file worst key delta median 0.147 — the error is concentrated in IK-interval in-between frames, everything else float-noise); the 109 structural fails decompose into feature gaps, all identified: morph MorphFactor tracks (emo anims), camera pos/roll/scale + PSTrigger (first-person/cutscene files), spawn_script SSS tracks (craft anims), and range/key-count mismatches where the reference's per-node key enumeration sees keys outside the 0x9155 keytracks (suspected Biped SubAnim float keys) or ignores keytracks selectively. Fauna biped optimized refs: 1/1279 within the strict optimizer tolerance — the IK approximation exceeds the 0.002 quat threshold broadly, so this bucket stays informational until the in-between solver is exact. Two fixes landed after the sweep baseline (numbers above predate them): the clavicle attach position now follows the last spine link during animation, and the sampled range unions only the keytracks whose bones exist in the scene (fixes arms-only first-person rigs).

10d. Non-transform anim tracks — NoteTrack/SSS, morph, camera, particle systems (2026-07-06, anim-completion session)

The remaining structural gaps of the anim exporter (the 109 direct-ref track-set/key-count fails of the §10c sweep, decomposed: 36 spawn_script + 30 weapon-scale + 26 key-count + 7 morph + 6 camera + 3 PSTrigger + 1 root-flag) are all closed. The storage decodes, each corpus-validated:

  • Note tracks (chunk 0x2140 on the Animatable). Max stores note tracks in the 0x2140 chunk CAnimatable has always preserved verbatim (now exposed read-only via CAnimatable::noteTracks(); the member is also initialized in the constructor now — it previously stayed uninitialized until parse): 0x0130 = note-track count (uint32, always 1 in the corpus; key-to-track assignment is unmarked in storage so >1 warns), then per note key 0x0100 = time ticks + 0x0110 = key flags (4 observed) + 0x0120 = UTF-16 note string.
  • SkeletonSpawnScript (36 craft anims). Nodes flagged NEL3D_APPDATA_EXPORT_SSS_TRACK contribute their note-track scripts (lspawn/wspawn/lkill/wkill <shape> lines); the CSSSBuild port compiles them into the spawn_script ConstString state-track plus the CAnimation::addSSSShape set exactly like the reference (shape lowercased + .shape appended, one state key per distinct note time, empty key at 0 when none lands there). The harness now parses and compares the animation header's SSS shape set and min-end-time as part of the direct-ref verdict.
  • Morpher (7 emo anims). The Morpher modifier (Morpher.dlm, ClassId (0x17bb6854, 0xa5cba2a3)) on the node's OSM Derived object: references 1..100 = per-channel ParamBlocks (ref 0 = the modifier's own), references 101+i = channel i's target node. The channel's factor controller is its ParamBlock's reference 0 (Bezier Float, percent values). Track name = <parentName><targetNodeName>MorphFactor, channels with a target node and ≥1 key only. The emo files' references are now byte-identical (the whole file carries nothing but morph tracks).
  • Cameras (6 first-person anims). The LookAt TM controller (0x2006): reference 0 = the target NODE, 1 = position, 2 = roll (Bezier Float), 3 = scale; GetRotationController is NULL (no rotquat track for LookAt nodes). Roll (roll) and the target node's position controller (target) export only for camera nodes (object superclass 0x20), replicating isCamera.
  • NeL particle systems (3 gun-shot anims). The scripted "Particle Sys" plugin object (ClassId PartA 0x58ce2893 — scripted-plugin PartB varies per script edit, match PartA only; definition in plugin_max/scripts/startup/nel_ps.ms): ParamBlock2 params 0=ps_file_name, 1..4=PSParam0..3 (float), 5=PSTrigger (bool). The PB2's 0x000e param records are [u16 param-id][u16 type][10 bytes][flag byte(+payload)]: flag bit 0x40 = constant value follows (no controller); records WITHOUT it are controller-backed and own the PB2's reference slots in record order (single-controller case validated; no multi-controller PS exists in the corpus). PSTrigger's controller is the On/Off bool controller (PartA 0x984b8d27): chunks 0x0130 = key count, per key 0x0100 = time + 0x0110 = flags, 0x0140 = boundary state; the value toggles at each key with 0x0140 read as the state before the first key (the corpus only carries even key counts, which cannot discriminate before/after — an odd-count file would). → ConstBool track. Whitelisted normalization: the reference's 0-key On/Off track serializes uninitialized stack floats as its range (2004-era exporter bug); ours writes [0,0].
  • TCB Scale typed controller (30 weapon-box scale tracks). See §10b — ClassId 0x442315, key chunk 0x2523.

Deliberately NOT implemented (zero corpus signal, rule §12.2): camera FOV (addObjTracks — the pblock controllers exist but never carry keys), material/texture tracks (no NEL3D_APPDATA_EXPORT_ANIMATED_MATERIALS set anywhere), light tracks (no LightmapController reference tracks), PSParam0-3 (controller-backed instances never keyed), and biped COMs nested under selected non-biped nodes (the reference's addBoneTracks re-enters the biped path there; no corpus file has that shape).

Corpus sweep after this round (full 4524 anim sources, --all --gate-t3 green, 2026-07-06): T1/T2 4452/4452 biped + 71/71 non-biped (roundtrip coherency including the new typed TCB Scale class over the whole corpus); T3 direct-ref structural fails 109 → 0 over the 3173 biped direct refs (7 byte-identical — the emo files, which carry nothing but morph tracks — + 3166 structural; worst-key-delta median 0.1356, concentrated in the IK-interval in-between class, 2150 files over the 0.25 informational tolerance); non-biped unchanged green (10/10 direct byte-identical, 1+60/61 fauna within optimizer tolerance); biped optimized refs 11/1279 within the strict tolerance (informational until the IK solve is exact). Newly identified informational outlier class: ship/cutscene rigs with a keyless vertical channel (ship_tank_karavan_mort_idle, COM z off by a constant ~4.7 m): the reference's COM height appears nowhere in the .max — Character Studio computes it at export time (dynamics/balance solve on the keyless channel), the same runtime-solver class as the IK in-betweens; expect the Max 9 differential anim dataset to pin both.

11. Known defects and hazards (verified in source; fix or guard before building on the affected path)

Fixed (2026-07-05, biped roundtrip-coherency session — these two are why T2 jumped from 2/169 to 168/169 on the biped corpus):

  • CAppData::build iterated m_Entries (a std::map<TKey, CAppDataEntry*>), silently re-sorting AppData entries by (ClassId,SuperClassId,SubId) on every rebuild (builtin/storage/app_data.h/.cpp). The Max exporter does not insert AppData entries in that sorted order (real files interleave entries in whatever order the plugin created them), so any file whose entries weren't already sorted got its AppData chunk reordered — byte-identical in aggregate content, wrong in chunk sequence. This was the single highest-impact defect found this session: biped rigs carry dozens of per-bone AppData entries, so nearly every biped Node object hit it. Fixed by adding m_EntryOrder (a std::vector<CAppDataEntry*> tracking parse/creation order) that build() iterates instead of the map; m_Entries remains the map used for keyed lookup (get/getOrCreate/erase), now kept in sync with m_EntryOrder at every mutation point.
  • CSceneClassContainer::createChunkById ignored the container bool (the file's own container-bit for that chunk) and always returned a CSceneClass (scene.cpp) — every CSceneClass is a CStorageContainer, whose isContainer() is hardcoded true, so a scene-object slot that the source file stored as a literal 0-byte leaf (container bit unset — observed on scattered class instances that carry no parseable data at all) got its container bit silently flipped to true on rebuild. Undetectable at parse time (both interpretations consume 0 bytes, so no exception fires) and invisible in per-object content diffs (an empty container and an empty leaf both serialize to 0 bytes) — only a whole-stream byte comparison catches it, which is exactly why it survived until the T2 corpus gate. Fixed generally, not just for scene objects: IStorageObject gained a virtual bool writeAsContainer() const (default { return isContainer(); }), and CStorageContainer::serial(CStorageChunks&)'s write loop now calls writeAsContainer() instead of isContainer() for the wrapper's container bit. CSceneClass overrides it via a new m_ReadAsLeaf flag, set by CSceneClassContainer::createChunkById whenever the file's container param is false, and exposed for other classes that might need the same escape hatch via setReadAsLeaf().

Fixed (2026-07-05, in the same session that stood up the pipeline_max native build):

  • CStorageValue<std::string>/<ucstring>::getSize returned size as bool, never wrote the out-parameter (storage_value.cpp) — now writes size and returns true. Same bug class in CStorageArraySizePre<T>::getSize adding 4 to a bool return (storage_array.h) — now propagates the inner call's true/false. Latent, since chunk sizes are back-patched by seeking (§3); harmless before the fix, still fix-first material for any future getSize-relying code path.
  • CSceneClass::putChunk inserted into m_OrphanedChunks with an iterator borrowed from m_Chunks (scene_class.cpp:271) — now m_Chunks.insert(...). Prior behavior "worked" because std::list splices the node into the iterator's owning list, but left both lists' cached sizes inconsistent, which quietly skewed the disown sanity check m_Chunks.size() < m_OrphanedChunks.size() — reliably surfaces as an "Not built" false alarm during parse→build→disown on real corpus files (observed on ge_mission_eolienne_tr.max, now green after the fix). T2 corpus must still re-verify byte fidelity once the harness is up — the byte-writing path was already correct via the splice; the size counters were the load-bearing lie.
  • CSceneClass::getChunkValue nlerror("… 0x%x …") with no format argument (scene_class.h:169) — passes (uint32)id now.
  • CTrackViewNode::parse copy-paste on 0x0150 chunk check (if (m_Empty0150) nlassert(m_Empty0140->Value.empty())) — now derefs m_Empty0150->Value.empty(). Latent nullptr crash if 0x0150 appears without 0x0140.
  • CStorageRaw::serial + CStorageValue<string>/<ucstring>::serial + CConfigScriptMetaString::serial took &Value[0] on possibly-empty containers — now guarded with if (Value.empty()) return; (or wrapped for the null-terminated case). serialBuffer no-ops on len 0 so it was benign; UB by the letter of the standard.
  • CSceneClassUnknownDesc::create had no return statement after nlassert(false) (scene_class_unknown.cpp:75) — added return NULL;. Path is unreachable (unknowns go through createUnknown on the superclass desc) but a release build would fall off the function end.

Fixed (2026-07-05, biped corpus-completion session):

  • Superclass 0x1190 ("Depth of Field (mental ray)") unregistered, nlerror-aborting the whole process — every *_fp.max first-person-hand file in the corpus carries this superclass (a camera/render effect, irrelevant to skeleton export) and crashed pipeline_max_export_skel outright. Fixed the same way as every other unimplemented superclass observed in the corpus: added CCameraEffectSuperClassDesc (CSuperClassDescUnknown<CReferenceTarget, 0x00001190>) to CBuiltin::registerClasses (builtin.cpp) so it falls through to the standard pass-through unknown instead of erroring. No behavior change for any file that doesn't carry this superclass.
  • Mixed 32-/64-bit chunk headers in the same stream widen everything to the outermost chunk's width on write — the last remaining T1/T2 failure in the skel corpus (tr_mo_estrasson.max) and, it turned out, a ~15% T1/T2 failure rate across the wider ~8.6k-file corpus (measured via two random samples, 400 + 800 files, against a clean-after-fix 0/1200). Root cause was exactly as diagnosed previously: CStorageContainer::m_Was64Bit (added by the earlier "64-bit chunks not writable" fix, below) is a single flag per container, so any container whose own direct children were a genuine mix of 32-bit and 64-bit headers wrote every child at one uniform width. Fixed with per-chunk width tracking: IStorageObject gained a tri-state wasRead64BitChunk()/wasRead64BitChunkKnown() pair (set by CStorageContainer::serial(CStorageChunks&) right after each child chunk is entered on read); the write loop uses the per-object value when known, and falls back to the containing container's own m_Was64Bit aggregate only for chunks a typed class rebuilds from scratch during build() (e.g. CNodeImpl's version/parent/name, which have no "original width" of their own since they're freshly constructed, not the object that was actually read). CStorageChunks::enterChunk(id, container, as64Bit) takes the width as an explicit parameter now instead of reading its own sticky m_Is64Bit flag on write. Verified: skel corpus T1/T2 169/169 (up from 168/169), plus two independent full-corpus samples (400 files seed 42, 800 files seed 7) fully clean where the same population measured ~15% failing before the fix — no regressions found in either sample.

Open (accepted, or need work beyond mechanical fix):

  1. CSceneClassContainer::parse assumes the builtin scene is the last chunk (m_StorageObjectByIndex[size()-1], scene.cpp:153) and hard-asserts the dynamic cast. Holds for the corpus; will fire on files where it doesn't. Acceptable as an assert, but know it's there.
  2. 64-bit chunks not writablenow writable (2026-07-05), refined the same day (see "Fixed" above — the initial per-container flag under-covered mixed-width containers, now per-chunk). CStorageContainer gained a per-container m_Was64Bit flag set on read from CStorageChunks::is64Bit(); the top-level serial(stream, size) (and the gsf-path variant) round-trips it through CStorageChunks::set64Bit() on the write side. CStorageChunks::enterChunk/leaveChunk now emit either a 6-byte 32-bit header or a 14-byte 64-bit header per-chunk (HeaderSize field on CChunk tracks the width for size back-patching). Rule preserved: a file containing any 64-bit chunk header round-trips 64-bit; a file with none stays 32-bit.
  3. Failure = nlerror/nlassert = process abort throughout (several files deliberately #define nlwarning nlerror — "elevate warnings to errors for stricter reading"; keep that convention for new class code, it's what makes silent misparses impossible). The corpus harness, however, must run file-scoped: wrap per-file work so one bad file records a failure and the sweep continues (NeL assert/error behavior is configurable enough to throw instead of abort; that, or a child-process-per-file runner — decide once, at harness level, not by softening the library's checks).
  4. Empty-AppData drop — the one whitelisted normalization (§5). Any future intentional normalization must be added to that whitelist and this list, in the same commit that introduces it.

12. Rules for the implementing session

  1. Corpus-gate every change. T1 green before starting; T2 green over all ~8.6k files after each newly typed chunk, class, or enabled #define. No batch of typing lands on a red corpus.
  2. Type only what you can prove. A chunk id gets a typed mapping only when the typed serializer reproduces every instance in the corpus byte-exactly (that's what T2 verifies). No guessing formats from the Max SDK docs without corpus confirmation; unknown ids keep falling through to raw/container defaults.
  3. Never bypass the lifecycle. New classes implement the full sextet (parse/clean/build/disown/init/toStringLocal + createChunkById), call the parent first (parse/build) or last (disown), guard on m_ChunksOwnsPointers, use getChunk/putChunk/m_ArchivedChunks exactly as §5 describes, and put chunks back in the order they were taken. CEditableMesh is the template for "typed class that mostly passes through"; CNodeImpl for "small fully-typed class"; CAppData for "container with its own entry bookkeeping"; CDllDirectory/CClassDirectory3 for "entry table with position cache" — start from the nearest template, don't invent a fifth shape.
  4. Preserve emission forms. Where the format has alternatives (0x2034 vs 0x2035 references, 32- vs 64-bit headers, present-vs-absent optional chunks), record which form was read and re-emit that form. Byte-identity is the spec; "semantically equivalent" is not.
  5. First milestones, in dependency order: (a) defect list items 1-2 done — see §11 "Fixed"; (b) build the corpus harness out of the dump tool (argv, per-stream compare, per-file isolation, version bucketing); (c) run T1/T2 baseline, inventory unparsed-class and 64-bit/compressed counts; (d) enable PMBS_GEOM_BUFFERS_PARSE and re-gate; (e) claim the noted tri/poly object ids (0x0901-0x0904, 0x0906-0x090c); (f) PARSE_NELDATA: typed AppData decoders per export_appdata.h; (g) materials/param blocks (fill the stubs, register them) as export needs them; (h) node transform controllers (currently unknowns — needed for object placement at export); then the nel_mesh_lib export logic retargeted onto this object model, T3-gated. The lightmapping phase is explicitly out of the exporter: shapes export unmapped, and calc_lm becomes a separate standalone tool validated in aggregate (see T3 caveats).
  6. Keep this document current. New chunk ids, new normalizations, new invariants, fixed defects — same commit that changes the code updates this page and the Tutorial Tech Tree status.

Clone this wiki locally