Skip to content

pipeline_max_design

Jump to bottom
Jan Boon edited this page Jul 5, 2026 · 9 revisions

title: Pipeline Max — Design and Coherency Contract description: Codified design of the headless .max parser/writer (nel/tools/3d/pipeline_max) — the storage model, the parse/build/disown lifecycle, roundtrip invariants, known defects, and the rules any extension work must follow published: true date: 2026-07-05T00:00:00.000Z tags: editor: markdown dateCreated: 2026-07-05T00:00:00.000Z

This document codifies the design of the headless 3ds Max file library at nel/tools/3d/pipeline_max (namespace PIPELINE::MAX, "RYZOM CORE PIPELINE", 2012), together with its driver tools pipeline_max_dump and pipeline_max_rewrite_assets. It exists so that the session extending this library into the full headless export pipeline (see the Tutorial Tech Tree, side-quest 5) preserves the two properties the whole design is built around, rather than re-deriving them halfway and getting them subtly wrong:

  1. Whole-library coherency — every class participates in one shared storage lifecycle with one ownership discipline; a new class that improvises its own variant breaks the guarantees for everything beneath it.
  2. Roundtrip coherency — a file that is read and written back must be byte-identical per stream, whether or not any of it was parsed into typed form, except where a normalization is explicitly documented and whitelisted. Unknown data is never guessed at, never dropped, never reordered.

Everything below was verified against the actual source (every .cpp and .h in the library was read for this document), not reconstructed from memory. File/line references are as of commit d8deff3f2-era develop.

1. Scope and dependencies

  • Reads and writes 3ds Max .max files without 3ds Max. The container is an OLE2 compound document, accessed through libgsf (gsf_infile_msole_* / gsf_outfile_msole_*). NeL's NLMISC::IStream is bridged onto gsf streams by CStorageStream (storage_stream.h/.cpp) — a thin seek/read/write adapter.
  • Target versions: Max 3 through Max 2010. typedefs.h defines the version constants (Version3 = 0x2004 … Version2010 = 0x2012). The corpus in ryzomcore_graphics spans this whole range; the format is essentially stable across them, with per-version variations inside individual classes (e.g. CSceneImpl gains a 12th reference after Max 3).
  • Compressed .max files exist (gsf_input_uncompress is noted in a comment in storage_stream.h) and are not handled. If the corpus contains any, they must be detected and either supported or explicitly skipped with a count — not silently failed.
  • derived_object.*, wsm_derived_object.*, storage_file.* are empty stubs (constructor/destructor only). Placeholders, no design in them.

2. Anatomy of a .max file

An OLE compound document with these streams, all of which must be preserved on rewrite (this is exactly the set pipeline_max_rewrite_assets reads and writes back):

Stream Handling today
Scene CScene — the real content; one version-numbered root chunk containing the scene class container
DllDirectory CDllDirectory — fully parsed (list of plugin DLL description/filename pairs)
ClassDirectory3 CClassDirectory3 — fully parsed (per-class: dll index, ClassId, SuperClassId, display name)
ClassData CClassData — container structure known (entries 0x2100, header 0x2110 with ClassId/SuperClassId); entry body 0x2120 depends on the header and is an open TODO, kept raw
Config CConfig — partially typed (0x20a0 entry table, 0x2180 ConfigScript with its recursive "meta container" value format); many leaf ids known only as unlabeled ints/strings
VideoPostQueue generic CStorageContainer (raw pass-through)
\05SummaryInformation, \05DocumentSummaryInformation raw byte blobs, copied verbatim
OLE class id (16 bytes) read via gsf_infile_msole_get_class_id, must be written back via gsf_outfile_msole_set_class_id

The roundtrip unit is the stream, not the file. libgsf will not reproduce Max's OLE sector layout, so whole-file byte comparison is meaningless; per-stream byte comparison is the invariant. The Scene stream's single top-level chunk id doubles as the file version (CScene::version() asserts exactly one chunk and returns its id).

3. Chunk layer — CStorageChunks

storage_chunks.h/.cpp. Max chunk format:

  • 6-byte header: uint16 id, uint32 size. size includes the header; bit 31 = "container" flag (chunk contains child chunks rather than leaf data).
  • 64-bit extension: size == 0 means a uint64 follows (14-byte header total), same bit-63 container flag. The reader accepts these and downgrades to 32-bit bookkeeping (throws EStream if ≥ 2 GiB); it sets a sticky m_Is64Bit on the stream. The writer refuses to write when m_Is64Bit is set (enterChunk throws). Consequence, codified: a file that contains any 64-bit chunk header can be read but not currently rewritten. The corpus run must count these; if any exist, 64-bit write support (with a "write 64-bit iff read 64-bit" rule to preserve byte-identity) becomes a prerequisite.
  • Write path: enterChunk(id, container) writes the id and a 0xFFFFFFFF size placeholder; leaveChunk() seeks back and patches the real size with the container bit. Sizes are therefore never computed from getSize() — see the defect list (§11) before ever relying on getSize().
  • Read path: leaveChunk() returns the number of bytes skipped (unconsumed) in the chunk. CStorageContainer::serial throws EStorage if this is non-zero (storage_object.cpp:268). This is the load-bearing read-side guarantee: every byte of every chunk must be consumed by whatever object type was instantiated for it. A typed reader that under-reads cannot silently pass; it either matches the format exactly or the file fails loudly.

4. Storage object layer

storage_object.h/.cpp, storage_value.h/.cpp, storage_array.h.

  • IStorageObject (abstract, extends NLMISC::IStreamable): serial(IStream&), toString(ostream&, pad) (a readable dump — diagnostics only, never a test criterion), setSize(sint32) (called before reading a leaf so it knows how much to consume), getSize(sint32&), isContainer().
  • CStorageContainer: an ordered std::list of (uint16 id, IStorageObject*) — order is significant and duplicate ids are normal (that's why it's a list, not a map). Reading: for each child chunk, createChunkById(id, isContainer) manufactures the object, setSize primes it, then it serials itself. Writing: iterate the list in order, wrap each in a chunk.
  • createChunkById is the single extension point for typing a chunk: subclasses switch on the id and return a typed object; the default (and mandatory fallback for every unrecognized id) is CStorageContainer for containers and CStorageRaw for leaves. CStorageRaw is an exact byte vector. This is how unknown data survives byte-perfectly: not by special-casing, but because raw is the default and typing is the exception.
  • Typed leaves: CStorageValue<T> (single value; std::string/ucstring specializations sized by the chunk), CStorageArray<T> (chunk size / sizeof(T) elements), CStorageArraySizePre<T> (uint32 count prefix), CStorageArrayDynSize<T> (count prefix, variable-size elements — used for poly faces).
  • A typed leaf must serialize to exactly the same bytes it consumed. CStorageValue<ucstring> reads/writes raw UTF-16 (size/2 chars); CConfigScriptMetaString shows the pattern for a length-prefixed, null-terminated string. When a format has optional fields, the bitfield discipline in CGeomPolyFaceInfo::serial (geom_buffers.cpp:155) is the model: parse each known bit, clear it, and nlerror on any residue — unknown variants abort rather than desync.

5. The four-phase lifecycle — the roundtrip mechanism

This is the heart of the library and the thing an implementing session must not gloss. Declared on CStorageContainer (storage_object.h:116-124), with the class-level protocol implemented in CSceneClass (scene_class.cpp):

serial(in) → parse(version[, filter]) → [use / modify typed data]
          → clean()                     (optional: drop raw source, forces full rebuild)
          → build(version[, filter])    (reconstruct m_Chunks from typed data)
          → disown()                    (return ownership to m_Chunks, drop typed refs)
          → serial(out)

Ownership is tracked by one flag, m_ChunksOwnsPointers:

  • After serial(in): m_Chunks owns everything; the object is pure storage. true.
  • parse (CSceneClass): parent's parse runs first; then all chunks are moved onto m_OrphanedChunks and the flag flips to false. Subclasses then claim their chunks with getChunk(id), which removes them from the orphan list — in order: taking a chunk that is not at the head of the orphan list logs a warning ("chunks out of order"), because it means the class's read order no longer mirrors the file's chunk order. Chunks the class keeps unmodified are pushed to m_ArchivedChunks (deleted on clean, forgotten on disown). Chunks the class never claims stay orphaned and are re-emitted verbatim by build — that's per-chunk forward compatibility inside otherwise-typed classes (see CEditableMesh::parse, which drains a dozen known-unknown ids in their observed sequence purely to keep them in position).
  • Failure discipline: if a class cannot make sense of its data, it calls disown() and aborts its own parse — the object degrades to raw pass-through for this file, without failing the file. Every subclass parse body is guarded by if (!m_ChunksOwnsPointers) so that a parent's disown short-circuits the children. Codified rule: graceful degradation is per-object, and "degrade" always means "keep the original bytes", never "drop".
  • clean: only legal after a successful parse. Deletes the raw chunks that the typed representation fully supersedes (and the archived ones). After clean, the original bytes are gone — build must regenerate them bit-perfectly from typed state. A class whose build cannot yet do that must not clean (compare CAppData, which cleans its raw entry values because CAppDataEntry::build re-serializes them, with CDllEntry::clean, which deliberately does nothing because m_Chunks retains ownership of the two value chunks it aliases).
  • build: CSceneClass re-inserts the remaining orphans into m_Chunks, then sets an insertion cursor before the orphans; putChunk(id, obj) inserts at the cursor, so class-written chunks precede leftover orphans while both keep their internal order. putChunk calls build on any container passed through it. The putChunkValue<T>/getChunkValue<T> pair handles single-value chunks and archives them automatically.
  • disown: nulls every typed reference, clears orphan/archive bookkeeping, flips the flag back to true, recurses. After disown, the object is again pure storage and can be serialized out. CSceneClass::disown sanity-checks that build actually ran (orphan count vs chunk count) — see the defect list for a caveat on that check.

The null-rewrite property (codify as the primary test): for any file in the corpus, serial(in) → serial(out) unparsed, and serial(in) → parse → build → disown → serial(out) parsed-but-unmodified, must both reproduce every stream byte-identically. The lifecycle above is precisely the machinery that makes the second path equal the first; any new typed class is a new way to break it, which is why the corpus gate (§9) reruns it wholesale.

Known deliberate normalization (whitelist, currently the only one): CAnimatable::build discards an AppData container that has zero entries (animatable.cpp:100-112). A source file carrying an empty AppData chunk will lose it on a parsed rewrite. Keep the whitelist explicit in the harness; do not let new normalizations in without listing them here.

6. Scene layer — directories, registry, unknown classes

  • CScene → CSceneClassContainer (scene.h/.cpp): the Scene stream's root chunk contains one child chunk per scene object. The child chunk's id is the object's index into ClassDirectory3, which in turn names the ClassId/SuperClassId/DLL. Object identity in references is the position (storage index) of the object chunk within this container.
  • Special ids 0x2032 ("OSM Derived", ClassId (0x29263a68, 0x405f22f5)) and 0x2033 ("WSM Derived", (0x4ec13906, 0x5578130e)) — derived-object wrappers that hold modifier stacks — are hardcoded in CSceneClassContainer::createChunkById as unknown-class instantiations rather than directory lookups.
  • CSceneClassRegistry maps (SuperClassId, ClassId) → ISceneClassDesc, and SuperClassId → ISuperClassDesc. Resolution order in createChunkById: (1) exact registered class; (2) createUnknown on the registered superclass — this instantiates CSceneClassUnknown<TSuperClass>, which inherits the typed superclass, so e.g. an unimplemented material still runs CReferenceMaker::parse and gets its reference wiring parsed and rebuilt while its own chunks stay orphaned/verbatim; (3) an invalid CSceneClassUnknown<CSceneClass> fallback. This three-tier scheme is why the library can parse every Ryzom file today with only ~15 typed classes: parse coverage and byte fidelity are decoupled.
  • CBuiltin::registerClasses (builtin.cpp) registers the typed classes and ~40 CSuperClassDescUnknown<CReferenceTarget, sclassid> entries covering every superclass observed in the corpus (controllers, materials, texmaps, param blocks, modifiers, lights, cameras, etc. — all currently routed to reference-target-typed unknowns). CUpdate1::registerClasses adds EditableMesh; CEPoly::registerClasses adds EditablePoly. A new typed class = write the class + add one registry->add line; nothing else changes. Plugin-DLL grouping is mirrored in namespaces (BUILTIN, UPDATE1, EPOLY) with a IDllPluginDesc per DLL (update1.dlo, EPoly.dlo); follow that pattern for new plugin classes (e.g. nelexport.dlu would get its own namespace if it ever needs scene classes — though NeL data itself doesn't; see §8).
  • CDllDirectory / CClassDirectory3 (marked "Parallel to" each other — keep them symmetrical): fully parsed into entry vectors plus a m_ChunkCache that remembers where the entry run sat among any non-entry chunks, so build re-emits entries at the original position. Both throw EStorageParse if entries are interleaved with other chunks (never observed). Internal pseudo-indices: dll index −1 = builtin, −2 = maxscript script. getOrCreateIndex appends new entries when writing new classes into a file. reset() exists to blank the directory for from-scratch writes.
  • Storage-index mapping is currently read-only: CSceneClassContainer::parse assigns indices by position, build's getOrCreateStorageIndex just looks up the frozen map ("Temporary 'readonly' implementation" — all four lifecycle methods). The scene-graph TODO in scene.h ("Evaluate the references tree to build new indexes") is the missing piece for adding/removing objects; until it's done, the pipeline can modify scenes but not restructure them. The export direction (max → NeL) doesn't need it; a TOML-sidecar → .max authoring direction eventually would.

7. Object model — references, nodes, geometry

Typed class hierarchy as implemented (✓ = has real parse/build logic; ✗ = registered but pass-through; stub = source file exists, empty 59-line skeleton, not registered):

CSceneClass
└─ CAnimatable ✓ (AppData 0x2150, unknown 0x2140)
   └─ CReferenceMaker ✓ (references 0x2034/0x2035, 0x204B, 0x2045/0x2047/0x21B0 raw)
      ├─ CSceneImpl ✓ (fixed 11/12 reference slots; the builtin scene singleton)
      └─ CReferenceTarget ✓ (no own chunks)
         ├─ INode ✓ (implicit children set)
         │  ├─ CNodeImpl ✓ (0x09ce version, 0x0960 parent+flags, 0x0962 name)
         │  └─ CRootNode ✓
         ├─ CTrackViewNode ✓ (0x0110/0x0120/0x0130 per child, empties 0x0140/0x0150)
         ├─ CBaseObject ✓ (no own chunks)
         │  └─ CObject → CGeomObject ✓ (0x08fe GeomBuffers, 0x0900)
         │     ├─ CTriObject ✓ (ids 0x0901-0x0904 noted, not yet claimed)
         │     │  └─ UPDATE1::CEditableMesh ✓ (known-unknown sequence kept in order)
         │     ├─ CPolyObject ✓ (ids 0x0906-0x090c noted, not yet claimed)
         │     │  └─ EPOLY::CEditablePoly ✓ (known-unknown sequence kept in order)
         │     └─ CPatchObject → CEditablePatch (✓ registered, pass-through bodies)
         ├─ CModifier, CMtlBase, CMtl, CStdMat, CStdMat2, CMultiMtl,
         │  CTexmap, CBitmapTex, CParamBlock, CParamBlock2   ← stubs (empty, unregistered)
         └─ (everything else: CSceneClassUnknown<CReferenceTarget> via superclass descs)
  • References (CReferenceMaker): two storage forms, mutually exclusive — 0x2034 (flat index array, position = reference slot, −1 = empty) and 0x2035 (leading unknown uint32, then (slot, index) pairs). m_ReferenceMap records which form was parsed so build re-emits the same form; the leading 0x2035 value is preserved. Indices resolve through container()->getByStorageIndex, live pointers are CRefPtr (non-owning, self-nulling — the scene container owns objects). Subclasses override getReference/setReference/nbReferences to give slots semantic names (CSceneImpl's 12 slots; CNodeImpl does not — node parent/name live in node chunks, and slot 1 of a node is its object, per INode::dumpNodes).
  • Version-dependent shape: CSceneImpl::parse asserts 11 references, 12 (m_TrackSetList) only when version > Version3. This is the template for all version forks: assert what the version implies, don't silently accept both.
  • Nodes: parent linkage is stored child→parent (0x0960: parent storage index + flags); setParent maintains the in-memory child sets. INode::find resolves by case-insensitive user name — that is the instance-name hook the TOML sidecar route will use.
  • Geometry (STORAGE::CGeomBuffers, chunk 0x08fe): typed serializers exist for the tri buffers (0x0914 vertices / 0x0912 CGeomTriIndexInfo faces incl. smoothing groups / 0x0916+0x0918 and 0x0938+0x0942 secondary vertex/index pairs) and the poly buffers (0x0100 vertices / 0x010a edges / 0x011a faces with material, smoothing, and triangulation cuts), but the id→type mapping is compiled out: PMBS_GEOM_BUFFERS_PARSE is 0 (geom_buffers.cpp:52), so geometry currently rides through as raw. Enabling it is the first real corpus test of the typed leaf serializers. CGeomObject::triangulatePolyFace (geom_object.cpp:162) converts poly faces + cuts into triangles via an edge-travel matrix, asserting triangles == cuts + 1; pipeline_max_dump::exportObj already produces valid .obj output from an EditablePoly through it. The old map-extender surgery in the rewrite tool shows the tri-object counterpart chunk pairs (0x2500→0x2512→0x03e9/0x03eb on derived objects).
  • Two-pass parse via filters: parse(version) with filter 0 handles the common object chunks; geometry is deferred until after the subclass has drained its own known chunks, then invoked explicitly as CTriObject::parse(version, PMB_TRI_OBJECT_PARSE_FILTER) → CGeomObject::parse(version, PMB_GEOM_OBJECT_PARSE_FILTER). The filter constants are magic per-class tokens (0x432a4da6 etc.), not flags. Separately, TParseLevel (PARSE_INTERNAL, PARSE_BUILTIN, and the commented-out PARSE_NELDATA, PARSE_NEL3D in storage_object.h:59) sketches the intended depth levels — NeL-data interpretation was always planned as a higher parse level on top of builtin parsing. The export pipeline should revive that enum rather than invent a new switch.

8. AppData — where the NeL data lives

STORAGE::CAppData (chunk 0x2150 on any CAnimatable): header 0x0100 = entry count; each entry 0x0110 = key 0x0120 (ClassId, SuperClassId, SubId, Size) + value 0x0130 (raw bytes). Entries live in a map keyed (ClassId, SuperClassId, SubId); parse fails safe (disown) on any structural surprise, and validates count against the header.

CAppDataEntry::value<T>() lazily reinterprets the raw bytes as a typed storage object by round-tripping through a CMemStream (and can re-cast, invalidating the previous pointer — one live typed view per entry). clean empties the raw copy; build re-serializes the typed view back into the raw chunk and fixes the key's Size. getOrCreate + init support authoring new entries.

All NeL export properties are AppData entries, not scene classes. The 3ds Max plugin stores them keyed by MAXSCRIPT_UTILITY_CLASS_ID (0x04d64858, 0x16d1751d) / superclass 4128 with NEL3D_APPDATA_* sub-ids defined in plugin_max/nel_mesh_lib/export_appdata.h (LOD config, accelerator/cluster flags, instance name/group, don't-export, collision, realtime light, water params, vegetable, lightmap settings, …). So the "parse the NeL data" milestone (PARSE_NELDATA) is: typed decoders for those AppData blobs on CAnimatable — zero new scene classes required. The dump tool's commented-out AppData walk (pipeline_max_dump/main.cpp:289) is the reference snippet.

9. Existing tools, and the test harness to grow out of them

  • pipeline_max_dump — single hardcoded file; parses directories + scene, and already performs the canonical smoke sequence: serial in → serial out to temp.bin → serial back in → parse → clean → build → disown → parse → dump, plus root-node tree dump and the .obj export prototype. This is the embryo of the per-file roundtrip check; it needs argv, per-stream byte comparison, and exception-based failure reporting instead of hardcoded paths and nlerror aborts.
  • pipeline_max_rewrite_assets — the 2012 database migration tool and the proof that parse-free structural rewriting works at corpus scale: it read every stream, optionally rewrote texture paths inside raw/string chunks (with a huge accumulated table of database renames — historical, not design), retargeted a modifier ClassId across the class directory, and wrote the OLE file back with the class id preserved. Its config is all hardcoded globals (w:\database\ mapping, WriteDummy, HaltOnIssue, overrideFF for ligo quirks). Treat it as reference code for the OLE read/write envelope (handleFile, main.cpp:1085) and the fix-up patterns; do not extend it in place.
  • Codified test protocol for the extension work (the tutorial-roadmap side-quest's "exhaustive unit test suite"):
    • T1 — structural roundtrip, no parse. Every stream of every corpus file: serial in → serial out → byte-compare per stream. Must pass 100%; failures are chunk-layer bugs (or 64-bit/compressed files, which must be counted and reported, not skipped silently).
    • T2 — parse/build roundtrip. serial → parse → build → disown → serial → byte-compare, whitelisted normalizations only (§5, currently: dropped empty AppData). Rerun over the full corpus after every newly typed chunk or class — this is the non-negotiable gate that keeps typed coverage from eroding fidelity. A clean variant (parse → clean → build) additionally proves typed state fully regenerates the bytes for classes that clean.
    • T3 — export snapshot. Once export logic exists: pipeline output diffed against the prebuilt Ryzom Core data set (~/core4_data) as reference — structural fields exact, floats within epsilon, mismatches triaged (plugin-version differences do exist in the reference data; the triage bucket is part of the design, decided per output type, not ad hoc). Known reference-data caveats, pre-triaged (Kaetemi, 2026-07-05):
      • Flare sizing flag (_FirstFlareKeepSize). Some flare assets carried this flag mis-set in the max sources; an export/build-side hack once flipped it based on file version to make the live data look right, and was later removed because it broke assets that expected the reverse. The prebuilt reference data predates parts of this cleanup, and the max files may since have been fixed. So flare-shape sizing-flag mismatches against the reference are expected: resolve per-asset from the max source (the max file is authoritative), never by teaching the pipeline to reproduce the reference. Note the engine loader still has its own version clamp — CFlareShape::serial forces the flag false for shape versions ≤ 4 (nel/src/3d/flare_shape.cpp:77) — so what the game renders is not always what the shape file stores; compare stored values, and account for the loader clamp when judging runtime-visible differences.
      • Non-standard UVW mapping modifier. Some assets use a custom UVW-mapping extension written by a Ubisoft employee (a non-standard plugin modifier — it announces itself in the file's DllDirectory, so detection is trivial and unambiguous). The reference .shape outputs of those assets contain garbage UVs; UV equality against the reference is meaningless there. The harness must auto-flag these files by DLL/class entry and route them to a dedicated bucket — either the modifier gets implemented properly (making our output better than the reference) or the assets are recorded as excluded-with-reason; silent inclusion in pass/fail statistics is not acceptable. The flagged set must be emitted as an explicit asset list report (file path, node names, modifier DLL/class entry): Kaetemi may be able to locate original exports from older live clients to serve as known-good UV references for these assets. Caveat on that caveat: such old exports may themselves derive from older revisions of the max files, so treat them as corroborating evidence for the UV channel, not as byte-exact snapshot references.
      • Lightmapped scenes. The headless pipeline exports these unmapped; the lightmapping phase (currently calc_lm* inside nel_mesh_lib, run at export time inside Max) is replicated as a separate standalone tool downstream. Consequently T3 compares unmapped exports structurally, and the lightmapper is validated in aggregate — scene-level comparison of its output against reference lightmaps (statistical/image-level tolerance), not per-shape byte equality.
    • Bucket all results per file version (Max 3 … 2010) — regressions cluster by version.
    • toString dumps are diagnostics for triage; never assert on them.

10. Reference: how the original 3ds Max exporter is actually run

The authoritative reference for export sequence, options, and outputs is the build_gamedata pipeline (nel/tools/build_gamedata, Python — Kaetemi's 2010 port), configured by the workspace in the ryzomcore_leveldesign repository (cloned locally at ~/ryzomcore_leveldesign; the workspace is workspace/). The headless pipeline's integration target is to drop in at the 1_export stage with the same workspace contract, replacing the 3ds Max invocation while honoring the same inputs, options, outputs, and tag protocol.

  • Structure: four phases per process — 0_setup.py / 1_export.py / 2_build.py / 3_install.py — under processes/<name>/. Eleven processes are Max-driven (have a maxscript/ directory): shape, anim, ig, ligo, zone, skel, swt, veget, clodbank, pacs_prim, rbank.
  • Workspace config: workspace/projects.py lists projects (common/objects, common/sfx, common/characters, per-continent, per-ecosystem, …). Each project directory holds directories.py (source directories inside the graphics database, export/build directory names, tag directories, lookup paths) and process.py (which processes run, plus per-project export options — e.g. ShapeExportOptExportLighting, ShapeExportOptShadow, ShapeExportOptLumelSize, ShapeExportOptOversampling; BuildQuality == 0 globally downgrades these for fast builds). The headless exporter must consume these same options with these same names.
  • Max invocation (see processes/shape/1_export.py): the Python driver template-substitutes %TOKENS% (source dir, output dirs, options, log path) into maxscript/shape_export.ms, installs it into the Max user scripts directory, and runs 3dsmax.exe -U MAXScript shape_export.ms -q -mi -mip — Max then iterates every .max in the source directory itself. Incrementality and crash-resilience live in the tag protocol: a .max.tag per successfully exported source file (compared by needUpdateDirByTagLog), a max_running.tag sentinel written before each Max launch (still present afterwards ⇒ Max crashed), and a retry loop that measures progress by tag-count delta with a zero-progress retry limit of 3. The headless pipeline keeps the tag files (other stages and the incremental logic depend on them) and can discard the sentinel/retry machinery — that exists purely because Max crashes.
  • Per-node export policy is in the maxscript, driven by AppData: shape_export.ms::isToBeExported skips nodes with NEL3D_APPDATA_DONOTEXPORT, collision flags, etc., and calls the NelExport plugin per eligible node. This logic moves into the headless exporter's own node filter, reading the same AppData keys (§8).
  • Shape export outputs (per project): shape_not_optimized/, shape_with_coarse_mesh/, shape_lightmap_not_optimized/, plus the anim directory — note that today lightmaps are computed at export time inside Max and then post-processed by lightmap_optimizer in 2_build.py. Under the unmapped-export design (§9 T3 caveats), the export stage stops producing shape_lightmap_not_optimized content and the standalone lightmapper takes over that slot in the flow, feeding the same 2_build optimizer.
  • Precedent for a non-Max route already exists in-tree: processes/shape/1_export.py first runs mesh_export (nel/tools/3d/mesh_export, assimp-based: .blend/.obj/.dae/.gltf) over the same source directories with the same tag protocol before falling through to the Max branch. The headless .max exporter should be wired in exactly the same shape as that branch — and mesh_export is also the natural home or template for the TOML-sidecar import route (tech tree A2).

11. Known defects and hazards (verified in source; fix or guard before building on the affected path)

  1. CStorageValue<std::string>::getSize and <ucstring>::getSize return the size as the boolean and never write the out-parameter (storage_value.cpp:74-84: return Value.size();). Currently harmless because chunk sizes are back-patched by seeking (§3), so getSize is dead on the write path — but any new code that trusts getSize will mis-size string chunks. Same bug class in CStorageArraySizePre<T>::getSize (storage_array.h:158: size = CStorageArray<T>::getSize(size) + sizeof(uint32); — adds 4 to a bool). Fix both before first use.
  2. CSceneClass::putChunk inserts into the wrong list: m_OrphanedChunks.insert(m_PutChunkInsert, …) with an iterator that build took from m_Chunks (scene_class.cpp:184,271). With node-based std::list this happens to splice the node into m_Chunks (the iterator's owner), which is why it "works" — but it is UB, and under C++11 ABIs both lists' cached sizes go inconsistent (m_OrphanedChunks.size() overcounts, m_Chunks.size() undercounts), which quietly skews the disown sanity check m_Chunks.size() < m_OrphanedChunks.size(). Change to m_Chunks.insert, then re-verify T2 across the corpus.
  3. CSceneClassContainer::parse assumes the builtin scene is the last chunk (m_StorageObjectByIndex[size()-1], scene.cpp:153) and hard-asserts the dynamic cast. Holds for the corpus; will fire on files where it doesn't. Acceptable as an assert, but know it's there.
  4. 64-bit chunks: readable (downgraded), not writable; sticky per-stream flag blocks rewrite of any file containing one (§3).
  5. CSceneClass::getChunkValue error message has a missing format argument (scene_class.cpp:169: nlerror("… 0x%x …") with no value) — UB in an error path; also several nlerror calls in header-side templates. Fix when touched.
  6. CTrackViewNode::parse checks the wrong member for the 0x0150 empty chunk (if (m_Empty0150) nlassert(m_Empty0140->Value.empty()), track_view_node.cpp:78) — copy-paste; latent nullptr deref if 0x0150 exists without 0x0140.
  7. CStorageRaw::serial and the string specializations take &Value[0] on possibly-empty vectors/strings — serialBuffer no-ops on len 0 so it's benign in practice, but it's technically UB; normalize when touched.
  8. Failure = nlerror/nlassert = process abort throughout (several files deliberately #define nlwarning nlerror — "elevate warnings to errors for stricter reading"; keep that convention for new class code, it's what makes silent misparses impossible). The corpus harness, however, must run file-scoped: wrap per-file work so one bad file records a failure and the sweep continues (NeL assert/error behavior is configurable enough to throw instead of abort; that, or a child-process-per-file runner — decide once, at harness level, not by softening the library's checks).
  9. Empty-AppData drop — the one whitelisted normalization (§5). Any future intentional normalization must be added to that whitelist and this list, in the same commit that introduces it.

12. Rules for the implementing session

  1. Corpus-gate every change. T1 green before starting; T2 green over all ~8.6k files after each newly typed chunk, class, or enabled #define. No batch of typing lands on a red corpus.
  2. Type only what you can prove. A chunk id gets a typed mapping only when the typed serializer reproduces every instance in the corpus byte-exactly (that's what T2 verifies). No guessing formats from the Max SDK docs without corpus confirmation; unknown ids keep falling through to raw/container defaults.
  3. Never bypass the lifecycle. New classes implement the full sextet (parse/clean/build/disown/init/toStringLocal + createChunkById), call the parent first (parse/build) or last (disown), guard on m_ChunksOwnsPointers, use getChunk/putChunk/m_ArchivedChunks exactly as §5 describes, and put chunks back in the order they were taken. CEditableMesh is the template for "typed class that mostly passes through"; CNodeImpl for "small fully-typed class"; CAppData for "container with its own entry bookkeeping"; CDllDirectory/CClassDirectory3 for "entry table with position cache" — start from the nearest template, don't invent a fifth shape.
  4. Preserve emission forms. Where the format has alternatives (0x2034 vs 0x2035 references, 32- vs 64-bit headers, present-vs-absent optional chunks), record which form was read and re-emit that form. Byte-identity is the spec; "semantically equivalent" is not.
  5. First milestones, in dependency order: (a) fix defect list items 1-2; (b) build the corpus harness out of the dump tool (argv, per-stream compare, per-file isolation, version bucketing); (c) run T1/T2 baseline, inventory unparsed-class and 64-bit/compressed counts; (d) enable PMBS_GEOM_BUFFERS_PARSE and re-gate; (e) claim the noted tri/poly object ids (0x0901-0x0904, 0x0906-0x090c); (f) PARSE_NELDATA: typed AppData decoders per export_appdata.h; (g) materials/param blocks (fill the stubs, register them) as export needs them; (h) node transform controllers (currently unknowns — needed for object placement at export); then the nel_mesh_lib export logic retargeted onto this object model, T3-gated. The lightmapping phase is explicitly out of the exporter: shapes export unmapped, and calc_lm becomes a separate standalone tool validated in aggregate (see T3 caveats).
  6. Keep this document current. New chunk ids, new normalizations, new invariants, fixed defects — same commit that changes the code updates this page and the Tutorial Tech Tree status.

Clone this wiki locally