Skip to content

pipeline_max_coverage

Jump to bottom
Jan Boon edited this page Jul 9, 2026 · 34 revisions

title: pipeline_max coverage vs the build_gamedata Max export surface description: Gap analysis of the headless .max exporters against every Max-side export build_gamedata needs, with sized work packages for follow-up sessions published: true date: 2026-07-09T00:00:00.000Z tags: editor: markdown dateCreated: 2026-07-08T00:00:00.000Z

Written 2026-07-08 (authoring session), updated 2026-07-09 (shape-coverage-2 session: EditablePoly + water landed). Pick-up map for the next few implementation sessions. Read pipeline_max_design FIRST — especially §12 (rules for the implementing session) — and the per-format sections referenced below. The goal state: build_gamedata's 1_export.py steps can run the headless pipeline_max_* tools instead of 3ds Max, for every asset type, with corpus-proven fidelity.

How the surface was enumerated

nel/tools/build_gamedata/processes/*/1_export.py + their maxscript/*.ms scripts. Every Max-side export goes through one of these NelExport* MAXScript entry points (implemented in nel/tools/3d/plugin_max/nel_export/nel_export_script.cpp → nel_mesh_lib/): NelExportShapeEx, NelExportAnimation (used by BOTH the anim and shape processes), NelExportSkeleton, NelExportSkeletonWeight, NelExportInstanceGroup (ig + ligo), NelExportCollision (rbank + ligo), NelExportPACSPrimitives, NelExportVegetable, NelExportLodCharacter, plus ExportRykolZone (zone) and LigoExportZone (ligo). The ps process is a plain file copy (no Max involvement) despite its log line. Everything else under processes/ (rbank build, ig_light, zone_light, banks, maps, sheets, …) is post-export C++ tooling that already runs headless.

Reference outputs live in ~/pipeline_export (continents/ecosystems/common trees, mirroring the workspace config under ~/ryzomcore_leveldesign/workspace); sources in ~/ryzomcore_graphics.

Coverage table

Entry point Process(es) Output Refs in ~/pipeline_export Headless tool Status
ExportRykolZone zone .zone 6306 (incl. ligo) pipeline_max_export_zone GREEN — corpus 1068 exact + 130 x87-tier + 3 budget (§10h)
LigoExportZone ligo .zone/.ligozone bricks (in above) pipeline_max_export_zone --ligo GREEN — .ligozone 100% byte-identical; .zone same tiers as ExportRykolZone above (§10h)
NelExportSkeleton skel .skel 203 pipeline_max_export_skel GREEN — legacy 99.4% drot, era classes documented (§10, §10e)
NelExportSkeletonWeight swt .swt 1 (+ corpus) pipeline_max_export_swt GREEN (§10f)
NelExportInstanceGroup (standalone ig) ig .ig 54 direct (→ 18497 final, downstream via land_exporter, out of scope) pipeline_max_export_ig GREEN — T1/T2 116/116; 54/54 direct-ref field-exact under uninit-mask (--max-direct-diff budget retired to 0, gate strict); processed-tier 40/40 match. 0 truly byte-identical remains a legitimate tolerance class (reference exporter's SunContribution/Light[]/STL-light-tie-order uninit heap). §10w closed the last open diff via the Edit Mesh 0x0130 created-verts decode
NelExportInstanceGroup (ligo bricks) ligo .ig 537 direct pipeline_max_export_ig --ligo MOSTLY (near green) — T1/T2 1201/1201; 529/537 (98.5%) match under uninit-mask (71 byte-identical + 458 uninit-only under x64/SSE; under VS2008/x87 shifts to 130 byte-identical + 399 uninit-only — same 8 open, x87 codegen zeros the uninit heap x64 leaves random). 8 open, budgeted DIFF_BUDGET=8 (§10w, was 89 → 28 → 8 in one session): ~5–6 PS-instance clusterize AABBox borderline containment (extra cluster links on rotated FX bboxes, LOW real-content risk — extra links harden culling rather than drop content) + 2–3 unclassified singleton files (taverne palme_type_C3 Scale ≈3e-5, jungle-2 ju_water_jungle_coude Pos ≈15cm, brandon fortress_int_03/04 Clusters + one Scale)
NelExportAnimation anim, shape .anim 9146 pipeline_max_export_anim MOSTLY — non-biped byte-identical; biped structural-exact, in-between IK bounded (median 0.069, §10r); waterfall texture-matrix material anims shipped (§10k, 642 byte-identical in the shape session sweep); remaining gap: biped exactness (§10s)
NelExportShapeEx shape .shape 11251 pipeline_max_export_shape PARTIAL, M2 tail closing — M1 plain-CMesh (§10i) + M2 CMeshMultiLod + parametric prims + lightmap-mask structural verify + shared-EDITMESH migration (§10y) + EditablePoly + CWaterShape (§10z). Corpus 2828 exported: 771 float-eq + 721 differ + 72 mapext-bucketed + 982 lightmap-VERIFIED + 281 lightmap-diff + 1 no-ref. 762 refs still not produced: skinned 654 (the sole big remaining frontier), remanence 82, mesh-eval 14, flare 13, interface 9, water 1
NelExportCollision rbank, ligo .cmb 445 (16 genuine direct Max exports; the other 429 are placed-instance copies land_exporter — a separate, already-headless C++ tool — writes into RBankCmbExportDirectory from the ligo-brick Max exports below, same mechanism as its .ig placement copy; porting that copy step is out of scope for pipeline_max, but it is real and load-bearing: it's what rbank's "Build rbank indoor" step reads to build the shipped .rbank/.gr/.lr) pipeline_max_export_cmb MOSTLY — direct tier (gated): 16/16 files export, 10/16 igs exact + 3/16 x87-tier + 3/16 differ; the 3 diffs' underlying quality moved massively via §10x (0x0208/0x0220 decode + face-attrib apply — FY_hall_reunion now 247/248 face-index match + exact material distribution vs the reference), but the residuals still fail cmp_close's strict equality. Ligo-brick tier is a genuine Max export too — nel_ligo_export.ms calls NelExportCollision per brick exactly like it calls NelExportInstanceGroup, landing brick-level .cmb in LigoEcosystemCmbExportDirectory, which is then land_exporter's RefCMBDir input for the placement copy above. It's measured but not gate-blocking in the corpus test (same soft-gating convention as the ig ligo tier's DIFF_BUDGET, not a sign the step is inert): §10x closed the "6 files produce zero output" XRef gap (shared XREFRESOLVE::resolveXRefObject from ig now routes them); 30 byte-identical + 83 float-eq + 41 diff + 16 export failures across 170 measured igs — up from 11 + 15 + 22 + 6 = 54 baseline
NelExportPACSPrimitives pacs_prim .pacs_prim 493 pipeline_max_export_pacs_prim GREEN — 493/493 byte-identical (§10v)
NelExportVegetable veget .veget 126 — MISSING
NelExportLodCharacter clodbank .clod 105 — MISSING

Cross-cutting infrastructure that is DONE and reusable: the storage/scene layer with full T1/T2 byte-identity over every corpus (max files parse→build byte-identical), the OLE backend (§2a), the VS2008/Wine x87 reference build (§2b), Edit Mesh / Editable Poly / Edit Patch geometry decode (ig/zone/shape sessions), the shared pipeline_max_export_common library — EDITMESH (Edit Mesh decode+apply, §10x, now driving ig+cmb+shape), XREFRESOLVE (§10x, ig+cmb), PRIMMESH (Box/Cylinder/Sphere/Plane topology, §10y, ig-source + shape-consumer), max_scene/max_math/OLDPBLOCK/db_path/appdata_util — materials + ParamBlock2 (§10j), the biped system incl. typed keytrack editing (§10t/§10u), AppData flag reading, and the corpus-test harness patterns (*_corpus.py, fork-per-file, tolerance tiers, --compare-lightmap-mask).

The gaps, sized

A. The remaining small formats (breadth completion)

.pacs_prim and .cmb landed 2026-07-08 (§10v) — .veget and .clod close the rest of the SURFACE.

  1. .cmb — NelExportCollision DONE, mostly (§10v landed, §10x closed most follow-ups) — pipeline_max_export_cmb. Open follow-ups (from §10x): the FY_hall_reunion created-vert positions (16 verts, ~10 unit systematic offset — candidate 0x2510 mod-context TM applied to created verts, unproven); the one face-index remap on FY_hall_reunion + its larger-scale counterpart on Zo_bt_hall_Reunion_vitrine (likely modern-format equivalent of legacy TOPO_FACEMAP_CHUNK 0x2780, not yet identified in the corpus MDELTA_CHUNK sub-trees); Zo_bt_Hall_Conseil vertex offset (still open on its own separate terms); ligo tier's 16 export failures.
  2. .pacs_prim — NelExportPACSPrimitives DONE (§10v) — pipeline_max_export_pacs_prim, 493/493 byte-identical.
  3. .veget — NelExportVegetable (126 refs; 817 veget-related source .max). Serializer: NL3D::CVegetableShape (nel/3d/vegetable_shape.h) — mesh + bend-weight painting (vertex colors/UV1?) + appdata params (bend mode, freq…). Reference impl: nel_mesh_lib/export_vegetable.cpp (buildVegetableShape). Plain-mesh scale; the vertex-paint channel decode is the only likely new ground.
  4. .clod — NelExportLodCharacter (105 refs; 111 *_lod.max sources under stuff/lod_actors/). Serializer: NL3D::CLodCharacterShapeBuild (nel/3d/lod_character_shape.h) — positions, tris, UVs, normals + per-vertex skin weights to a bone-name table. Reference impl: nel_mesh_lib/export_lod_character.cpp. NOTE: this needs the SKIN modifier decode (bone weights) — the same decode the shape-MRM work needs; whichever session lands first should put the skin-weight reader in pipeline_max_export_common for the other to reuse. (The .swt exporter already reads skeleton-weight data — check what it shares before writing anything new.)

B. Shape depth (the flagship remaining work — Opus-sized)

NelExportShapeEx beyond plain CMesh, in rough dependency order (post-§10z state):

  1. Skinned + MRM meshes (CMeshMRM / CMeshMRMSkinned) — characters, fauna; the bulk of the 11251 refs, still the biggest single remaining not-produced class (654 files) and now essentially the sole big coverage frontier. Two halves: (a) the SKIN decode (Physique/Skin modifier chunks → per-vertex bone weights; corpus era is Physique — decode lives in nel_mesh_lib/export_skinning.cpp for reference) and (b) the MRM build (NL3D::CMRMBuilder) — deterministic NeL code; link the CURRENT NeL builder like anim/shape M1 already link NeL, then measure the 2004-vs-now divergence per file and bucket tolerance tiers exactly like §10h/§10i did (expect an era tier; the VS2008 build is the instrument if it matters). NOTE: pipeline_max_export_shape already dispatches CMeshMRMSkinned::isCompatible on the LOD_MRM path (§10i), so the exporter-side plumbing is in place; the SKIN modifier decode is what's missing.
  2. CMeshMultiLod / coarse meshes DONE §10y — LOD_NAME_COUNT-driven build with per-slot CMeshGeom or CMeshMRMGeom, per-slot DistMax/BlendLength/BlendIn/BlendOut/CoarseMesh flags from slave appdata; 418 files added to the exported set (+63 float-eq, +274 lightmap-verified, +79 differ). One residual class: multi-lod slaves whose materials aren't all in the parent's list produce ref-larger material counts (e.g. common/construction/gen_mo_charognemammal 'materials: 3 vs 4') — likely the reference plugin merges slave materials into the base list; not yet root-caused. Coarse-mesh routing (with-coarse-mesh output dir) was already handled by M1's NEL3D_APPDATA_LOD_COARSE_MESH scan.
  3. Water shapes (CWaterShape) (221 files) DONE §10z — WATERBUILD::buildWaterShape replicates CExportNel::buildWaterShape: mesh eval + node-local transform + Z=0 projection + convex hull → CPolygon2D::setShape, PB2 reads for pool id / wave height / slot enables + texmaps, textures via a lean BitmapTex + NeL-Multi-Bitmap loader; default transformation from decompMatrix(getLocalMatrix). Same-session byte-identity rounds 1+2 landed on top of the coverage path: (r1) CTextureBlend day/night envmap pairing when slot 2's TEXMAP is present + CWaterShape stream v7→v4 downgrade (patch version byte + truncate trailing 14 bytes) → file size matches reference exactly (698 vs 698 on tr_water_00), byte diffs 111 → 35; (r2) correct v14-water-material NLB_MAIN param IDs pinned empirically via PMB_WATER_DUMP — fBumpU/VScale/Speed=0x09..0x0c, fDisplaceU/VScale/Speed=0x0d..0x10, fWaterHeightFactor=0x11, iWaterPoolID=0x12 — plus tail bool defaults (splash=true from ctor, UsesSceneWaterEnvMap[0/1]=false explicit) → byte diffs 35 → 3. 220 of 221 water files close (the 1 residual is a genuine missing-required-map case matching reference behavior). Open items on the 219 differ tail: the 3-byte DefaultRotQuat MSB residual (ref has sign bit set on x/y/z; ours has 0.0 — likely the same double-cover boundary class as CMesh's rot-quat sign flip §10i already documents); the water script's bEnableWaterSplash + bWaterUseSceneEnvMap{Above,Under} name-to-id mapping still unmapped in-code (initial guesses at 0x19..0x1b turned out to be the standard material's iBlendSrcFunc/iBlendDestFunc); setColorMapMat (2x3 affine, only fires when diffuse slot 7 is enabled — most corpus files don't use it); a CWaterShape field-walk in compareShapesFields (analogous to compareMeshBase) would promote the 219 water-differ files to FLOATEQ.
  4. CSegRemanence (82), CFlareShape (13), interface (9) — the small M2 tail. Flare = nel_flare scripted-plugin PB2 (like nel_ps §10g); remanence = USE_REMANENCE appdata + segment geometry; interface = INTERFACE_FILE border-weld (loads another .max, welds normals along the interface polygon).
  5. Waterfall texture-matrix material anims DONE §10k — landed pre-shape-session; 642 byte-identical .anim outputs in the M2 shape sweep.
  6. Lightmap-diff tail (281 files) — the calc-lm-mask compare says these have structural divergence beyond just the lightmap-added fields. Candidates: multi-lod material merging (above), missing UV generation on parametric prims, Unwrap UVW modifier effects (§10y's mod:02df2e3a, 25 uses). Triage after (1)–(3) close.
  7. Modifier tail — UVW Map (76 uses, standard-plugin planar/cylindrical/spherical/box projection, §10i handoff's biggest UV-differ class), Unwrap UVW (25 uses, user-authored overlays), FFD(box) (20 uses, Free-Form Deformer lattice), and Nel VertexTreePaint (110 uses, our own plugin at plugin_max/nel_vertex_tree_paint/ — vertex color paint modifier, likely quickest to decode since we own the format). Class ids catalogued in §10y.
  8. UV mapping-coord generation for parametric primitives — the 27 new differ files from §10y's parametric-prim path. Each primitive has its own UV formula (Box → 6-face box, Cylinder → cylindrical, Sphere → spherical, Plane → planar) plus a "generate mapping coords" checkbox; Max SDK reference. Extend ~/prim_mesh_dataset's gen_prim_mesh_dataset.ms to dump map channel 1 per primitive as a GT source (§10i handoff already flagged this).

C. Anim depth (parked, documented)

The biped §10s experiment list (arm-pin space decode, in-plant foot rotation probe, turn-rule probe, blend-ramp weighting, PODA leads (6)–(8) in poda_1985_notes) — pick up only when biped exactness becomes the priority; the export is already structurally exact with bounded in-between error.

D. Ig export — open correctness items

Standalone (§10g) is GREEN as of §10w (2026-07-08): budget retired to 0, 54/54 direct-ref field-exact under uninit-mask. Ligo (§10g-bis) is MOSTLY near-green with 8 open diffs (DIFF_BUDGET=8, down from 89 → 28 → 8 in one session); the two large open classes that occupied this section (cluster-containment defect + selection-order divergence) both closed via §10w. The remaining 8 are documented as a punch list for the next session — small and mostly precision-tier, not structural. Historical §10g/§10g-bis budgets and their un-root-caused states are kept in the design doc; this section is now the current open list.

  1. Cluster-membership / vertex-in-volume containment. CLOSED §10w — the shared defect that surfaced as the standalone's TR_hall_reu_vitrine diff AND ~65 of the ligo path's 89 was the Edit Mesh 0x0130 created-verts chunk never being read (16-byte srcTag+Point3, mis-decoded as 0x0210 in the §10g decode note); any node whose base mesh got fully deleted and rebuilt in Edit Mesh evaluated to an empty world mesh, missing all cluster containment. Both cases green after the fix. The gen_decors_probe.ms Max 9 dump the user contributed (~/decors_probe/manifest.txt) also refuted the earlier "Mirror plane two stories down" hypothesis — the mirror offset=-24.9164, axis=Z, gizmo tz=1.78 produce exactly our current world-Z range.
  2. Ligo selection-order divergence. CLOSED §10w — closed by two orthogonal fixes: (a) depth-first scene-tree walk for the per-category iteration (buildTreeOrder from ssc->scene()->rootNode(), matches MaxScript's $geometry/$lights/$helpers), which closed the sibling-parent-interleave cases (desert nb01..nb05 etc.); (b) the ligo XRef pre-pass now filters exactly like the maxscript — single-step source resolution (resolveXRefObject() recurses, so peel one step and stop at node->getReference(1)), reject nested-XRef sources, gate on {Geom, Helper, Light} superclass — which closed the 23 village-bundle-file class (fy_module_village_nb_01..05, zo_imm_village_nb_01..05, tr_village_nb_01..05, jungle foret-18..21_village_a..d, etc.). Kaetemi's Max 9 gen_ig_selorder_probe.ms run pinned the maxscript rule that made the fix possible.
  3. Byte-identity backlog. Under x64/SSE the standalone remains 0/54 truly byte-identical — the reference exporter's SunContribution/Light[] uninit-heap + STL-light-tie-order class is a legitimate tolerance not a bug; ligo is 71/537. Under VS2008/x87 the same code produces 130/537 byte-identical (59 files gain full byte-identity, x87 codegen zeros the uninit heap x64 leaves random). Standalone doesn't shift because its uninit fields are per-instance (every ig has some), whereas ligo has many small igs where the uninit fields don't get exercised. Byte-identity is a stretch goal, not the gating bar (matches the zone/skel precedents).
  4. Remaining 8 ligo diffs (the current open list).
    • ~5-6 files, PS-instance clusterize AABBox borderline containment (fy_sheriff inst 23/26, fy_warschool inst 48/49/79, street inst 103/113/117, oasis_kami_auberge, ma_serre_village_a): the pss->getAABBox() corners re-boxed in world space (via CAABBox::transformAABBox — the reference plugin's own behavior, corpus-validated) land within CLUSTERPRECISION = 5mm of a cluster plane on rotated FX-instance bboxes; ours attach the PS to 2/3 clusters where reference attaches to 1/2. Confirmed VS2008/x87 build produces the SAME diffs → this isn't x87 codegen precision, it's a genuine borderline case. LOW real-content risk (extra links harden visibility culling rather than drop content), acceptable tolerance-tier candidate. Would need a Max-side dump of the shape's getAABBox() bytes to fully pin whether the reference used a smaller/different bbox or the CLUSTERPRECISION test itself.
    • 3 singleton unclassified files: taverne.ig inst 53 palme_type_C3_taverne Scale ≈3e-5 (may be a tighter epsilon needed on Scale — same magnitude class as the standalone POS_EPS); jungle-2.ig inst 0 ju_water_jungle_coude Pos differs by ≈15cm in X, ≈1.4cm in Y (real content difference — likely a modifier/controller decode gap for this water node); fy_cn_fortress_a_brandon.ig inst 38 Scale ≈1e-5 + inst 97/99 Clusters (1 vs 3) / (0 vs 2) — this one is the opposite direction from the PS borderline class (we find FEWER clusters, not more), suggesting a mesh evaluation issue on FY_CN_fortress_int_03/04 specifically. Not yet triaged individually.

Whoever picks this up: the punch list is small enough now to close each item on its own. Retire by fixing diffs and re-tightening the budget — don't raise. When it hits 0, tighten to 0 in ligo_ig_corpus.py and re-grade the table row from MOSTLY to GREEN.

Suggested session plans

Session 1 (Sonnet-scale): .pacs_prim + .cmb. DONE 2026-07-08 (§10v/§10x) — pipeline_max_export_pacs_prim (493/493 byte-identical) and pipeline_max_export_cmb (16/16 direct files T1/T2, 13/16 T3 exact-or-tolerance, 3/16 documented open gaps; ligo-brick tier measured but not gate-blocking, same real Max export as the direct tier, feeding land_exporter → the shipped rbank). pacs_prim_corpus.py/cmb_corpus.py wired into ctest. Open follow-ups listed in §10v/§10x and item A.1 above.

Session 2 (Opus-scale): shape M2 coverage frontier. DONE 2026-07-08 (§10y) — the M2 breadth landing that added CMeshMultiLod (+418 files), parametric-prim shapes (+37), Edit Mesh shared-library migration (defect fix + code consolidation), and --compare-lightmap-mask mode (1263 lightmap-bucketed → 982 verified + 281 diff). Corpus: exported 2141 → 2596 (+21% coverage). See §10y for the full session summary + open items handoff.

Session 3 (Opus-scale): skinned/MRM shapes. The big one, still open (654 files, the biggest remaining not-produced class). Approach exactly like §10i M1: replicate CExportNel::buildShape's MRM branch against the real NeL builders, corpus-bucket the outcomes (byte-exact / float-eq / era-tier / differ), and expect several probe rounds. Reuse the skin decode from session 2 if it landed. Success metric: the character/fauna shape corpus classified into explained tiers with zero unexplained structural fails — byte-identity is the stretch goal, not the bar (see the zone precedent). The exporter-side dispatch (CMeshMRMSkinned::isCompatible + LOD_MRM path in buildShapeForNode) is already wired; this session is entirely (a) the Physique/Skin modifier decode and (b) measuring the MRM builder's 2004-vs-now divergence.

Session 3b (Sonnet-scale, after Session 3): the M2 tail — CWaterShape CSegRemanence, CFlareShape, interface meshes. Water DONE 2026-07-09 (§10z, 220/221 files closed with 3-byte quat-MSB residual on the differ tail). Remaining 82 + 13 + 9 = 104 files. Each is its own PB2 decode + own NL3D builder, none require skinning. Flare is small and follows the nel_ps pattern (§10g); remanence + interface are the smallest. Together they close the M2 shape surface below skinned. Additionally: an EditablePoly extraction path landed same session (§10z, +13 files exported, closing 60 of 73 residual mesh-eval skips — the remaining 14 are non-EditablePoly object classes obj:0000000a/obj:00001040/obj:00001065 to be identified).

Session 4 (Sonnet-scale, anytime): shape-process modifier tail — UVW Map (76 uses), Unwrap UVW (25), FFD(box) (20), Nel VertexTreePaint (110, our own plugin). Each is a modifier-app decode + a UV/geometry rewrite pass; all are self-contained. Nel VertexTreePaint is likely quickest (we own the format via plugin_max/nel_vertex_tree_paint/) and closes the largest single unknown-modifier count. UV mapping-coord generation for parametric prims (Box/Cylinder/Sphere/Plane) rides along here.

Session 5 (Sonnet-scale): .veget, then .clod if the skin decode from Session 3 is available. .veget first (no skinning). For .clod, put the skin-weight decode in pipeline_max_export_common — Session 3 will have already done this.

Waterfall texture-matrix anims DONE §10k, before the shape session.

Non-negotiables for every session (from §12, restated)

  • Load the session baseline first: all pipeline_max sources + the design doc (see the pipeline-max-session-baseline memory note / §12).
  • Corpus-gate EVERYTHING: T1/T2 byte-identity must stay green over the full .max corpus for any parser change; new exporters get their own T3 vs ~/pipeline_export references with explicit tolerance tiers; per-file forked runs (parse→clean→build→disown; COFile, not CMemStream).
  • The .ms scripts + workspace config (~/ryzomcore_leveldesign/workspace) are the ground truth for WHAT gets exported from each source file and with which options — read them before the C++.
  • Emission-form preservation, typed-chunk order preservation, and the §11 hazard list apply unchanged.
  • Don't rebuild binaries mid-sweep; don't trust per-file probes over full-corpus A/B (§10r method notes).

Clone this wiki locally