Skip to content

pipeline_max_coverage

Jump to bottom
Jan Boon edited this page Jul 8, 2026 · 34 revisions

title: pipeline_max coverage vs the build_gamedata Max export surface description: Gap analysis of the headless .max exporters against every Max-side export build_gamedata needs, with sized work packages for follow-up sessions published: true date: 2026-07-08T00:00:00.000Z tags: editor: markdown dateCreated: 2026-07-08T00:00:00.000Z

Written 2026-07-08 (authoring session), as the pick-up map for the next few implementation sessions. Read pipeline_max_design FIRST — especially §12 (rules for the implementing session) — and the per-format sections referenced below. The goal state: build_gamedata's 1_export.py steps can run the headless pipeline_max_* tools instead of 3ds Max, for every asset type, with corpus-proven fidelity.

How the surface was enumerated

nel/tools/build_gamedata/processes/*/1_export.py + their maxscript/*.ms scripts. Every Max-side export goes through one of these NelExport* MAXScript entry points (implemented in nel/tools/3d/plugin_max/nel_export/nel_export_script.cpp → nel_mesh_lib/): NelExportShapeEx, NelExportAnimation (used by BOTH the anim and shape processes), NelExportSkeleton, NelExportSkeletonWeight, NelExportInstanceGroup (ig + ligo), NelExportCollision (rbank + ligo), NelExportPACSPrimitives, NelExportVegetable, NelExportLodCharacter, plus ExportRykolZone (zone) and LigoExportZone (ligo). The ps process is a plain file copy (no Max involvement) despite its log line. Everything else under processes/ (rbank build, ig_light, zone_light, banks, maps, sheets, …) is post-export C++ tooling that already runs headless.

Reference outputs live in ~/pipeline_export (continents/ecosystems/common trees, mirroring the workspace config under ~/ryzomcore_leveldesign/workspace); sources in ~/ryzomcore_graphics.

Coverage table

Entry point Process(es) Output Refs in ~/pipeline_export Headless tool Status
ExportRykolZone zone .zone 6306 (incl. ligo) pipeline_max_export_zone GREEN — corpus 1068 exact + 130 x87-tier + 3 budget (§10h)
LigoExportZone ligo .zone/.ligozone bricks (in above) pipeline_max_export_zone --ligo GREEN — .ligozone 100% byte-identical; .zone same tiers as ExportRykolZone above (§10h)
NelExportSkeleton skel .skel 203 pipeline_max_export_skel GREEN — legacy 99.4% drot, era classes documented (§10, §10e)
NelExportSkeletonWeight swt .swt 1 (+ corpus) pipeline_max_export_swt GREEN (§10f)
NelExportInstanceGroup (standalone ig) ig .ig 54 direct (→ 18497 final, downstream via land_exporter, out of scope) pipeline_max_export_ig GREEN — T1/T2 116/116; 54/54 direct-ref field-exact under uninit-mask (--max-direct-diff budget retired to 0, gate strict); processed-tier 40/40 match. 0 truly byte-identical remains a legitimate tolerance class (reference exporter's SunContribution/Light[]/STL-light-tie-order uninit heap). §10w closed the last open diff via the Edit Mesh 0x0130 created-verts decode
NelExportInstanceGroup (ligo bricks) ligo .ig 537 direct pipeline_max_export_ig --ligo MOSTLY (near green) — T1/T2 1201/1201; 529/537 (98.5%) match under uninit-mask (71 byte-identical + 458 uninit-only under x64/SSE; under VS2008/x87 shifts to 130 byte-identical + 399 uninit-only — same 8 open, x87 codegen zeros the uninit heap x64 leaves random). 8 open, budgeted DIFF_BUDGET=8 (§10w, was 89 → 28 → 8 in one session): ~5–6 PS-instance clusterize AABBox borderline containment (extra cluster links on rotated FX bboxes, LOW real-content risk — extra links harden culling rather than drop content) + 2–3 unclassified singleton files (taverne palme_type_C3 Scale ≈3e-5, jungle-2 ju_water_jungle_coude Pos ≈15cm, brandon fortress_int_03/04 Clusters + one Scale)
NelExportAnimation anim, shape .anim 9146 pipeline_max_export_anim MOSTLY — non-biped byte-identical; biped structural-exact, in-between IK bounded (median 0.069, §10r); gaps: texture-matrix material anims (waterfalls), biped exactness (§10s)
NelExportShapeEx shape .shape 11251 pipeline_max_export_shape PARTIAL — M1 plain-CMesh path only (704 float-eq + 989 lightmap-bucketed + 383 differ, §10i); skinned/MRM, MultiLod/coarse, water NOT attempted
NelExportCollision rbank, ligo .cmb 445 (16 genuine direct; the rest are a downstream non-Max copy, out of scope) pipeline_max_export_cmb MOSTLY — direct tier (gated): 16/16 files export, 10/16 igs exact + 3/16 x87-tier + 3/16 differ; the 3 diffs' underlying quality moved massively via §10x (0x0208/0x0220 decode + face-attrib apply — FY_hall_reunion now 247/248 face-index match + exact material distribution vs the reference), but the residuals still fail cmp_close's strict equality. Ligo tier (informational): §10x closed the "6 files produce zero output" XRef gap (shared XREFRESOLVE::resolveXRefObject from ig now routes them); 30 byte-identical + 83 float-eq + 41 diff + 16 export failures across 170 measured igs — up from 11 + 15 + 22 + 6 = 54 baseline
NelExportPACSPrimitives pacs_prim .pacs_prim 493 pipeline_max_export_pacs_prim GREEN — 493/493 byte-identical (§10v)
NelExportVegetable veget .veget 126 — MISSING
NelExportLodCharacter clodbank .clod 105 — MISSING

Cross-cutting infrastructure that is DONE and reusable: the storage/scene layer with full T1/T2 byte-identity over every corpus (max files parse→build byte-identical), the OLE backend (§2a), the VS2008/Wine x87 reference build (§2b), Edit Mesh / Editable Poly / Edit Patch geometry decode (ig/zone/shape sessions), materials + ParamBlock2 (§10j), the biped system incl. typed keytrack editing (§10t/§10u), AppData flag reading, and the corpus-test harness patterns (*_corpus.py, fork-per-file, tolerance tiers).

The gaps, sized

A. The remaining small formats (breadth completion)

.pacs_prim and .cmb landed 2026-07-08 (§10v) — .veget and .clod close the rest of the SURFACE.

  1. .cmb — NelExportCollision DONE, mostly (§10v landed, §10x closed most follow-ups) — pipeline_max_export_cmb. Open follow-ups (from §10x): the FY_hall_reunion created-vert positions (16 verts, ~10 unit systematic offset — candidate 0x2510 mod-context TM applied to created verts, unproven); the one face-index remap on FY_hall_reunion + its larger-scale counterpart on Zo_bt_hall_Reunion_vitrine (likely modern-format equivalent of legacy TOPO_FACEMAP_CHUNK 0x2780, not yet identified in the corpus MDELTA_CHUNK sub-trees); Zo_bt_Hall_Conseil vertex offset (still open on its own separate terms); ligo tier's 16 export failures.
  2. .pacs_prim — NelExportPACSPrimitives DONE (§10v) — pipeline_max_export_pacs_prim, 493/493 byte-identical.
  3. .veget — NelExportVegetable (126 refs; 817 veget-related source .max). Serializer: NL3D::CVegetableShape (nel/3d/vegetable_shape.h) — mesh + bend-weight painting (vertex colors/UV1?) + appdata params (bend mode, freq…). Reference impl: nel_mesh_lib/export_vegetable.cpp (buildVegetableShape). Plain-mesh scale; the vertex-paint channel decode is the only likely new ground.
  4. .clod — NelExportLodCharacter (105 refs; 111 *_lod.max sources under stuff/lod_actors/). Serializer: NL3D::CLodCharacterShapeBuild (nel/3d/lod_character_shape.h) — positions, tris, UVs, normals + per-vertex skin weights to a bone-name table. Reference impl: nel_mesh_lib/export_lod_character.cpp. NOTE: this needs the SKIN modifier decode (bone weights) — the same decode the shape-MRM work needs; whichever session lands first should put the skin-weight reader in pipeline_max_export_common for the other to reuse. (The .swt exporter already reads skeleton-weight data — check what it shares before writing anything new.)

B. Shape depth (the flagship remaining work — Opus-sized)

NelExportShapeEx beyond plain CMesh, in rough dependency order:

  1. Skinned + MRM meshes (CMeshMRM) — characters, fauna; the bulk of the 11251 refs. Two halves: (a) the SKIN decode (Physique/Skin modifier chunks → per-vertex bone weights; corpus era is Physique — decode lives in nel_mesh_lib/export_skinning.cpp for reference) and (b) the MRM build (NL3D::CMRMBuilder) — deterministic NeL code; link the CURRENT NeL builder like anim/shape M1 already link NeL, then measure the 2004-vs-now divergence per file and bucket tolerance tiers exactly like §10h/§10i did (expect an era tier; the VS2008 build is the instrument if it matters).
  2. CMeshMultiLod / coarse meshes — the shape_with_coarse_mesh* reference dirs exist in ~/pipeline_export/common; the coarse-mesh texture step is part of the shape process.
  3. Water shapes (CWaterShape) — few files, self-contained (see the existing nel/3d/water_* docs and the waterfall/texture-matrix notes).
  4. Shape-process animations — shape_export.ms ALSO calls NelExportAnimation for animated materials: the waterfall texture-matrix gap (see waterfall-texmat-anim finding: bExportTextureMatrix + StdUVGen U/V Offset controllers; ref anims in ~/core4_data/*_shapes/waterfall*.anim). This is a small, well-scoped export_anim extension and can ride along with any session.

C. Anim depth (parked, documented)

The biped §10s experiment list (arm-pin space decode, in-plant foot rotation probe, turn-rule probe, blend-ramp weighting, PODA leads (6)–(8) in poda_1985_notes) — pick up only when biped exactness becomes the priority; the export is already structurally exact with bounded in-between error.

D. Ig export — open correctness items

Standalone (§10g) is GREEN as of §10w (2026-07-08): budget retired to 0, 54/54 direct-ref field-exact under uninit-mask. Ligo (§10g-bis) is MOSTLY near-green with 8 open diffs (DIFF_BUDGET=8, down from 89 → 28 → 8 in one session); the two large open classes that occupied this section (cluster-containment defect + selection-order divergence) both closed via §10w. The remaining 8 are documented as a punch list for the next session — small and mostly precision-tier, not structural. Historical §10g/§10g-bis budgets and their un-root-caused states are kept in the design doc; this section is now the current open list.

  1. Cluster-membership / vertex-in-volume containment. CLOSED §10w — the shared defect that surfaced as the standalone's TR_hall_reu_vitrine diff AND ~65 of the ligo path's 89 was the Edit Mesh 0x0130 created-verts chunk never being read (16-byte srcTag+Point3, mis-decoded as 0x0210 in the §10g decode note); any node whose base mesh got fully deleted and rebuilt in Edit Mesh evaluated to an empty world mesh, missing all cluster containment. Both cases green after the fix. The gen_decors_probe.ms Max 9 dump the user contributed (~/decors_probe/manifest.txt) also refuted the earlier "Mirror plane two stories down" hypothesis — the mirror offset=-24.9164, axis=Z, gizmo tz=1.78 produce exactly our current world-Z range.
  2. Ligo selection-order divergence. CLOSED §10w — closed by two orthogonal fixes: (a) depth-first scene-tree walk for the per-category iteration (buildTreeOrder from ssc->scene()->rootNode(), matches MaxScript's $geometry/$lights/$helpers), which closed the sibling-parent-interleave cases (desert nb01..nb05 etc.); (b) the ligo XRef pre-pass now filters exactly like the maxscript — single-step source resolution (resolveXRefObject() recurses, so peel one step and stop at node->getReference(1)), reject nested-XRef sources, gate on {Geom, Helper, Light} superclass — which closed the 23 village-bundle-file class (fy_module_village_nb_01..05, zo_imm_village_nb_01..05, tr_village_nb_01..05, jungle foret-18..21_village_a..d, etc.). Kaetemi's Max 9 gen_ig_selorder_probe.ms run pinned the maxscript rule that made the fix possible.
  3. Byte-identity backlog. Under x64/SSE the standalone remains 0/54 truly byte-identical — the reference exporter's SunContribution/Light[] uninit-heap + STL-light-tie-order class is a legitimate tolerance not a bug; ligo is 71/537. Under VS2008/x87 the same code produces 130/537 byte-identical (59 files gain full byte-identity, x87 codegen zeros the uninit heap x64 leaves random). Standalone doesn't shift because its uninit fields are per-instance (every ig has some), whereas ligo has many small igs where the uninit fields don't get exercised. Byte-identity is a stretch goal, not the gating bar (matches the zone/skel precedents).
  4. Remaining 8 ligo diffs (the current open list).
    • ~5-6 files, PS-instance clusterize AABBox borderline containment (fy_sheriff inst 23/26, fy_warschool inst 48/49/79, street inst 103/113/117, oasis_kami_auberge, ma_serre_village_a): the pss->getAABBox() corners re-boxed in world space (via CAABBox::transformAABBox — the reference plugin's own behavior, corpus-validated) land within CLUSTERPRECISION = 5mm of a cluster plane on rotated FX-instance bboxes; ours attach the PS to 2/3 clusters where reference attaches to 1/2. Confirmed VS2008/x87 build produces the SAME diffs → this isn't x87 codegen precision, it's a genuine borderline case. LOW real-content risk (extra links harden visibility culling rather than drop content), acceptable tolerance-tier candidate. Would need a Max-side dump of the shape's getAABBox() bytes to fully pin whether the reference used a smaller/different bbox or the CLUSTERPRECISION test itself.
    • 3 singleton unclassified files: taverne.ig inst 53 palme_type_C3_taverne Scale ≈3e-5 (may be a tighter epsilon needed on Scale — same magnitude class as the standalone POS_EPS); jungle-2.ig inst 0 ju_water_jungle_coude Pos differs by ≈15cm in X, ≈1.4cm in Y (real content difference — likely a modifier/controller decode gap for this water node); fy_cn_fortress_a_brandon.ig inst 38 Scale ≈1e-5 + inst 97/99 Clusters (1 vs 3) / (0 vs 2) — this one is the opposite direction from the PS borderline class (we find FEWER clusters, not more), suggesting a mesh evaluation issue on FY_CN_fortress_int_03/04 specifically. Not yet triaged individually.

Whoever picks this up: the punch list is small enough now to close each item on its own. Retire by fixing diffs and re-tightening the budget — don't raise. When it hits 0, tighten to 0 in ligo_ig_corpus.py and re-grade the table row from MOSTLY to GREEN.

Suggested session plans

Session 1 (Sonnet-scale): .pacs_prim + .cmb. DONE 2026-07-08 (§10v) — pipeline_max_export_pacs_prim (493/493 byte-identical) and pipeline_max_export_cmb (16/16 direct files T1/T2, 13/16 T3 exact-or-tolerance, 3/16 documented open gaps; ligo tier informational). pacs_prim_corpus.py/cmb_corpus.py wired into ctest. Open follow-ups listed in §10v and item A.1 above.

Session 2 (Sonnet-scale): .veget, then .clod if the skin decode goes fast. .veget first (no skinning). For .clod, put the skin-weight decode in pipeline_max_export_common with its own mini-gate so the MRM session can build on it. If the Physique decode balloons, stop, document the chunk findings in the wiki, and leave .clod for the MRM session — do not half-land a skinning decode without a gate.

Session 3 (Opus-scale): skinned/MRM shapes. The big one. Approach exactly like §10i M1: replicate CExportNel::buildShape's MRM branch against the real NeL builders, corpus-bucket the outcomes (byte-exact / float-eq / era-tier / differ), and expect several probe rounds. Reuse the skin decode from session 2 if it landed. Success metric: the character/fauna shape corpus classified into explained tiers with zero unexplained structural fails — byte-identity is the stretch goal, not the bar (see the zone precedent).

Session 4 (Sonnet-scale, anytime): waterfall texture-matrix anims + shape-process anim wiring — small, high-polish-value, mostly export_anim plumbing.

Non-negotiables for every session (from §12, restated)

  • Load the session baseline first: all pipeline_max sources + the design doc (see the pipeline-max-session-baseline memory note / §12).
  • Corpus-gate EVERYTHING: T1/T2 byte-identity must stay green over the full .max corpus for any parser change; new exporters get their own T3 vs ~/pipeline_export references with explicit tolerance tiers; per-file forked runs (parse→clean→build→disown; COFile, not CMemStream).
  • The .ms scripts + workspace config (~/ryzomcore_leveldesign/workspace) are the ground truth for WHAT gets exported from each source file and with which options — read them before the C++.
  • Emission-form preservation, typed-chunk order preservation, and the §11 hazard list apply unchanged.
  • Don't rebuild binaries mid-sweep; don't trust per-file probes over full-corpus A/B (§10r method notes).

Clone this wiki locally