Skip to content

pipeline_max_coverage

Jump to bottom
Jan Boon edited this page Jul 8, 2026 · 34 revisions

title: pipeline_max coverage vs the build_gamedata Max export surface description: Gap analysis of the headless .max exporters against every Max-side export build_gamedata needs, with sized work packages for follow-up sessions published: true date: 2026-07-08T00:00:00.000Z tags: editor: markdown dateCreated: 2026-07-08T00:00:00.000Z

Written 2026-07-08 (authoring session), as the pick-up map for the next few implementation sessions. Read pipeline_max_design FIRST — especially §12 (rules for the implementing session) — and the per-format sections referenced below. The goal state: build_gamedata's 1_export.py steps can run the headless pipeline_max_* tools instead of 3ds Max, for every asset type, with corpus-proven fidelity.

How the surface was enumerated

nel/tools/build_gamedata/processes/*/1_export.py + their maxscript/*.ms scripts. Every Max-side export goes through one of these NelExport* MAXScript entry points (implemented in nel/tools/3d/plugin_max/nel_export/nel_export_script.cpp → nel_mesh_lib/): NelExportShapeEx, NelExportAnimation (used by BOTH the anim and shape processes), NelExportSkeleton, NelExportSkeletonWeight, NelExportInstanceGroup (ig + ligo), NelExportCollision (rbank + ligo), NelExportPACSPrimitives, NelExportVegetable, NelExportLodCharacter, plus ExportRykolZone (zone) and LigoExportZone (ligo). The ps process is a plain file copy (no Max involvement) despite its log line. Everything else under processes/ (rbank build, ig_light, zone_light, banks, maps, sheets, …) is post-export C++ tooling that already runs headless.

Reference outputs live in ~/pipeline_export (continents/ecosystems/common trees, mirroring the workspace config under ~/ryzomcore_leveldesign/workspace); sources in ~/ryzomcore_graphics.

Coverage table

Entry point Process(es) Output Refs in ~/pipeline_export Headless tool Status
ExportRykolZone zone .zone 6306 (incl. ligo) pipeline_max_export_zone GREEN — corpus 1068 exact + 130 x87-tier + 3 budget (§10h)
LigoExportZone ligo .zone/.ligozone bricks (in above) pipeline_max_export_zone --ligo GREEN — .ligozone 100% byte-identical; .zone same tiers as ExportRykolZone above (§10h)
NelExportSkeleton skel .skel 203 pipeline_max_export_skel GREEN — legacy 99.4% drot, era classes documented (§10, §10e)
NelExportSkeletonWeight swt .swt 1 (+ corpus) pipeline_max_export_swt GREEN (§10f)
NelExportInstanceGroup (standalone ig) ig .ig 54 direct (→ 18497 final, downstream via land_exporter, out of scope) pipeline_max_export_ig MOSTLY — T1/T2 116/116; 53/54 direct-ref field-exact under uninit-mask (0 truly byte-identical); 1 open cluster-link diff (TR_hall_reu_vitrine, budgeted --max-direct-diff 1, un-root-caused — the Mirror default-gizmo plane, §10g)
NelExportInstanceGroup (ligo bricks) ligo .ig 537 direct pipeline_max_export_ig --ligo MOSTLY — T1/T2 1201/1201; 448/537 (83.4%) match under uninit-mask (71 byte-identical + 377 uninit-only); 89 open, budgeted DIFF_BUDGET=89, un-root-caused (§10g-bis): ~65 cluster-membership (same vertex-in-volume defect as the standalone diff) + a selection-order class (same instance set, reordered → functionally equivalent but byte-different)
NelExportAnimation anim, shape .anim 9146 pipeline_max_export_anim MOSTLY — non-biped byte-identical; biped structural-exact, in-between IK bounded (median 0.069, §10r); gaps: texture-matrix material anims (waterfalls), biped exactness (§10s)
NelExportShapeEx shape .shape 11251 pipeline_max_export_shape PARTIAL — M1 plain-CMesh path only (704 float-eq + 989 lightmap-bucketed + 383 differ, §10i); skinned/MRM, MultiLod/coarse, water NOT attempted
NelExportCollision rbank, ligo .cmb 445 (16 genuine direct; the rest are a downstream non-Max copy, out of scope) pipeline_max_export_cmb MOSTLY — direct tier (the gated corpus): 16/16 files export, 10/16 igs exact + 3/16 x87-tier + 2/16 Edit-Mesh-created-geometry gap + 1/16 unresolved offset (§10v). Ligo tier (informational, ungated): 6/1201 files produce zero output (every collision node in the file is an unresolved XRefObject, or the file's only group fails outright for the same reason) — a real gap, not a diff-tier nuance; 22 more of the 48 exported igs differ, root cause unexplored
NelExportPACSPrimitives pacs_prim .pacs_prim 493 pipeline_max_export_pacs_prim GREEN — 493/493 byte-identical (§10v)
NelExportVegetable veget .veget 126 — MISSING
NelExportLodCharacter clodbank .clod 105 — MISSING

Cross-cutting infrastructure that is DONE and reusable: the storage/scene layer with full T1/T2 byte-identity over every corpus (max files parse→build byte-identical), the OLE backend (§2a), the VS2008/Wine x87 reference build (§2b), Edit Mesh / Editable Poly / Edit Patch geometry decode (ig/zone/shape sessions), materials + ParamBlock2 (§10j), the biped system incl. typed keytrack editing (§10t/§10u), AppData flag reading, and the corpus-test harness patterns (*_corpus.py, fork-per-file, tolerance tiers).

The gaps, sized

A. The remaining small formats (breadth completion)

.pacs_prim and .cmb landed 2026-07-08 (§10v) — .veget and .clod close the rest of the SURFACE.

  1. .cmb — NelExportCollision DONE, mostly (§10v) — pipeline_max_export_cmb. Open follow-ups: XRef collision-node resolution (--ligo mode), Edit Mesh created-face topology decode, one unresolved direct-tier vertex-offset file, triage the 22 ligo-tier diffs.
  2. .pacs_prim — NelExportPACSPrimitives DONE (§10v) — pipeline_max_export_pacs_prim, 493/493 byte-identical.
  3. .veget — NelExportVegetable (126 refs; 817 veget-related source .max). Serializer: NL3D::CVegetableShape (nel/3d/vegetable_shape.h) — mesh + bend-weight painting (vertex colors/UV1?) + appdata params (bend mode, freq…). Reference impl: nel_mesh_lib/export_vegetable.cpp (buildVegetableShape). Plain-mesh scale; the vertex-paint channel decode is the only likely new ground.
  4. .clod — NelExportLodCharacter (105 refs; 111 *_lod.max sources under stuff/lod_actors/). Serializer: NL3D::CLodCharacterShapeBuild (nel/3d/lod_character_shape.h) — positions, tris, UVs, normals + per-vertex skin weights to a bone-name table. Reference impl: nel_mesh_lib/export_lod_character.cpp. NOTE: this needs the SKIN modifier decode (bone weights) — the same decode the shape-MRM work needs; whichever session lands first should put the skin-weight reader in pipeline_max_export_common for the other to reuse. (The .swt exporter already reads skeleton-weight data — check what it shares before writing anything new.)

B. Shape depth (the flagship remaining work — Opus-sized)

NelExportShapeEx beyond plain CMesh, in rough dependency order:

  1. Skinned + MRM meshes (CMeshMRM) — characters, fauna; the bulk of the 11251 refs. Two halves: (a) the SKIN decode (Physique/Skin modifier chunks → per-vertex bone weights; corpus era is Physique — decode lives in nel_mesh_lib/export_skinning.cpp for reference) and (b) the MRM build (NL3D::CMRMBuilder) — deterministic NeL code; link the CURRENT NeL builder like anim/shape M1 already link NeL, then measure the 2004-vs-now divergence per file and bucket tolerance tiers exactly like §10h/§10i did (expect an era tier; the VS2008 build is the instrument if it matters).
  2. CMeshMultiLod / coarse meshes — the shape_with_coarse_mesh* reference dirs exist in ~/pipeline_export/common; the coarse-mesh texture step is part of the shape process.
  3. Water shapes (CWaterShape) — few files, self-contained (see the existing nel/3d/water_* docs and the waterfall/texture-matrix notes).
  4. Shape-process animations — shape_export.ms ALSO calls NelExportAnimation for animated materials: the waterfall texture-matrix gap (see waterfall-texmat-anim finding: bExportTextureMatrix + StdUVGen U/V Offset controllers; ref anims in ~/core4_data/*_shapes/waterfall*.anim). This is a small, well-scoped export_anim extension and can ride along with any session.

C. Anim depth (parked, documented)

The biped §10s experiment list (arm-pin space decode, in-plant foot rotation probe, turn-rule probe, blend-ramp weighting, PODA leads (6)–(8) in poda_1985_notes) — pick up only when biped exactness becomes the priority; the export is already structurally exact with bounded in-between error.

D. Ig export — open correctness items (was mislabeled GREEN/done)

.ig export is MOSTLY, not done — both the standalone (§10g) and ligo-brick (§10g-bis) paths carry open, un-root-caused diffs that the harnesses budget (--max-direct-diff 1, DIFF_BUDGET=89) so ctest exits 0. The budgets are regression guards; they are NOT a pass verdict, and the coverage table must never be read off the exit code. Current measured state (ig_corpus.py --all / ligo_ig_corpus.py --all, 2026-07-08): standalone 53/54 direct-ref match, 1 diff; ligo 448/537 match, 89 diff.

  1. Cluster-membership / vertex-in-volume containment (shared defect, the one that can be functionally wrong). The clusterize link test (accel&32 instance vs CCluster volume) returns a different containment verdict than the reference on borderline meshes. Surfaces as the standalone's single TR_hall_reu_vitrine cluster-link diff AND ~65 of the ligo path's 89 (Clusters field). Same root cause on both paths; not yet pinned. Cluster membership drives portal/visibility culling, so this is a real-content risk, not cosmetic. gen_decors_probe.ms (dumps the TR_hall_reu_vitrine node's evaluated mesh + per-modifier gizmo from Max) is the instrument to settle the standalone case; expect it to inform the ligo cases too.
  2. Ligo selection-order divergence (functionally equivalent, byte-different). A subset of ligo files (desert nb01..nb05, jungle foret-18..21_village_a/b/c/d, some ilot_butte) emit the same instance set in a different sequence — verified instance-by-instance on desert-nb01 (our instance 29 = ref instance 31; the two FY_Acc_Reverb PS entries sort before vs after the luciole block). Our walk = Scene-stream storage order; reference = Max's live per-category node array. Order doesn't affect the rendered ig, but blocks byte-identity. Same class as the shape exporter's "candide" duplicate-node-name ordering note (§10i).
  3. Byte-identity backlog (standalone). 0/54 standalone igs are truly byte-identical — all 53 "matches" rely on --mask-uninit (reference serializes uninitialized SunContribution/Light[] heap + STL-implementation-defined light-tie order). Legitimate tolerance class, but "field-exact" ≠ "byte-identical"; the Box/Plane/Cylinder first-triangle 1-ULP class is the last precision gap (§10g).

Whoever picks this up: retire the budgets by fixing the diffs, don't raise them. When a budget drops to 0, tighten the constant and re-grade the table row.

Suggested session plans

Session 1 (Sonnet-scale): .pacs_prim + .cmb. DONE 2026-07-08 (§10v) — pipeline_max_export_pacs_prim (493/493 byte-identical) and pipeline_max_export_cmb (16/16 direct files T1/T2, 13/16 T3 exact-or-tolerance, 3/16 documented open gaps; ligo tier informational). pacs_prim_corpus.py/cmb_corpus.py wired into ctest. Open follow-ups listed in §10v and item A.1 above.

Session 2 (Sonnet-scale): .veget, then .clod if the skin decode goes fast. .veget first (no skinning). For .clod, put the skin-weight decode in pipeline_max_export_common with its own mini-gate so the MRM session can build on it. If the Physique decode balloons, stop, document the chunk findings in the wiki, and leave .clod for the MRM session — do not half-land a skinning decode without a gate.

Session 3 (Opus-scale): skinned/MRM shapes. The big one. Approach exactly like §10i M1: replicate CExportNel::buildShape's MRM branch against the real NeL builders, corpus-bucket the outcomes (byte-exact / float-eq / era-tier / differ), and expect several probe rounds. Reuse the skin decode from session 2 if it landed. Success metric: the character/fauna shape corpus classified into explained tiers with zero unexplained structural fails — byte-identity is the stretch goal, not the bar (see the zone precedent).

Session 4 (Sonnet-scale, anytime): waterfall texture-matrix anims + shape-process anim wiring — small, high-polish-value, mostly export_anim plumbing.

Non-negotiables for every session (from §12, restated)

  • Load the session baseline first: all pipeline_max sources + the design doc (see the pipeline-max-session-baseline memory note / §12).
  • Corpus-gate EVERYTHING: T1/T2 byte-identity must stay green over the full .max corpus for any parser change; new exporters get their own T3 vs ~/pipeline_export references with explicit tolerance tiers; per-file forked runs (parse→clean→build→disown; COFile, not CMemStream).
  • The .ms scripts + workspace config (~/ryzomcore_leveldesign/workspace) are the ground truth for WHAT gets exported from each source file and with which options — read them before the C++.
  • Emission-form preservation, typed-chunk order preservation, and the §11 hazard list apply unchanged.
  • Don't rebuild binaries mid-sweep; don't trust per-file probes over full-corpus A/B (§10r method notes).

Clone this wiki locally