Skip to content

feat(mesh): DomainMeshZarrSink emitting the physicsnemo mesh-zarr schema#64

Open
negin513 wants to merge 2 commits into
NVIDIA:mainfrom
negin513:feat/domainmesh-zarr-sink
Open

feat(mesh): DomainMeshZarrSink emitting the physicsnemo mesh-zarr schema#64
negin513 wants to merge 2 commits into
NVIDIA:mainfrom
negin513:feat/domainmesh-zarr-sink

Conversation

@negin513

@negin513 negin513 commented Jul 7, 2026

Copy link
Copy Markdown
Member

Summary

Adds DomainMeshZarrSink: a mesh-domain sink that writes physicsnemo.mesh.DomainMesh (and plain Mesh) objects as Zarr group trees in the physicsnemo mesh-zarr schema — the chunked, compressed, random-access counterpart of MeshSink's .pdmsh/.pmsh memmap output. Together with the reader side (NVIDIA/physicsnemo#1798), this makes raw CFD data → training-ready zarr a one-hop curation: DrivAerMLSource (multi mode) already yields DomainMesh objects, so the sink is a drop-in alternative to MeshSink in the DrivAerML example pipeline.

Design

  • DomainMesh schema mirrors the .pdmsh tree: case-level global_data at the group root (single source of truth, merged at read time by the reader — never copied into member meshes), interior/ and boundaries/<name>/ as mesh-schema subgroups.
  • Verified layout attr: with soup_boundaries=True, boundary meshes are denormalized to a per-cell vertex soup so training-time contiguous block subsamples map to sequential, chunk-aligned reads (recommended for cell-centric surface training on meshes without point-index locality, e.g. DrivAerML). The attr is verified from the written cells, never caller-asserted; connectivity-dependent consumers use the default indexed layout.
  • Cell-axis chunking (chunk_cells) aligned with the training reader's subsample size; blosc-zstd compression (level 0 disables).
  • Curator conventions throughout: params() descriptors (wizard/serialization-visible), registered in the mesh-domain registry, atomic temp-then-rename writes for safe resume, {run_id} naming via set_source.

Schema contract

physicsnemo-domainmesh-zarr / physicsnemo-mesh-zarr, version 1, specified in physicsnemo/datapipes/MESH_ZARR_SCHEMA.md (see NVIDIA/physicsnemo#1798). The sink implements the schema self-containedly (no dependency on unreleased physicsnemo APIs); parity is enforced by a cross-repo round-trip test — sink output read back through physicsnemo's ZarrMeshReader with subpath navigation and read-time global_data merge — which skips gracefully on physicsnemo releases without the reader.

Validation

  • 8 tests in test/domains/mesh/test_mesh_domain_zarr.py (schema shape, soup relayout geometry preservation, compression, atomicity, naming, plain-Mesh items, registry, cross-repo round-trip).
  • End-to-end on real DrivAerML: converted cases (all 5 boundaries + 166M-point interiors + in-group full-resolution STL, 6.1 GB/case zstd vs 6.5 GB .pdmsh) train GeoTransolver surface and volume through physicsnemo's unified external aero recipe with losses matching the .pdmsh path; dataloading is ~2.5x faster (details and benchmarks in the companion PR).

Add a sink that writes DomainMesh (and plain Mesh) objects as Zarr group
trees in the physicsnemo mesh-zarr schema -- the chunked, compressed,
random-access counterpart of MeshSink's .pdmsh/.pmsh memmap output,
readable by physicsnemo's ZarrMeshReader / ZarrDomainMeshReader.

- DomainMesh schema mirrors the .pdmsh tree: case-level global_data at
  the group root (single source of truth), interior/ and
  boundaries/<name>/ as mesh-schema subgroups with verified layout
  attrs.
- Optional soup_boundaries denormalizes boundary meshes to a per-cell
  vertex soup so training-time contiguous block subsamples map to
  sequential, chunk-aligned reads (recommended for cell-centric surface
  training on meshes without point-index locality, e.g. DrivAerML).
- Cell-axis chunking (chunk_cells) aligned with the training reader's
  subsample size; blosc-zstd compression (level 0 disables).
- Atomic temp-then-rename writes for safe resume; registered in the
  mesh-domain registry; {run_id} naming via set_source.

Tests include a cross-repo round-trip (sink output read back through
physicsnemo's ZarrMeshReader with subpath navigation and read-time
global_data merge) that skips gracefully on physicsnemo releases
without the reader. Schema contract: physicsnemo-domainmesh-zarr v1,
specified in physicsnemo's physicsnemo/datapipes/MESH_ZARR_SCHEMA.md.
@codecov-commenter

codecov-commenter commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.40559% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.11%. Comparing base (4111fce) to head (59b8b31).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...emo_curator/domains/mesh/sinks/mesh_domain_zarr.py 94.24% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #64      +/-   ##
==========================================
- Coverage   63.39%   63.11%   -0.29%     
==========================================
  Files          88       89       +1     
  Lines        9488     9630     +142     
==========================================
+ Hits         6015     6078      +63     
- Misses       3473     3552      +79     
Flag Coverage Δ
integration 63.11% <94.40%> (-0.29%) ⬇️
unit 63.11% <94.40%> (-0.29%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…mantics

Align the sink with the schema-v1 contract tightened during review of
NVIDIA/physicsnemo#1798 (MESH_ZARR_SCHEMA.md):

- Write data subgroups (point_data/cell_data/global_data, incl. the
  case-level global_data at the DomainMesh root) via a recursive
  _write_tensordict helper: tensor leaves become arrays, nested
  TensorDicts become nested zarr groups, at any depth.
- Reject empty field names and names containing '/' (zarr path
  separator) at write time instead of silently creating hierarchy.
- Reject non-tensor leaves (NonTensorData/strings) at write time;
  an attrs-based encoding is reserved for schema v2.
- Pin zarr_format=3 explicitly so a zarr.config default override
  cannot produce a format-2 store.
- _to_cell_soup: gather point_data via TensorDict batch indexing
  (handles nested leaves, fixes ruff SIM118).

Tests: nested-TensorDict group structure, '/'-name and NonTensorData
rejection (with atomic-write cleanup), verified-soup layout attr,
cells dtype preservation, zarr-format-3 + plain-JSON attrs, and a
cross-repo check that physicsnemo's validate_mesh_zarr accepts sink
output (skips when unavailable, like the reader round-trip test).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants