feat(mesh): DomainMeshZarrSink emitting the physicsnemo mesh-zarr schema#64
Open
negin513 wants to merge 2 commits into
Open
feat(mesh): DomainMeshZarrSink emitting the physicsnemo mesh-zarr schema#64negin513 wants to merge 2 commits into
negin513 wants to merge 2 commits into
Conversation
Add a sink that writes DomainMesh (and plain Mesh) objects as Zarr group
trees in the physicsnemo mesh-zarr schema -- the chunked, compressed,
random-access counterpart of MeshSink's .pdmsh/.pmsh memmap output,
readable by physicsnemo's ZarrMeshReader / ZarrDomainMeshReader.
- DomainMesh schema mirrors the .pdmsh tree: case-level global_data at
the group root (single source of truth), interior/ and
boundaries/<name>/ as mesh-schema subgroups with verified layout
attrs.
- Optional soup_boundaries denormalizes boundary meshes to a per-cell
vertex soup so training-time contiguous block subsamples map to
sequential, chunk-aligned reads (recommended for cell-centric surface
training on meshes without point-index locality, e.g. DrivAerML).
- Cell-axis chunking (chunk_cells) aligned with the training reader's
subsample size; blosc-zstd compression (level 0 disables).
- Atomic temp-then-rename writes for safe resume; registered in the
mesh-domain registry; {run_id} naming via set_source.
Tests include a cross-repo round-trip (sink output read back through
physicsnemo's ZarrMeshReader with subpath navigation and read-time
global_data merge) that skips gracefully on physicsnemo releases
without the reader. Schema contract: physicsnemo-domainmesh-zarr v1,
specified in physicsnemo's physicsnemo/datapipes/MESH_ZARR_SCHEMA.md.
5 tasks
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #64 +/- ##
==========================================
- Coverage 63.39% 63.11% -0.29%
==========================================
Files 88 89 +1
Lines 9488 9630 +142
==========================================
+ Hits 6015 6078 +63
- Misses 3473 3552 +79
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
5 tasks
…mantics Align the sink with the schema-v1 contract tightened during review of NVIDIA/physicsnemo#1798 (MESH_ZARR_SCHEMA.md): - Write data subgroups (point_data/cell_data/global_data, incl. the case-level global_data at the DomainMesh root) via a recursive _write_tensordict helper: tensor leaves become arrays, nested TensorDicts become nested zarr groups, at any depth. - Reject empty field names and names containing '/' (zarr path separator) at write time instead of silently creating hierarchy. - Reject non-tensor leaves (NonTensorData/strings) at write time; an attrs-based encoding is reserved for schema v2. - Pin zarr_format=3 explicitly so a zarr.config default override cannot produce a format-2 store. - _to_cell_soup: gather point_data via TensorDict batch indexing (handles nested leaves, fixes ruff SIM118). Tests: nested-TensorDict group structure, '/'-name and NonTensorData rejection (with atomic-write cleanup), verified-soup layout attr, cells dtype preservation, zarr-format-3 + plain-JSON attrs, and a cross-repo check that physicsnemo's validate_mesh_zarr accepts sink output (skips when unavailable, like the reader round-trip test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
DomainMeshZarrSink: a mesh-domain sink that writesphysicsnemo.mesh.DomainMesh(and plainMesh) objects as Zarr group trees in the physicsnemo mesh-zarr schema — the chunked, compressed, random-access counterpart ofMeshSink's.pdmsh/.pmshmemmap output. Together with the reader side (NVIDIA/physicsnemo#1798), this makes raw CFD data → training-ready zarr a one-hop curation:DrivAerMLSource(multi mode) already yieldsDomainMeshobjects, so the sink is a drop-in alternative toMeshSinkin the DrivAerML example pipeline.Design
.pdmshtree: case-levelglobal_dataat the group root (single source of truth, merged at read time by the reader — never copied into member meshes),interior/andboundaries/<name>/as mesh-schema subgroups.layoutattr: withsoup_boundaries=True, boundary meshes are denormalized to a per-cell vertex soup so training-time contiguous block subsamples map to sequential, chunk-aligned reads (recommended for cell-centric surface training on meshes without point-index locality, e.g. DrivAerML). The attr is verified from the written cells, never caller-asserted; connectivity-dependent consumers use the default indexed layout.chunk_cells) aligned with the training reader's subsample size; blosc-zstd compression (level 0 disables).params()descriptors (wizard/serialization-visible), registered in the mesh-domain registry, atomic temp-then-rename writes for safe resume,{run_id}naming viaset_source.Schema contract
physicsnemo-domainmesh-zarr/physicsnemo-mesh-zarr, version 1, specified inphysicsnemo/datapipes/MESH_ZARR_SCHEMA.md(see NVIDIA/physicsnemo#1798). The sink implements the schema self-containedly (no dependency on unreleased physicsnemo APIs); parity is enforced by a cross-repo round-trip test — sink output read back through physicsnemo'sZarrMeshReaderwith subpath navigation and read-timeglobal_datamerge — which skips gracefully on physicsnemo releases without the reader.Validation
test/domains/mesh/test_mesh_domain_zarr.py(schema shape, soup relayout geometry preservation, compression, atomicity, naming, plain-Mesh items, registry, cross-repo round-trip)..pdmsh) train GeoTransolver surface and volume through physicsnemo's unified external aero recipe with losses matching the.pdmshpath; dataloading is ~2.5x faster (details and benchmarks in the companion PR).