add: RSM Volume Builder — merge scan(s) into a gridded reciprocal-space volume

[pecomyint]

Summary

Adds the one thing rsMap3D can do that DashPVA could not: gridding every frame of one or more completed scans onto a single interpolated 3D reciprocal-space volume (xu.Gridder3D). We already converted angles to Q per frame, but could only show a bounded live point cloud — never a merged volume.

Launch with DashPVA rsmgrid or the Post Analysis launcher tile. Output uses the existing HDF5 volume format, so results open in Workbench's 3D viewer.

Design notes worth reviewing

• Two passes are required, not an optimization. Gridder3D latches fixed_range on its first call when KeepData is set, after which points outside the first batch's range are silently dropped. Global bounds must be known and set via dataRange(fixed=True) before any gridding.
• Masked pixels are excluded, never zeroed. Gridder3D.data is a per-bin mean, so a zeroed-but-present pixel would count as a real intensity = 0 measurement and bias its voxel low.
• Energy/UB differences warn, they don't block. Each file's own UB is applied, so the merge already lands in a common crystal-fixed HKL frame — that's why rsMap3D applies UB per scan and doesn't block either. HKL is also geometrically energy-independent.
• Axes are labelled H/K/L, not Q — passing UB to Ang2Q.area returns hkl, not Q in Å⁻¹.
• Optional I0 normalization from entry/data/metadata/ca/<name>, matching rsMap3D's monitor division. Without it, scans at different exposure/attenuation won't match in intensity where they overlap.
• Batch size comes from a byte budget, not a frame count — a fixed 32 frames needs ~6 GB of coordinate arrays on a 2048² detector.
• grid_origin is shifted half a voxel: Gridder3D's axes are bin centers, but the PyVista viewer treats the origin as a cell corner.
• The mask is stored in the viewer's display orientation and that state is never persisted, so the dialog asks rather than guessing.

Memory safety

The dense result is unavoidably nx · ny · nz voxels — Gridder3D holds two float64 arrays and .data materializes a third, so ~24 B/voxel for the grid plus one processing batch.

estimate_grid_memory() computes a conservative peak from the grid plus the largest per-file batch, and ensure_memory_available() refuses a build exceeding a safe fraction of currently available RAM. The dialog shows the estimate live as nx/ny/nz change (detector shapes are read once and cached, not re-read per keystroke) and blocks Start rather than letting the process OOM.

Thresholds live in settings.py per the "no inline config" rule. RSM_GRID_WORKING_BYTES_PER_PIXEL drives both the estimator and _batch_size_for, so the two cannot drift apart. Scripted/HPC callers can bypass the guard with build_volume(..., memory_limit_fraction=None).

Physics validation

Metadata that the conversion treats as static — photon energy, detector center/size/distance, beam and sample reference directions, UB — is now validated for consistency across frames, along with finiteness, positivity, non-degeneracy, and UB rank.

Tolerances are settings-driven and sized for EPICS readback jitter: these values are written per-frame from live PVs (hdf5_writer._write_scan_pv_dataset), so an exact-equality check would reject valid scans. Energy uses RSM_GRID_ENERGY_RELATIVE_TOLERANCE (1e-4, ≈1 eV at 10 keV); other static metadata uses RSM_STATIC_METADATA_*. A genuinely swept value is still rejected, since one static geometry cannot represent it.

Non-finite coordinates and intensities are excluded from binning and reported. Bounds come from finite coordinates only, so the binned set is always a strict subset of the bounded set — nothing can silently fall outside the latched range.

Circles beyond 4 sample / 2 detector are rejected explicitly, citing #132, rather than silently truncated.

Provenance

Output metadata now records per-file energies and UB matrices (source_energies_eV, source_ub_matrices, source_ub_matrices_shape) rather than only the first file's, plus coordinate_system, gridder, monitor_dataset, mask state, and excluded-point counts. A merge across files with differing energy/UB stays auditable after the fact.

Breaking changes in utils/rsm_converter.py

build_file_geometry() / q_for_frames() are additive — QConversion/HXRD/init_area are built once per file instead of once per frame, using xrayutilities' batched angle arguments. area_det_viewer.py and hpc_rsm_consumer.py have their own inline setups and are untouched.

Signatures are preserved, but three behaviors changed deliberately:

1. The cross-circle POSITION fallback is removed. When an axis group had no POSITION dataset, the old code walked the whole HKL tree and returned whichever POSITION it found first — i.e. a different circle's angle, silently corrupting HKL. It now raises.
2. Detector centers are float, not int — subpixel centers are no longer truncated.
3. Static-metadata validation is applied where the old code just took element [0].

These affect create_rsm() and the Workbench point-cloud path, not only the new gridder.

Workbench volume rendering (consumer side)

workspace_3d.py gains build_volume_grid() and volume detection in load_data(), reusing the existing cloud_volume actor name that on_3d_colormap_changed and update_intensity already handle. Worth reviewing as a distinct concern — it's included because the gridder's output is unviewable without it.

Separate commit

687e943 fixes a pre-existing bug found while testing: the HDF5 metadata readers used hasattr(ds, 'asstr') to detect string datasets, but every Dataset has that method — it only raises on numeric ones, and the error was swallowed. voxel_spacing/grid_origin/grid_dimensions_cells/intensity_range were coming back as reprs like '[6 6 6]'. Reviewable/cherry-pickable on its own.

Testing

252 tests pass (34 new). ruff check src/dashpva/ is clean.

Grid dimensions in tests are deliberately asymmetric — with nx==ny==nz a transpose bug still produces a correctly-shaped array.

Fixture correction: _synthetic_hdf5.py previously wrote SIZE and CENTER_CHANNEL_PIXEL direction-2-first, contradicting all three production paths (rsm_converter.py:157, area_det_viewer.py:2013-2017, hpc_rsm_consumer.py:271-274, which all pair SIZE[0] with detector direction 1). Square fixtures hid it; the new asymmetric (3,5) fixture catches it.

test_asymmetric_detector_matches_independent_reference checks the batched path against a hand-built xrayutilities pipeline on a non-square detector, catching wiring and argument-order bugs in build_file_geometry/q_for_frames that the old batched-vs-per-frame test structurally could not — both sides shared any bug. Note it builds its reference with the same cch1/cch2/Nch1/Nch2 convention the converter assumes, so it does not resolve the detector axis-0 question below.

Other coverage: hot-pixel→voxel round trip through save/load, mask exclusion vs zeroing, mask transpose, I0 normalization, cell-corner origin, a guard on the fixed_range latch we depend on, EPICS-jitter acceptance vs swept-value rejection, memory estimation and guard bypass, legacy MU/DELTA circle names, and the Workbench grid builder.

Widget-level regression caught in review

The Min/Max intensity controls were plain integer QSpinBox in workspace_3d.ui, and _load_volume seeded them with floats — which PyQt5 rejects outright (TypeError: setRange(self, min: int, max: int)), so every volume load ended in "3D Load Failed". The unit tests passed over it because they only covered build_volume_grid(), the pure helper.

This matters specifically for gridded output: Gridder3D.data is a per-bin mean, not a raw count, and with I0 normalization the whole range can sit below 1 — an integer box would collapse 0.0034 → 12.57 to 0 → 12, or to an empty 0 → 0.

Both widgets are now QDoubleSpinBox, seeded by seed_intensity_spinboxes() which picks decimals from the span (2 for counts, 6 for normalized means). The regression test loads the shipped .ui headless and asserts a fractional range round-trips rather than raising or truncating.

Disjoint from #127 — zero file overlap.

Not yet validated on real data

• Detector axis-0 convention (cch1/pw1 vs row/col) against a known Bragg peak — pre-existing in rsm_converter, but now load-bearing. A swap would produce a transposed volume that still looks plausible. Only real data closes this.
• Half-voxel origin, visually in Workbench.
• Which metadata/ca/ key is I0 at the beamline.
• DISTANCE validates at rtol=1e-6 (0.5 µm on a 500 mm distance). Fine for a static setpoint; if a beamline exposes it as a live encoder readback it will raise a clear error. Commissioning item, not a correctness risk.

Known gaps vs rsMap3D

No flat-field correction, no user-specified grid crop, no detector tilt (pre-existing rsm_converter limitation), and no out-of-core gridding — the volume is held entirely in RAM, so a ~1000³ grid (~24 GB) is rejected by the memory guard rather than degrading to chunked accumulation.

[pecomyint]

@Osayi-ANL Please let me know if you have started working on this. Open to hearing your thoughts on this since you have extensively worked on 3D HKL work. This is a new feature that beamlines would find useful.

[pecomyint]

@Osayi-ANL when you have a chance, please work on this.

[Osayi-ANL]

Doesn't render on the workbench for post analysis