Skip to content

Implement baseline ReflectionTable with eager HDF5 loading#10

Merged
dimitrivlachos merged 38 commits intomainfrom
refl
May 15, 2025
Merged

Implement baseline ReflectionTable with eager HDF5 loading#10
dimitrivlachos merged 38 commits intomainfrom
refl

Conversation

@dimitrivlachos
Copy link
Collaborator

@dimitrivlachos dimitrivlachos commented Apr 17, 2025

This PR introduces a canonical implementation of the ReflectionTable,
where all datasets are eagerly loaded into memory. While not optimized
for large-scale performance, this version provides a correct and
complete foundation for future enhancements such as lazy or chunked
loading. It simplifies the data model and establishes the foundation for
the public API and testing interface.

  • Introduce a ReflectionTable class that eagerly loads HDF5 datasets
    into memory.
  • Add support for extracting dataset shape via
    read_array_with_shape_from_h5_file().
  • Refactor h5read_processed.hpp to support recursive group traversal,
    metadata extraction, and shape-aware reading.
  • Add write_raw_data_to_h5_file() and related helpers in h5write.hpp
    for raw and high-level writing with shape/attribute support.
  • Add h5dispatch.hpp to centralize C++/HDF5 type dispatch and
    registration.
  • Integrate the mdspan backport for flexible memory views.
  • Add DX2_DEBUG-controlled logging via utils.h.
  • Add comprehensive unit tests covering filtering, writing, round-trip
    loading, and column type access.

@dimitrivlachos dimitrivlachos force-pushed the refl branch 2 times, most recently from 0d161f8 to 5e62e92 Compare April 22, 2025 16:54
@dimitrivlachos dimitrivlachos force-pushed the refl branch 2 times, most recently from 8c60fd5 to faa1918 Compare April 25, 2025 13:02
@dimitrivlachos dimitrivlachos changed the title Refl Reflection Table Apr 29, 2025
@dimitrivlachos dimitrivlachos changed the title Reflection Table Implement baseline ReflectionTable with eager HDF5 loading Apr 29, 2025
@dimitrivlachos dimitrivlachos self-assigned this Apr 29, 2025
@dimitrivlachos dimitrivlachos added the enhancement New feature or request label Apr 29, 2025
@dimitrivlachos dimitrivlachos force-pushed the refl branch 5 times, most recently from 042fd7f to b8a9fde Compare May 1, 2025 12:49
@dimitrivlachos
Copy link
Collaborator Author

Yes, you are correct @jbeilstenedmands. I am currently working on this while integrating this PR into the Fast Feedback Service and will push those changes once they are working.

@dimitrivlachos dimitrivlachos force-pushed the refl branch 2 times, most recently from d682a67 to d9ba0c7 Compare May 8, 2025 12:50
Copy link
Contributor

@jbeilstenedmands jbeilstenedmands left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for undertaking this extensive piece of work, this looks really nice and functions well. I have verified that I can create and save h5 files that can be read by dials, so I think this can now be merged in.

@dimitrivlachos dimitrivlachos force-pushed the refl branch 2 times, most recently from f312cd6 to e924310 Compare May 9, 2025 16:41
@dimitrivlachos dimitrivlachos merged commit 2c85855 into main May 15, 2025
5 checks passed
@dimitrivlachos dimitrivlachos deleted the refl branch May 15, 2025 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants