add bitshuffle support, compress binned datasets using bitshuffle/LZ4 #125
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
YooSunYoung
approved these changes
Mar 16, 2025
Member
YooSunYoung
left a comment
YooSunYoung
left a comment
There was a problem hiding this comment.
Thank you! I just added some docstring based on your issue and then an option argument to not-compress the dataset.
We typically try not to keep the full control over this kind of side-effect/advanced settings.
But for conveniences, I kept the compression True as default.
Member
|
Fixes #124 |
Collaborator
Author
|
in nexus.py, you write:
This is not necessarily true; after all, each image is three panels of (1280,1280,50) datasets. That's ~2e8 integers. Whether those counts are all zero, the dataset will still be big. Fortunately, it is easy to compress big sets of integers like this. Bitshuffle is especially good at it. |
YooSunYoung
added a commit
that referenced
this pull request
Mar 25, 2025
…#125) * add bitshuffle support, compress binned datasets using bitshuffle/LZ4 * Apply automatic formatting * Add docstring and option. --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: YooSunyoung <luysunyoung9@gmail.com>
YooSunYoung
added a commit
that referenced
this pull request
Mar 27, 2025
* Executable module. * Save crystal rotation. * Fix crystal rotation. * Remove all zero lines, not just the first one (#123) * remove all zero lines, not just the first one * Apply automatic formatting * Update src/ess/nmx/mcstas/load.py * Apply automatic formatting --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Sunyoung Yoo <luysunyoung9@gmail.com> * Apply suggestions from code review Co-authored-by: Mridul Seth <mail@mriduls.com> * Apply automatic formatting * Add bitshuffle support, compress binned datasets using bitshuffle/LZ4 (#125) * add bitshuffle support, compress binned datasets using bitshuffle/LZ4 * Apply automatic formatting * Add docstring and option. --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: YooSunyoung <luysunyoung9@gmail.com> * Update mtz_io.py make default space group P1 (lowest symmetry space group) * Update default space group in tests. --------- Co-authored-by: YooSunyoung <luysunyoung9@gmail.com> Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Mridul Seth <mail@mriduls.com>
YooSunYoung
added a commit
that referenced
this pull request
Mar 27, 2025
* Executable module. * Save crystal rotation. * Fix crystal rotation. * Remove all zero lines, not just the first one (#123) * remove all zero lines, not just the first one * Apply automatic formatting * Update src/ess/nmx/mcstas/load.py * Apply automatic formatting --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Sunyoung Yoo <luysunyoung9@gmail.com> * Apply suggestions from code review Co-authored-by: Mridul Seth <mail@mriduls.com> * Apply automatic formatting * Add bitshuffle support, compress binned datasets using bitshuffle/LZ4 (#125) * add bitshuffle support, compress binned datasets using bitshuffle/LZ4 * Apply automatic formatting * Add docstring and option. --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: YooSunyoung <luysunyoung9@gmail.com> * Update default space group in mtz io module (#126) * Executable module. * Save crystal rotation. * Fix crystal rotation. * Remove all zero lines, not just the first one (#123) * remove all zero lines, not just the first one * Apply automatic formatting * Update src/ess/nmx/mcstas/load.py * Apply automatic formatting --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Sunyoung Yoo <luysunyoung9@gmail.com> * Apply suggestions from code review Co-authored-by: Mridul Seth <mail@mriduls.com> * Apply automatic formatting * Add bitshuffle support, compress binned datasets using bitshuffle/LZ4 (#125) * add bitshuffle support, compress binned datasets using bitshuffle/LZ4 * Apply automatic formatting * Add docstring and option. --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: YooSunyoung <luysunyoung9@gmail.com> * Update mtz_io.py make default space group P1 (lowest symmetry space group) * Update default space group in tests. --------- Co-authored-by: YooSunyoung <luysunyoung9@gmail.com> Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Mridul Seth <mail@mriduls.com> --------- Co-authored-by: Aaron Finke <45569605+aaronfinke@users.noreply.github.com> Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Mridul Seth <mail@mriduls.com>
YooSunYoung
added a commit
that referenced
this pull request
Mar 27, 2025
* Lauetof export interface. * Raw data metadata as dataclass * Allow arbitrary metadata and export time of flight from the coordinate. * Specify unit Co-authored-by: Simon Heybrock <12912489+SimonHeybrock@users.noreply.github.com> * Add docstring to export methods. * Add missing attributes. * Remove comments * Write unit only when applicable. * Update comment. * Validity check when appending histogram on top of metadata (#116) * Separate metatadata from event data for easy export. * Add raw data metadata retrieval part. * Lauetof export interface. * Raw data metadata as dataclass * Allow arbitrary metadata and export time of flight from the coordinate. * Separate metatadata from event data for easy export. * Add raw data metadata retrieval part. * Satety check in the export function. * Add warning filter. * Apply automatic formatting * Apply automatic formatting * Fix typo * Move functions to more proper module. * Lauetof export interface. * Raw data metadata as dataclass * Allow arbitrary metadata and export time of flight from the coordinate. * Specify unit Co-authored-by: Simon Heybrock <12912489+SimonHeybrock@users.noreply.github.com> * Add docstring to export methods. * Add missing attributes. * Remove comments * Fix typo. * Apply automatic formatting --------- Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Simon Heybrock <12912489+SimonHeybrock@users.noreply.github.com> * Wrap detector processing step. * Lower pin of essreduce to use accumulators. * Specify dtype of string. [skip ci] * Use essreduce accumulator. * Export NXsource. * Fix type hint * Apply automatic formatting * Data reduction wrapper interface (#122) * Executable module. * Save crystal rotation. * Fix crystal rotation. * Remove all zero lines, not just the first one (#123) * remove all zero lines, not just the first one * Apply automatic formatting * Update src/ess/nmx/mcstas/load.py * Apply automatic formatting --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Sunyoung Yoo <luysunyoung9@gmail.com> * Apply suggestions from code review Co-authored-by: Mridul Seth <mail@mriduls.com> * Apply automatic formatting * Add bitshuffle support, compress binned datasets using bitshuffle/LZ4 (#125) * add bitshuffle support, compress binned datasets using bitshuffle/LZ4 * Apply automatic formatting * Add docstring and option. --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: YooSunyoung <luysunyoung9@gmail.com> * Update default space group in mtz io module (#126) * Executable module. * Save crystal rotation. * Fix crystal rotation. * Remove all zero lines, not just the first one (#123) * remove all zero lines, not just the first one * Apply automatic formatting * Update src/ess/nmx/mcstas/load.py * Apply automatic formatting --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Sunyoung Yoo <luysunyoung9@gmail.com> * Apply suggestions from code review Co-authored-by: Mridul Seth <mail@mriduls.com> * Apply automatic formatting * Add bitshuffle support, compress binned datasets using bitshuffle/LZ4 (#125) * add bitshuffle support, compress binned datasets using bitshuffle/LZ4 * Apply automatic formatting * Add docstring and option. --------- Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: YooSunyoung <luysunyoung9@gmail.com> * Update mtz_io.py make default space group P1 (lowest symmetry space group) * Update default space group in tests. --------- Co-authored-by: YooSunyoung <luysunyoung9@gmail.com> Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Mridul Seth <mail@mriduls.com> --------- Co-authored-by: Aaron Finke <45569605+aaronfinke@users.noreply.github.com> Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Mridul Seth <mail@mriduls.com> --------- Co-authored-by: Simon Heybrock <12912489+SimonHeybrock@users.noreply.github.com> Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com> Co-authored-by: Aaron Finke <45569605+aaronfinke@users.noreply.github.com> Co-authored-by: Aaron Finke <aaron.finke@ess.eu> Co-authored-by: Mridul Seth <mail@mriduls.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added bitshuffle support and compression of reduced datasets with bitshuffle/lz4