Stream Tools #204
Merged
Stream Tools #204
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shaomeng
added a commit
that referenced
this pull request
Sep 13, 2023
* minor * WIP * WIP * finish integer SPECK3D encoder. Now implement a decoder * encoder and decoder complete * WIP * add header and more unit tests for SPECK3D_INT * add a truly random test * minor * WIP * WIP * add a function, change header definition * WIP * rename * SPERR3D_INT_Driver is working * add a mid-tread executable * experimental mid-tread compiles * mid-riser SPERR doesn't do outlier correction * minor * reorg, WIP * SPECK3D_INT_Driver passes unit tests * WIP: add subclass of SPECK3D_INT_Driver * SPERR3D compiles * class SPERR3D appears to be working * WIP * implement query of number of coded values * a fix on accounting the number of coded values * introduce base class SPECK_INT * minor * a small change to Set3D data structure * SPECK1D_INT compiles * SPECK1D_INT passes unit test * WIP: add base class SPECK_INT_Driver * WIP: add SPERR_Driver * WIP: keep working on SPERR3D unit tests * WIP * alias name change * SPERR3D works * sync * incr * incr * remove a bunch of files from this branch. Will need to focus on the integer implementation * ignore install directory * add ZFP bitstream files * added ZFP_bitstream * minor * add random read to ZFP_bitstream * add read_n_bits function * improvement on ZFP_stream * add compact stream generate and parse functions * ZFP_bitstream allocates its handle on stack now * more improvements * use zfp::stream_open() again * improve ZFP_bitstream * resolve ZFP bitstream naming and linking issues * minor improvement * minor improvements * minor * minor * minor * Custom bitmask for m_LSP_mask (#197) * minor * use bitmask for m_LSP_mask * add Bitmask * use a custom bitmask to store m_LSP_mask data structur, and actually speed up the program quite significantly! * clang-format * clang-format * clang-format * clang-format --------- Co-authored-by: Samuel Li <Sam@Navada> * minor * Use Sam's implementation of Bitstream * improve documentation * Improve API, so get_bitstream() is const now * implement write_bitstream() function in Bitstream * update Bitstream API * minor * minor * Fast bit buffer (#198) * WIP * class SPECK_INT compiles * WIP: SPECK3D_INT compiles, need to work on SPECK1D_INT_ENC * using Bitstream in SPECK3D and SPECK1D * minor * clang-format * clang-format --------- Co-authored-by: Samuel Li <shaomeng@cisl-vapor> Co-authored-by: Samuel Li <Sam@Navada> * minor API change * clean up * bring encode() and decode() functions to the base SPECK_INT class * minor * SPECK_INT compiles * test branch * Revert "test branch" This reverts commit 18e39f6. * Uint template (#202) * SPECK libraries compile * Still working on SPERR_Driver. Will need to give it a header when assembling bitstreams. Need to record the length of uint in use. * improve SPECK_INT * improve documentation * add function to report integer type * minor * WIP: SPERR_Driver needs to be able to select integer length * SPERR_Driver compiles * SPERR3D compiles, but doesn't link... * template classes compile * WIP * minor: comments * Bitmask can export and import a bitstream * add unit test for Bitmask * re-enable unit tests for integer speck * improv. on Bitmask --------- Co-authored-by: Samuel Li <Sam@Navada> Co-authored-by: Samuel Li <sam@cisl-m121a> * WIP * rename so that we have SPECK_FLT classes! * SPECK3D_FLT runs well * minor udpate to sperr_helper; better in sync with the main branch * Use the Conditioner from the main branch * remove the keyword virtual in a few places * API change: append to a bitsteram always * add function to SPECK_FLT to retrieve integer length in use * add more unit test * a little cleaner code * minor optimization * WIP: Outlier_Coder * Outlier_Coder compiles * minor interface change * Outlier_Coder compiles * finish unit test for outlier coder * add a note on bitstream definitions * add Outlier_Coder bitstream definition * WIP: SPECK_FLT also corrects outlier * add functions so Outlier Coder can report its stream length * SPECK_FLT compiles * WIP, want to restructure SPECK_FLT * minor * SPECK_INT can correctly handle all zero inputs * outlier correction working! * clang-tidy * clean up * SPECK3D_FLT add m_estimate_mse() method * update SPECK_FLT bitstream definition * fix an integer overflow bug in SPECK_INT.cpp * SPECK_FLT class is more polished * Outlier Coder uses a biased inverse quantization scheme * minor * WIP: working on SPERR3D_OMP_C * SPERR3D_OMP_C compiles * minor * SPERR3D_OMP_D compiles * WIP: debug a variance issue... * Fix some bugs, but still crashes * fix a nasty bug of not clearing LIP in SPECK1D_INT * use the same order in variants in SPECK_FLT * SPERR3D_OMP passing unit tests * clean up return error values * remove ZSTD usage * API change of SPERR3D_OMP_C * minor * use std::copy() * WIP: sperr3d.cpp * finish settings * sperr3d working! * use optional in return values * minor * lower memory high-water mark when calculating stats * minor API change * minor * LIS mask (#203) * improve Bitmask API * update Bitmask unit test * WIP: m_LIP_mask, there's seg fault still * fixes * fix bug, pass all unit tests * minor improvement * make compiler happy * remove unused variable --------- Co-authored-by: Samuel Li <sam@cisl-m121a> Co-authored-by: Samuel Li <Sam@Navada> Co-authored-by: Samuel Li <shaomeng@cisl-vapor> * minor * minor * fix integer overflow * minor * WIP: SPECK2D_INT * minor improv on SPECK3D_INT * WIP: SPECK2D_INT * minor improvement on SPECK3D_INT * WIP on SPECK2D_INT * minor improv * SPECK3D_INT finishes * minor * minor * SPECK2D_INT_ENC compiles * remove variable m_bit_idx * improve on comments * minor change * SPECK2D_INT_DEC compiles * SPECK2D_INT pass unit tests * clang-format * SPECK2D_INT is in * remove a few virtual functions * SPECK2D_FLT compiles * add SPECK2D_FLT unit tests * sperr2d compiles; need to do some tests! * sperr2d works! * utilities can perform decompression tasks too * CLI11 precompiled is default ON now * remove the option to use old GoogleTest * minor * change header definition; using 1 bit to indicate portion of a bitstream * add read_sections() function to sperr_helper * Stream Tools (#204) * add function get_header_len() * introduce new class: SPERR3D_Stream_Tools * minor, robust improv * SPERR3D_OMP_D now makes use of SPERR3D_Stream_Tools * SPERR3D_Stream_Tools compiles * stream_tools pass unit tests --------- Co-authored-by: Samuel Li <Sam@Navada> * More re-work on the code to prepare for the implementation of progressive access. (#205) * add function get_header_len() * introduce new class: SPERR3D_Stream_Tools * minor, robust improv * SPERR3D_OMP_D now makes use of SPERR3D_Stream_Tools * SPERR3D_Stream_Tools compiles * stream_tools pass unit tests * WIP * add more Bitstream unit test * minor * WIP * Conditioner doesn't support custom filter anymore * Conditioner stream has constant size now! * minor --------- Co-authored-by: Samuel Li <Sam@Navada> Co-authored-by: Samuel Li <shaomeng@cisl-vapor> * minor * Bitmask is able to count_true() * Add reset() function to Bitstream * minor * Progressive (#208) * add function get_header_len() * introduce new class: SPERR3D_Stream_Tools * minor, robust improv * SPERR3D_OMP_D now makes use of SPERR3D_Stream_Tools * SPERR3D_Stream_Tools compiles * stream_tools pass unit tests * WIP * add more Bitstream unit test * minor * WIP * Conditioner doesn't support custom filter anymore * Conditioner stream has constant size now! * minor * update SPERR.pc * WIP * WIP * minor * minor * I think SPECK_INT supports partial decoding now * I think SPECK_FLT also supports partial decoding now * Handle the situation where only part of the outlier correction bitstream is available. * file re-organization * WIP sperr3d_trunc utility * SPERR3D_Stream_Tools doesn't look correct... Need to work on this first. * fix up SPERR3D_Stream_Tools * utility sperr3d_trunc seems working * minor * The progressive access API takes a percentage of bitstream to read. * minor * minor * clang-format * clang-format --------- Co-authored-by: Samuel Li <Sam@Navada> Co-authored-by: Samuel Li <shaomeng@cisl-vapor> Co-authored-by: Samuel Li <sam@cisl-m121a> * sperr3d_trunc release more memory * add code testing double values having less precision than ints * re-work on some data members * One approach to achieve fixed-rate compression (#209) * SPECK_FLT now calculates m_q in fixed bitrate mode * WIP * SPECK_INT and SPECK_FLT both support a bit budget now * target BPP seems to be working! * add sperr2d Target BPP unit test * sperr3d supports fixed-bitrate too * adjust output format * minor * minor * minor * clang-format --------- Co-authored-by: Samuel Li <Sam@Navada> Co-authored-by: Samuel Li <sam@cisl-m121a> * add clearer comment * Improved progressive decoding (#210) * minor * improve logic of progressive decoding * Implement the new decoding strategy! * minor * implement logic to use uint32_t first in fixed-rate mode, and if not enough bits produced, use uint64_t * make sure we're comparing num of bits * refinement_pass_decode terminates when bitstream is exhausted * bring back fixed rate unit tests * minor --------- Co-authored-by: Samuel Li <Sam@Navada> Co-authored-by: Samuel Li <sam@cisl-m121a> * tweak quantization formula when decoding * very minor * use raw loops instead of std::find_if() in set significance tests * Revert "use raw loops instead of std::find_if() in set significance tests" This reverts commit 9b5c5ef. * Morton (#211) * minor change to the logic of choosing between dyadic and wavelet_packets 3D wavelet transforms * working version! * remove unused function: decide_significance * minor improvement * Set3D uses 16-bit integers to record integers * use a more compact representation of morton offset, and also remove the use of SetType * bring the Set3D object size down to 18 bytes! * minor * minor * minor * minor * move 3 functions from 3D encoder/decoder to SPECK3D_INT; reduce code redundancy * minor * remove the use of SetType in SPECK2D_INT * Remove the use of SetType from SPECK1D, thus from the codebase entirely * reduce the size of Set1D down to 16 bytes * minor * minor * remove [[likely]] * minor reformatting * m_process_P() checks pixel significance from the morton buffer too * clang-format * minor --------- Co-authored-by: Samuel Li <shaomeng@cisl-vapor> Co-authored-by: Samuel Li <sam@cisl-m121a> * minor * remove custom filter * minor * remove unused test data files * various changes during review * clang-tidy * remove unused variables * remove the use of std::vector<bool> * code style improvement on quantization and inverse quantization * better use of std::visit() in Outlier_Coder * minor * minor --------- Co-authored-by: Samuel Li <shaomeng@cisl-vapor> Co-authored-by: Samuel Li <Sam@Navada> Co-authored-by: Samuel Li <sam@cisl-m121a>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP to achieve progressive access. This PR completes
SPERR3D_Stream_Toolsclass.