Skip to content

v1.1.0: V2 Encoding Scheme

Choose a tag to compare

@trueb2 trueb2 released this 13 Dec 22:39
· 23 commits to main since this release
a517cfc

This release includes a large change. It represents a change in the encoding scheme to support Delta and Delta-Delta (de)compression. It marks a significant improvement to compression rates and ratios on real-world data. Additionally, the new encoding scheme is compatible with a second pass compression scheme like LZ4 for excellent compression ratios in small payloads and even better ratios in large payloads.

Adding a HalfVec lazy data structure and moving away from bitvec + deku leads to big performance and power improvements on embedded systems.

What's Changed

  • Add real world xyz accelerometer data index file compression benchmark by @trueb2 in #2
  • Refactor BitBuffer push usage during encode by @ameykasbe in #6
  • Fix clippy warning on unnecessary lifetime by @ameykasbe in #7
  • Add static ring buffer FIFO Queue by @trueb2 in #10
  • Update workspace resolver by @trueb2 in #11
  • Add CompressV2 proc macro by @trueb2 in #12
  • Add impl of ExactSizeIterator for CompressionQueueIter by @ameykasbe in #13
  • Delta Compression and Decompression by @ameykasbe in #14
  • Nibble-aligned Delta and Delta-Delta Encoding by @trueb2 in #18
  • Fix emission of 5 samples of 6 bits by @trueb2 in #19
  • Emit column start header prior to first row emission by @trueb2 in #20
  • Fix decoding with additional error messages by @trueb2 in #21
  • Handle decoding only no rows or one row by @trueb2 in #23
  • Add seperators in binary representations by @ameykasbe in #22
  • Update test cases by @ameykasbe in #24
  • Add emit delta i8 by @ameykasbe in #25
  • Fix bit masks and width while decoding by @ameykasbe in #30
  • Require compressor to track and expose the total number of rows consumed by @trueb2 in #31
  • Reduce branching during single row compression by @trueb2 in #32
  • Use power of two compression queue size by @trueb2 in #33
  • Hold zigzag bits in CompressionQueue by @trueb2 in #34
  • Add Test: Compress all possible deltas for i16 values by @ameykasbe in #35
  • Update unit test for HalfVec by @ameykasbe in #38
  • Add unit tests & related functionality for emit i32 delta by @ameykasbe in #36
  • Update CompressionQueue fuzz test by @ameykasbe in #37
  • Fix mod for Cargo Test pass by @ameykasbe in #39
  • Skip tracking number of column values emitted by @trueb2 in #40
  • Fix: Implement zigzag for each type and add tests by @ameykasbe in #41
  • Update queue zigzag to an array of usize. Add impl Bits for i64. by @ameykasbe in #42
  • Replace hardcoded header values with named constants by @ameykasbe in #44
  • Add field attributes in proc macros for delta by @ameykasbe in #43
  • Code Refinement by @ameykasbe in #45
  • Implement 64 bits 1 sample delta compression case by @ameykasbe in #46
  • Update README with changes from feature branch by @trueb2 in #47
  • Real world benchmark in a non-publishable bench crate by @trueb2 in #48
  • Nibble aligned encoding scheme implementation by @trueb2 in #8

New Contributors

Full Changelog: v1.0.4...v1.1.0