New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Small data compression performance #51

Merged

mwlon merged 5 commits into main from small-data-opt

Jul 24, 2022

Owner

mwlon commented Jul 24, 2022 •

edited

Reduced number of prefixes used for small data. Default compression ratio should balance compression time vs compressed size for all n (defaulting to 256 was especially slow in the 100 < n < 16k range). E.g. in the 2^10 <= n < 2^12 range this uses up to 64 prefixes by default.
Improved prefix optimization speed by about 30%.
Added a new dataset of decimal-valued floats

mwlon added 5 commits

July 24, 2022 09:16


          better compression levels for small data

8d16f05


          made prefix optimization up to 10% faster

dc89ce0


          miniscule improvement to comp ratio and 20% faster prefix optimization

702d55c


          renaming

38d3084


          rename

d73e8f6

mwlon merged commit 9bb9fb2 into main

mwlon deleted the small-data-opt branch

September 4, 2022 22:48

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment