C Makefile
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
include
src tests have been moved to own directory (better modularity) Apr 14, 2016
tests
.gitignore
.travis.yml
LICENSE
README.md
example.c
makefile

README.md

MaskedVByte

Build Status

Fast decoder for VByte-compressed integers in C.

It includes fast differential coding.

It assumes a recent Intel processor (e.g., haswell) but should work with most x64 processors (supporting SSE instruction sets).

The code should build using most standard-compliant modern C compilers (C99). The provided makefile expects a Linux-like system.

Usage:

  make
  ./unit 

See example.c for an example.

Short code sample:

size_t compsize = vbyte_encode(datain, N, compressedbuffer); // encoding
// here the result is stored in compressedbuffer using compsize bytes
size_t compsize2 = masked_vbyte_decode(compressedbuffer, recovdata, N); // decoding (fast)

Interesting applications

Greg Bowyer has integrated Masked VByte into Lucene, for higher speeds :

https://github.com/GregBowyer/lucene-solr/tree/intrinsics

Reference

  • Daniel Lemire, Nathan Kurz, Christoph Rupp, Stream VByte: Faster Byte-Oriented Integer Compression, Information Processing Letters (to appear) https://arxiv.org/abs/1709.08990
  • Jeff Plaisance, Nathan Kurz, Daniel Lemire, Vectorized VByte Decoding, International Symposium on Web Algorithms 2015, 2015. http://arxiv.org/abs/1503.07387

See also