number-codec

Lightweight implementation of popular lossless numerical compression algorithms

Support encodings

• Gorilla: Fast lossless float point number compression
• Chimp: Adaptive lossless float point number compression
• VarInt: Variable-length integer encoding for unsigned integer
• ZigZag: Variable-length integer encoding for signed integer
• Simple8: Packing multiple integers into a single 64-bit word
• Delta2: Store the difference between consecutive values only

When could you benefit from it ?

These algorithms are particularly useful in the realm of time-series data:

• Trading messages in financial markets
• IoT network data streams generated by sensors
• System metrics collected by performance monitor

In our case, the data volume can be reduced over 30% by applying these compression algorithms. This approach effectively slashes network bandwidth consumption and curtails disk storage expenses, ultimately trimming down the company's infrastructure overheads.

Why should you try it ?

• Pure Java implementation without any external dependency
• Efficient memory access through Unsafe
• Deeply optimized for Simple8 encoding

How to use it ?

1. Import maven dependency

<project>

    <dependencies>
        <dependency>
            <groupId>com.curioloop</groupId>
            <artifactId>number-codec</artifactId>
            <version>1.2.0</version>
        </dependency>
    </dependencies>

</project>

2. Try the CodecHelper or custom you own workflow (recommend) like this

public static void main(String[] args) {

    CodecSlice slice = new CodecSlice();
    CodecBuffer buffer = new CodecBuffer(128);

    // Encode and decode with delta2
    CodecResult cr1 = CodecHelper.encodeDelta2(i -> i, 10000, buffer);
    CodecHelper.decodeDelta2(slice.wrap(cr1.data()), cr1.codecs(), Assertions::assertEquals);

    // Encode and decode integers
    CodecResult cr2 = CodecHelper.encodeInt(i -> i, 10000, true, buffer);
    CodecHelper.decodeInt(slice.wrap(cr2.data()), cr2.codecs(), Assertions::assertEquals);

    // Encode and decode longs
    CodecResult cr3 = CodecHelper.encodeLong(i -> i, 10000, true, buffer);
    CodecHelper.decodeLong(slice.wrap(cr3.data()), cr3.codecs(), Assertions::assertEquals);

    // Encode and decode floats
    CodecResult cr4 = CodecHelper.encodeFloat(i -> i, 10000, buffer);
    CodecHelper.decodeFloat(slice.wrap(cr4.data()), cr4.codecs(), Assertions::assertEquals);

    // Encode and decode doubles
    CodecResult cr5 = CodecHelper.encodeDouble(i -> i, 10000, buffer);
    CodecHelper.decodeDouble(slice.wrap(cr5.data()), cr5.codecs(), Assertions::assertEquals);
    
}