Performance and Benchmarks

Performance & Benchmarks

moezip is engineered specifically for natural language text. By embedding a pre-trained dictionary and Markov state router directly in RAM, it eliminates the frame-header inflation that causes general-purpose algorithms to expand small inputs.

---

The "Warm-Up Penalty" Explained

General-purpose compression algorithms (such as raw Zstandard, Gzip, or Deflate) are designed to handle arbitrary binary streams. When compressing short strings (< 500 bytes), these algorithms incur a frame header penalty (~18 bytes or more) and lack sufficient history to build an entropy model, often causing the compressed output to become larger than the input text.

Because moezip pre-loads its state transition matrix and dictionary into memory during library initialization, it compresses short inputs instantly with zero metadata overhead.

---

1. Short Text & Micro-String Benchmarks (< 500 Bytes)

The following benchmark compares moezip against raw Zstandard (Level 3) across typical short text payloads.

Benchmark Data Table

Test Payload	Original Size	moezip Compressed	moezip Ratio	Raw Zstandard (L3)	moezip Latency	Status
Tiny Phrase	23 Bytes	10 Bytes	43.5%	32 Bytes (139% Expanded)	20.7 µs	PASS
Short Sentence	57 Bytes	22 Bytes	38.6%	66 Bytes (116% Expanded)	38.7 µs	PASS
Chat Message	32 Bytes	13 Bytes	40.6%	41 Bytes (128% Expanded)	24.2 µs	PASS
JSON Payload	121 Bytes	68 Bytes	56.2%	109 Bytes (90%)	74.8 µs	PASS
News Snippet	299 Bytes	134 Bytes	44.8%	208 Bytes (70%)	186.7 µs	PASS

Summary Comparison

Across 532 cumulative bytes of micro-string test samples:

• moezip: Reduced total size to 247 bytes (53.6% total space savings).
• Raw Zstd (L3): Reduced total size to 456 bytes (14.3% total space savings).

---

2. Medium Text Clips (4 KB – 16 KB)

On medium-sized text blocks, moezip maintains a consistent 36%–42% compression ratio (~58%–64% space savings), continuing to outperform unassisted LZ dictionary algorithms on natural text:

• 4.8 KB Text Clip: Compresses to 2,072 bytes (42.6% of original) vs. Zstandard's 2,499 bytes (51.4%).
• 5.4 KB Text Clip: Compresses to 2,286 bytes (42.3% of original) vs. Zstandard's 2,595 bytes (48.0%).
• 16.0 KB Document: Compresses down to 5,773 bytes (36.1% of original).

Throughput & Execution Latency

When executed via memory-resident interfaces (moezip.pyd native extension or moezip.dll / libmoezip.so), medium-clip compression passes complete in under 3 milliseconds.

---

Key Performance Drivers

1. Greedy Phrase Matching: Matches common multi-word phrases (e.g., "united states", "of the") and affixes in a single token pass, skipping character-by-character dictionary builds.
2. Predictive Expert Routing: The $32 \times 32$ state matrix accurately predicts the expert partition of upcoming words, allowing most matches to be encoded using Tag 0 (short local index) rather than Tag 1 (full expert override).
3. Variable Byte-Length State Flush: The 32-bit rANS encoder calculates exact state bytes on finalize and writes a 1-byte length header, avoiding 4-byte fixed padding penalties at stream EOF.