Home

moezip Documentation

moezip is a C++20 learning-augmented text compression engine designed for natural language text. It combines a Mixture of Experts (MoE) predictive state router, Asymmetric Numeral Systems (rANS) arithmetic coding, and LZ-style back-referencing.

Unlike standard dictionary-less compression algorithms (such as raw Zstandard or Gzip) that incur frame overhead on short inputs, moezip embeds a pre-trained vocabulary and state transition matrix directly into the compiled binary or shared library. This allows pattern prediction on small payloads with zero metadata overhead.

---

Documentation Index

• Architecture & Design
  How the tokenization pipeline, 32x32 MoE state router, rANS coder, and make.py asset generator work internally.

• Building & Compilation
  Step-by-step CMake compilation guides for MSVC (Windows) and GCC/Clang (Linux).

• CLI Usage & Training
  Commands for file compression, decompression, hex decoding, and training custom domain router matrices.

• API Reference
  Complete documentation for the native Python module (import moezip), C-API shared library (moezip.dll / libmoezip.so), and C++ native headers.

• Performance & Benchmarks
  Performance metrics and space savings comparisons against Zstandard across micro-strings (< 500 B) and medium documents.

---

Core Characteristics

• Zero Warm-Up Penalty: Efficiently compresses micro-strings and short chat messages without frame-header inflation.
• Self-Contained Binaries: Running make.py bakes words_final.txt and router_stateless_v4.json directly into embedded_assets.hpp. The resulting binaries run anywhere without needing external asset files on disk.
• Flexible Build Targets: Builds standalone CLI executables, C-API dynamic libraries, or native PyBind11 Python extensions from a single CMake configuration.
• Domain Trainable: Includes a training command (moezip train) to optimize state transition weights for specialized domain text (such as medical, legal, or source code).

---

Output Artifacts

Depending on your platform and CMake options, the build process produces these artifacts:

Target Platform	Standalone CLI	Shared Library (C-API)	Python Extension Module
Windows (MSVC)	moezip.exe	moezip.dll / moezip.lib	moezip.pyd
Linux (GCC / Clang)	moezip	libmoezip.so	moezip.so

---

Quick Example

Python Extension (import moezip)

import moezip

# In-memory compression (embedded assets load automatically)
compressed = moezip.compress("Hello world, compress this text!")
restored = moezip.decompress(compressed)

assert restored == "Hello world, compress this text!"

Command Line Interface

# Compress a text file
moezip compress input.txt -o archive.moe

# Decompress back to text
moezip decompress archive.moe -o restored.txt

For setup instructions and deep dives into the codebase, navigate using the links above or the sidebar.