Architecture and Design

moezip is built around a hybrid compression pipeline that combines tokenization, Markov-chain expert prediction, LZ back-referencing, and Asymmetric Numeral Systems (rANS) entropy coding.

[Raw UTF-8 Text]
       │
       ▼
[Tokenizer & N-Gram Matching] ─── (Phrases, Words, Spacing, Casing)
       │
       ▼
[LZ Cache & Match Search] ──────── (Identifies repetitive token sequences)
       │
       ▼
[MoE State Router] ────────────── (Predicts expert partition based on previous expert)
       │
       ▼
[Tag & Symbol Generator] ──────── (Tag 0: Match, Tag 1: Mispredict, Tag 2: OOV, Tag 3: LZ)
       │
       ▼
[Adaptive rANS Engine] ────────── (Encodes symbols via dynamic cumulative models)
       │
       ▼
[Compressed .moe Stream]

---

1. Asset Embedding Engine (make.py & embedded_assets.hpp)

moezip embeds its dictionary (words_final.txt) and pre-trained router matrix (router_stateless_v4.json) directly into the compiled executable or library.

C++ Literal Limit Bypass

To avoid MSVC compiler error C2026 (string literal limit of 65,535 characters), make.py slices raw text assets into 16,000-character chunks using C++ raw string literals:

// Generated in embedded_assets.hpp
inline const char* const EMBEDDED_VOCAB_CHUNKS[] = {
    R"EMBED_VOCAB(word1\nword2\n...)EMBED_VOCAB",
    R"EMBED_VOCAB(...)EMBED_VOCAB",
};
inline const size_t EMBEDDED_VOCAB_CHUNKS_COUNT = 2;

Runtime Asset Resolution Logic

When load_and_partition_wordlist() or load_router() executes:

1. It attempts to open the requested file from disk.
2. If the disk file is missing, it automatically falls back to streaming the embedded C++ raw string chunks from memory into an std::stringstream.
3. If an explicit file is found on disk, it overrides the embedded defaults.

---

2. Tokenizer & Phrase Matching (tokenizer.cpp)

The tokenizer transforms arbitrary UTF-8 text into a sequence of Token pairs consisting of a word string and trailing whitespace spacing (std::pair<std::string, std::string>).

Greedy N-Gram Phrasing

Before single-word tokenization occurs, tokenize() runs a multi-word greedy matching pass:

• Checks 3-word combinations against known multi-word entries (e.g., "shah rukh khan", "united states").
• Checks 2-word combinations (e.g., "of the", "in the", "new york").
• If matched, the phrase is emitted as a single token, reducing symbol count downstream.

Sub-Word Fallback Splitting

For long words not present in the dictionary, tokenize() splits the word into recognized sub-word chunks (length >= 3) to maximize vocabulary coverage before falling back to character literals.

Casing Extraction

Words are normalized to lowercase for dictionary lookup, and their original casing transformation is recorded separately (get_casing()):

• 0: Lowercase (word)
• 1: Capitalized (Word)
• 2: Uppercase (WORD)
• 3: Titlecase (Word Word)

---

3. Predictive MoE State Router (router.cpp)

moezip partitions its global vocabulary into 32 equal-sized expert blocks (expert_count = 32).

Vocabulary Partitioning

$$\text{expert size} = \left\lceil \frac{\text{total words}}{32} \right\rceil$$

A token's global index $G$ maps to an expert index $E$ and a local offset $L$: $$E = \lfloor G / \text{expert size} \rfloor, \quad L = G \bmod \text{expert size}$$

Transition Matrix & Online Learning

The router maintains a $32 \times 32$ transition matrix $T$, where $T[i][j]$ tracks how frequently expert $j$ follows expert $i$.

• Prediction (predict_next_expert): Given the previous expert $E$, the router selects expert $j$ that maximizes row $T[E][j]$.
• Online Adaptation (update_transition_matrix): Every processed word increments $T[E_{\text{prev}}][E_{\text{actual}}]$. When a row sum exceeds 128, all values in that row are halved (val / 2), allowing the matrix to adapt to local document topic changes without exploding count values.

---

4. Symbol Tagging & Codec (codec.cpp)

The encoder scans tokens sequentially and evaluates match conditions, emitting one of four symbol tags to the ANS stream:

Tag	Condition	Encoded Data
Tag 0	Expert matched prediction	local_idx, casing (if applicable), spacing
Tag 1	Expert misprediction	actual_expert, local_idx, casing, spacing
Tag 2	Out of Vocabulary (OOV) token	Word character length, raw bytes, spacing
Tag 3	LZ Back-Reference match found	match_distance, match_length

LZ Cache Searching

To compress repetitive sequences, find_best_token_match() queries an LZCache (std::unordered_map<std::string, std::deque<int>>). If a sequence of tokens has appeared earlier within the window, a Tag 3 LZ match is written instead of encoding words individually.

---

5. 32-Bit Range ANS Engine (ans.cpp)

Entropy coding is handled by a 32-bit rANS (ranged Asymmetric Numeral Systems) implementation.

Adaptive Probability Models (AdaptiveModel)

Symbol probabilities are tracked dynamically using integer cumulative frequency tables:

• Initialized with uniform counts.
• Updated on every encoded or decoded symbol.
• Rescaled ((freq >> 1) | 1) when the total symbol count reaches 16,383 to maintain precision bounds.

Two-Pass Encoding Pass

Because rANS operates as a Last-In, First-Out (LIFO) stack, encoding operations are pushed onto an ANSAction queue during the forward pass of compress_adaptive_moe(). The queue is then processed in reverse during ANSStream::finalize().

Zero-Overhead Stream Termination

ANSStream::finalize() flushes the internal state $x$ using a variable byte length strategy:

1. Calculates the minimum bytes required to represent the final state $x$.
2. Emits a 1-byte length prefix specifying the precise state byte count.
3. This syncs the decoder stream (ANSDecoder) on read without requiring fixed 4-byte padding at the end of payloads.

---

6. Decoded Data Flow Synchronization

Decompression (decompress_adaptive_moe) mirrors the state updates of the encoder step-by-step:

1. Decodes the symbol tag from tag_model.
2. Reads corresponding fields (expert overrides, local indices, or LZ distances).
3. Reconstructs word text, applies casing transformations, and appends trailing spacing.
4. Updates the local transition matrix $T$ and adaptive models identically to maintain decoder alignment.