Dictionary

Dictionary

Overview

Laren ships with a 375,193 word Arabic dictionary with frequency scores, stored as a simple TSV file.

في	7540
من	7500
على	7180
أن	7030
هذا	6670
...

How It Works

flowchart LR
    A["dictionary.tsv<br/><i>375k words + frequencies</i>"] -->|"load at startup"| B["Sorted-Vector Trie"]
    B -->|"exact lookup"| C["Word found?<br/>Return frequency"]
    B -->|"prefix search"| D["Find all words<br/>starting with prefix"]

    style A fill:#fff9c4
    style B fill:#e3f2fd

Loading

Trie Structure

The dictionary is stored in a sorted-vector trie — a trie where each node's children are stored in a sorted vector for cache-friendly traversal:

graph TD
    R["root"] --> S["س"]
    R --> ع["ع"]

    S --> SL["سل"]
    SL --> SLA["سلا"]
    SLA --> SLAM["سلام<br/><i>freq: 5230</i>"]
    SLA --> SLAH["سلاح<br/><i>freq: 4890</i>"]

    ع --> عR["عر"]
    عR --> عRB["عرب<br/><i>freq: 5100</i>"]
    عR --> عRI["عري<br/><i>freq: 3200</i>"]

    style SLAM fill:#c8e6c9
    style SLAH fill:#c8e6c9
    style عRB fill:#c8e6c9
    style عRI fill:#c8e6c9

Loading

Candidate Selection

When the DFS expansion produces Arabic forms, each form is checked against the dictionary:

flowchart TD
    A["DFS produces 2000+ forms"] --> B{"Each form"}
    B -->|"In dictionary"| C["✅ Dictionary candidate<br/><i>with frequency score</i>"]
    B -->|"Not in dictionary"| D["Skeletal form pool"]

    C --> E["Sort by frequency"]
    D --> F["Fill remaining slots<br/><i>up to 50 candidates</i>"]

    E --> G["Final candidate list"]
    F --> G

    style C fill:#c8e6c9
    style D fill:#ffecb3

Loading

Dictionary candidates are always ranked higher than skeletal fallbacks.

Word Sources

The dictionary was generated from:

1. wordfreq — 353k Arabic words with zipf frequency >= 1.5
2. ه-ending variants — For every word ending in ة (ta marbuta), a ه (ha) variant is auto-generated. This covers Egyptian informal spelling (e.g., مدرسه alongside مدرسة)
3. Manual additions — Common Egyptian dialect/slang words

Frequency Scoring

Frequencies use zipf scale (logarithmic, 1-8):

Score	Meaning	Example
7+	Extremely common	في، من، على
5-7	Common	سلام، كتاب
3-5	Moderate	فلسفة، ديمقراطية
1.5-3	Rare	Specialized/archaic terms

Higher frequency = ranked higher in candidate list.