Dictionary pipeline

The dictionary behind completion and autocorrect is a .wmdict file: a memory-mapped, frequency-ranked trie compiled from a plain-text wordlist. It isn't the only word source (the learned lexicon is JSON, and imported custom lists are packed into a trie in memory at load), but it is the one that ships in the APK and the one every download produces. This page covers the compiler that builds them, the format itself, where the wordlists come from, and how to add or improve one.

For the user-facing side of this (the per-language download screen, size tiers, and custom-dictionary import), see Downloadable dictionaries. This page is about the pipeline behind it.

From text file to binary trie

A .wmdict starts life as a plain-text wordlist, one entry per line, word<space>frequency. Lines starting with # are skipped, and so are blank lines. Anything else becomes an entry. A line with no space, or whose trailing token isn't a number, defaults to frequency 1 rather than being dropped. That's DictionaryLoader's format, and both the bundled dictionaries and user-imported custom dictionaries parse a file exactly this way. The downloadable catalog's own streaming parser (below) follows the same word<space>frequency convention, but it's a separate implementation with its own stricter filtering rather than a call into DictionaryLoader.

tools/dictc is the compiler: a small JVM command-line tool, dictc <srcDir> <outDir>, that reads every *.txt in srcDir and writes a matching <name>.wmdict to outDir. Its own code is just that Main.kt entry point. The trie-building logic isn't a separate reimplementation. build.gradle.kts adds core/prediction's source directory as an extra Kotlin source directory (Gradle's srcDir, not a symlink) and filters it down to the seven files it actually needs: Trie.kt, TrieWalker.kt, TrieCompleter.kt, PackedTrie.kt, PackedTrieCodec.kt, RankFloorCache.kt and DictionaryLoader.kt. So the writer and the runtime reader are literally the same code. The build's own comment on this: "one implementation, so the emitted bytes can never drift from what MappedTrie/PackedTrieCodec read."

You never invoke dictc by hand for a normal build. app/build.gradle.kts registers it as the compileBundledDictionaries task, which runs dictc over app/dictionaries-src/*.txt through javaexec and wires its output into every build variant's generated assets. That includes both full and lite, since the dictionary source doesn't change between flavors. The result lands under the APK's assets at dictionaries/<name>.wmdict, which is what DictionaryStore opens by name at runtime. See building from source for the rest of the build wiring.

Only two languages are bundled this way today: app/dictionaries-src/en.txt (17,306 word entries) and bn.txt (20,645 word entries). To add a word or adjust a frequency in either one, edit the .txt file and rebuild. No code changes, no manual compile step.

The .wmdict format, briefly

PackedTrieCodec defines the on-disk layout: an uncompressed, big-endian image of a compressed-sparse-row (CSR) trie, magic "WMDC", version 3. The header carries a flags word, word, node and edge counts and a symbol count, plus seven section offsets (symbols, childStart, checkpoint, edgeLabel, freq, maxSubtree, isWord), each section 4-byte aligned. Every node's outgoing edges are a contiguous, label-sorted slice of the edgeLabel array, so a lookup is a binary search rather than a hash lookup. There is no edgeChild array: version 3 numbers nodes breadth-first so edge e always leads to node e + 1, which is what version 1 was storing an identity function for. Two flag bits say how a given file spelled its labels and its child counts, since a Hangul or Japanese list blows past the one-byte encodings an alphabetic script fits inside. maxSubtree, the highest word frequency anywhere below a node, is precomputed at write time so completion can prune the search instead of walking every candidate.

MappedTrie is the reader. It mmaps the file read-only and does every lookup as a raw ByteBuffer read against the mapped pages. The trie never reaches the Java heap (only the file's edge-label alphabet is copied out at open, at most 512 bytes), so "loading" a dictionary is a single mmap call, and pages your typing never touches are never read at all. MappedTrie.open() returns null silently on a missing, truncated or wrong-version file rather than throwing, so a corrupt or half-written download falls back to whatever other word sources are available instead of crashing the keyboard.

The format itself carries no compression. That happens one layer up: the APK deflates bundled .wmdict assets the ordinary way, and the download pipeline inflates a .gz wordlist before writing it into this format. A .wmdict on disk is always raw.

Full byte-level layout: file formats reference.

The downloadable catalog

Beyond the two bundled languages, DictionaryCatalog lists every wordlist available from the companion wmkeyboard-data repository: currently 333 entries covering 332 languages. English and Bangla also have downloadable, larger replacements for their bundled lists, and Portuguese has two entries, European and Brazilian, that share one download slot. Each DictionaryEntry carries a language id, a repo code, the full list's word count and compressed size (both display hints, not checksums), and an optional rom suffix for one of the 14 romanized/Latin-script variants (Bangla, Hindi, Arabic, Russian, and others). A handful of language codes are remapped between the data repo's naming and the app's own registry (roa_rup→rup, mhr→chm, bxr→bua, nrm→nrf), and both Portuguese entries feed the single pt registry id.

Users pick a size tier before downloading: Small (50,000 words), Medium (150,000), Large (300,000, the default) or Everything, which takes the list whole however long it is. Tiers past the end of a short list all keep the same words, so the picker only offers the first one that reaches the whole list. The source files are pre-sorted by descending frequency, so WordlistDownloadManager can stop reading the stream as soon as it has kept enough words rather than transferring the whole file. That's what makes a 41 MB compressed Thai list a fast Medium download. It stops well short of the end.

While parsing, it also drops any line with frequency below 2 (the file is sorted descending, so hitting one means only noise follows), skips words over 48 characters or containing a space, and does a pre-flight free-space check with an 8 MiB margin before starting. There's no HTTP range or resume support, by design: a capped download is cheap enough to restart. The pipeline is stream, inflate, parse the first N frequency-sorted lines, build a PackedTrie, write main.wmdict.part, then atomically rename to main.wmdict. A file only exists on disk once it's completely valid.

On disk, bundled and downloaded dictionaries live under filesDir/dict/: dict/bundled/<name>.wmdict for the two inflated-from-APK lists, dict/<langId>/main.wmdict for a downloaded one. That tree is deliberately separate from filesDir/dictionaries/, the user-imported custom-dictionary tree covered below. At query time, CompositeWordSource merges whichever of these are present for a language. If a word appears in more than one source, it keeps the highest frequency across all of them.

Adding or improving a wordlist

There are three distinct paths, depending on which dictionary you mean:

1. The two bundled lists (English, Bangla). Edit app/dictionaries-src/en.txt or bn.txt directly, in the same word<space>frequency format as everywhere else. Then rebuild. compileBundledDictionaries regenerates the .wmdict automatically, so there's no separate compile step to remember.
2. The 332-language downloadable catalog. This data lives in the separate wmkeyboard-data repository, not in this codebase. DictionaryCatalog.kt only holds the metadata table (id, repo code, sizes) describing it. Regenerating that table against a fresh repo checkout is a manual or scripted process outside this repo.
3. Any other language, without touching either repo. A user can import their own word list, whether a Hunspell .dic, a frequency list or a plain word column, from Typing / Suggestions / Custom dictionaries (also reachable from a language's own Dictionary screen). Files land at filesDir/dictionaries/<langId>/<name>.txt, capped at 32 MiB each, and they stack additively. Several imported lists can sit under one language, and for English and Bangla they layer on top of the bundled list rather than replacing it. This is the fastest way to get real completions for a language with no bundled or downloadable dictionary. See Downloadable dictionaries → Options for the user-facing details.

There's no CONTRIBUTING.md or issue template in this repo yet. See contributing for what does exist around proposing a change.

Emoji keyword pack generation

Emoji search and suggestions have their own per-language dictionary pipeline: same shape, different codec (EmojiDictCodec, not PackedTrieCodec). See Emoji customization for the user-facing download UI. Two scripts in tools/emoji/ generate that data:

• export_keyword_pack.py converts a CLDR language's hand-translated emoji annotations into the app's importable keyword-pack TSV format (emojikeyword,keyword,...name). It cross-references the app's own emoji catalog, so it only emits entries for emoji the app actually carries. CLDR already has annotations for roughly 100 languages, so producing a pack for any of them is a format conversion rather than a translation job.
• generate_dict_catalog.py regenerates EmojiDictCatalog.kt from the wmkeyboard-data repo tree. It walks the tree for data/<dir>/<code>_emoji.json.gz files, taking the code from the file name (the folder is the code too, except for Banglish, filed as data/bn/bn_rom_emoji.json.gz), and writes the Kotlin table between two markers. (--check verifies without writing; --tree reads a saved API tree to dodge GitHub rate limits.)

The generated catalog currently holds 126 entries, down from the repo's 142 language packs, after three filters. Codes the repo spells differently are remapped (no→nb). Exact duplicates of a plainer code already present are dropped (fil==tl, pt_br==pt, zh_cn==zh), as are codes with no matching app language (blo, bs, ccp, quc, rhg, zh_tw). Finally, seven near-empty upstream stubs carrying 1–16 emoji each are dropped (bgn, ceb, ckb, mni, su, syr, vec).

The rest of the directory does a different job from anything else on this page: generate_catalog.py and generate_gemoji.py build the base emoji catalog and its :shortcode: triggers, add_names.py fills in emoji names, and generate_animated.py regenerates the list of emoji Noto ships an animated version of.

Details & edge cases

• The compiler is shared code, not a reimplementation. Beyond its own Main.kt entry point, tools/dictc has no trie-building code of its own. Its build points straight at core/prediction's source directory (an extra Gradle source directory, not a symlink) for the exact Trie/TrieWalker/TrieCompleter/PackedTrie/PackedTrieCodec/RankFloorCache/DictionaryLoader classes the app runs. A bug fix or format change to the runtime reader applies to the compiler automatically, so the two can never silently drift apart.
• One compile task feeds both flavors. compileBundledDictionaries isn't per-variant. It runs once and its output is wired into every build variant's generated assets, since full and lite read identical dictionary data.
• A corrupt or partial .wmdict degrades silently. MappedTrie.open() returns null on a bad magic number, wrong version or truncated file instead of throwing, so a failed or interrupted download leaves that word source empty rather than crashing the input method.
• Downloaded and custom dictionaries are additive, never exclusive. They live in separate on-disk trees (filesDir/dict/ and filesDir/dictionaries/), and CompositeWordSource reads from all of them at once for a language, keeping each word's highest frequency across sources.
• Direct boot only ever sees the bundled pair. The dict/bundled/ copies are extracted from the APK into device-protected storage, which exposes nothing user-specific, so they're available before the user unlocks the device. Downloaded and custom dictionaries live behind the credential, so they aren't there yet in that window. See architecture → direct boot for the fuller picture.
• Catalog sizes and counts are hints, not checksums. DictionaryCatalog's word counts and compressed sizes are display and progress values, generated against a point-in-time snapshot of the data repo. Drift against a newer repo state is expected and harmless.