Gukhanmun 0.3.0: Han character variants, particle-based reading selection, punctuation profiles, and loanword spellings

[dahlia]

Gukhanmun is a library and CLI tool for converting Korean text written in mixed hanja/hangul script into consistently annotated output. Academic texts, historical documents, and legal materials routinely mix Chinese characters (hanja) with Korean script (hangul); Gukhanmun reads them and renders the result in your chosen mode: hangul-only, ruby annotation, original script, and more. It is available as a Rust crate, a Node.js package, a WebAssembly module, and a standalone CLI binary.

0.3.0 adds four things that are reachable from every surface: dictionary matching for han character variants, so 沖縄県 reads as 오키나와현 and 藝術 can be emitted as 芸術; reading selection from the Korean particle that follows a hanja word, which picks 일체 over 일절 in 一切를; opt-in punctuation profiles, for output that should not keep the source's punctuation; and the National Institute of Korean Language's loanword spelling examples, which give 香港 its usual 홍콩. The dictionary layer underneath now keeps competing readings instead of collapsing them at extraction time. If you are upgrading from 0.2.x, check the upgrading notes below, because the loanword dictionary changes ko-kr output.

Han character variants

Dictionary headwords record hanja in orthodox forms. Real mixed-script text arrives in whatever register its source used: Japanese material writes 沖縄県 where the Standard Korean Dictionary (標準國語大辭典) has 沖繩縣, simplified Chinese uses 发 where the dictionary has 發, and older printings carry CJK compatibility ideographs. Those spellings used to miss the dictionary and fall back to per-character readings, which produced a plausible-looking but wrong conversion.

Recognition now expands each character of a source token into its variant equivalence class and lets the dictionary decide which combination is a real word. For 沖縄県 the engine tries the exact spelling first, then Joyo, traditional, simplified, z-variant, and curated Asahi candidates, and 沖繩縣 is the one that exists as a headword:

$ echo '沖縄県' | gukhanmun
오키나와현

$ echo '沖縄県' | gukhanmun --rendering hangul-hanja-parens
오키나와현(沖繩縣)

The annotation now carries the dictionary spelling alongside the source spelling. Directives, first-occurrence tracking, and homophone checks use the dictionary spelling as the lexical identity, so a variant on the input side does not change what the rest of the pipeline considers the same word.

Output is a separate policy. --hanja-variant-set chooses which register the renderer emits whenever the selected mode keeps hanja: as-dictionary (the default), shinjitai, kanxi, simplified, or asahimoji. In Rust the same setting is Builder::hanja_variant_set(), and both JavaScript bindings take a hanjaVariantSet option.

$ echo '藝術' | gukhanmun --rendering original --hanja-variant-set shinjitai
芸術

The limits are deliberate. Only variants that differ at the code-point level are handled, so regional glyph differences that Unicode unifies through variation selectors stay out of scope. When a variant merges several characters, as 弁 merges 弁, 辯, 辨, and 瓣, the engine accepts a spelling only if the dictionary lists it, and leaves the token unresolved rather than guessing when more than one dictionary word survives. Expansion is capped at 32 characters and 256 candidates per lookup, so ordinary hanja runs do not turn into scans of the generated tables. The simplified mapping is many-to-one and lossy per character, and the Asahi profile covers only the encoded subset of that character set.

The variant tables are pinned and reviewable: gukhanmun-unihan records the Unicode 17.0.0 inputs, the checked-in Joyo pairs, and the small verified Asahi subset, and mise run generate-unihan-check fails when the generated tables drift. The original proposal is in #11 and the implementation in #12.

Reading selection from following particles

Dictionary extraction used to collapse competing readings into a single value per key before the runtime dictionary was ever built. That kept lookup simple and discarded the evidence a sentence supplies. 一切 is both 일절 and 일체, 東京 is both 동경 and 도쿄, and which one a text means depends on how the rest of the sentence is written.

0.3.0 keeps the candidates from extraction through storage and lookup, and selects one during conversion. A higher-priority dictionary still replaces the whole candidate set for a key, so an explicit override is not quietly reopened by a bundled reading.

The first consumer of that information is the particle that follows the word. In 그는 東京을 좋아한다 the particle 을 fits 동경 and not 도쿄; 東京를 says the opposite. Gukhanmun now reads a whole particle after a matched key, compares each candidate's final syllable with the form the particle requires, and keeps the candidates that fit:

$ echo '一切' | gukhanmun
일절

$ echo '一切를 바꾸었다' | gukhanmun
일체를 바꾸었다

This is evidence, not grammar correction. The source particle is left exactly as written, and the previous reading survives when the answer is uncertain: only whole particles count (을/를, 은/는, 이/가, 과/와, and 으로/로 with its ㄹ exception, plus combinations such as 으로는, 과도, and 와만), a particle separated from the word by inline markup is not a match, and the copula 이다 is not mistaken for the subject particle 이. If no candidate fits, or if several fit equally, conversion falls back to the recorded default rather than inventing a reading. An explicit dictionary override stays authoritative throughout.

Rust callers can also create and inspect candidate sets: ReadingCandidate, ReadingCandidates, and MapDictionary::insert_candidates() build them, and lookup().candidates() reads them back. gukhanmun-mkdict accepts preference and suffix_reading columns in TSV and CSV input, or preference and suffixReading in JSONL. Binary dictionaries are written as v1 for scalar data and as v2 when candidates or suffix readings require it; the bundled Standard Korean Dictionary and Open Korean Dictionary data now ship as v2. Homophone marking indexes only the selected candidate, including its suffix reading, so the annotations and the conversion agree.

The design rationale is in #19, the storage and API work in #22, and the Open Korean Dictionary half in #21 and #24. The particle filter, including its streaming lookahead, is in #20 and #25.

Punctuation profiles

Hanja conversion leaves punctuation alone, which is usually right: a document that quotes classical text keeps its 、/。 and 「/」. It is the wrong default for a document being prepared for modern hangul-only presentation, where the converted characters sit next to source punctuation and the two styles no longer agree.

Punctuation normalization is now available as an opt-in output profile that runs after annotation policies and before rendering, so reading selection, generated glosses, and ruby spans are untouched. It works on hangul-only text too, and it is independent of hanja rendering, hanja variants, dictionaries, and the Korean presets.

Profile	Comma/period	Single/double delimiters	Space after stop
preserve (default)	unchanged	unchanged	unchanged
modern	,/.	〈〉/《》	contextual single space
traditional	、/。	「」/『』	removed
traditional-vertical	︑/︒	﹁﹂/﹃﹄	removed

$ echo '하나、둘。' | gukhanmun --punctuation modern
하나, 둘.

$ echo '하나, 둘.' | gukhanmun --punctuation traditional
하나、둘。

In Rust the profile is Builder::punctuation(PunctuationProfile::modern()); JavaScript bindings take punctuation: "modern". A custom profile can start from a base and override individual roles, and is accepted as JSON from --punctuation-profile, from a JavaScript object, or through PunctuationConfig in Rust with the serde feature:

{
  "base": "modern",
  "period": null,
  "comma": { "mark": "、", "after": "remove" },
  "single": { "open": "︿", "close": "﹀" }
}

Input recognition is shared across profiles, so mixed styles in one document converge on the chosen output: ASCII, fullwidth, ideographic, and vertical comma and period forms, and the 〈〉, 《》, 「」, 『』 delimiter families including their vertical and U+2329/U+232A spellings. The transformation favors technical text over consistency. Stops between digits, runs of periods, and periods next to Latin letters or digits are left alone, as are ASCII and fullwidth commas inside numbers or before Latin words, so 3.14, 1,000, example.com, and Hello, world survive while IMF, 세계은행 can still be normalized. URLs and mailto: runs are protected. There is no smart-quote, ellipsis, or dash handling, and unmatched delimiter pairs are not repaired.

HTML gets its own handling. Named, decimal, and hexadecimal character references are recognized in text and normalize like literal characters, decoding happens exactly once so . stays literal text, and untouched references keep their original spelling. Generated marks are escaped so they cannot recombine with adjacent source text into a different reference. Markdown needs more care, because changing punctuation can create a list, a heading, or a link, or break emphasis delimiters. With normalization enabled the writer escapes text with numeric references and falls back to local inline HTML where emphasis would otherwise be lost, which is why the Markdown source spelling changes while the rendered meaning does not:

$ echo '하나、둘。' | gukhanmun -f text/markdown --punctuation modern
하나, 둘.

The feature, its guards, and the HTML and Markdown adapters are described in the punctuation guide. The proposal is in #27 and the implementation in #28.

Loanword spellings

The Standard Korean Dictionary maps 香港 to 샹강, after the Mandarin reading. The dictionary does contain a 홍콩 entry, but it lists only the English original spelling, so no extraction rule could recover a 香港 → 홍콩 mapping from that snapshot. The loanword spelling examples record the hanja spelling directly, so the mapping is extractable.

0.3.0 bundles the National Institute of Korean Language's loanword spelling examples as a separate dictionary. The source snapshot keeps all 67,303 records with provenance, and the generated dictionary has 3,249 keys, with eight explicit editorial decisions where the source offers competing readings. Dictionary precedence is user dictionaries first, then loanword examples, then the preset's other dictionaries.

$ echo '香港' | gukhanmun
홍콩

$ echo '香港' | gukhanmun --no-loanwords
샹강

The ko-kr preset now includes this dictionary, in Rust and in the CLI, so 香港 converts to 홍콩 by default. The ko-kp preset leaves it off, since the source's South Korean spellings are the wrong norm for North Korean text; pass --loanwords to enable it there. --no-loanwords disables only this dictionary and keeps the rest of the preset, and it cannot be combined with --no-bundled-dictionaries.

For Rust, the umbrella crate's loanwords feature is enabled by default, Preset::KoKr includes the dictionary, and Builder::bundled_loanwords() and Builder::no_bundled_loanwords() control it explicitly. The data lives in its own gukhanmun-loanwords crate, whose source can also be used directly as a HanjaDictionary.

JavaScript presets never add dictionary data on their own, so the bindings reach the same data through two new packages:

npm  add     @gukhanmun/loanwords-fst
pnpm add     @gukhanmun/loanwords-fst
yarn add     @gukhanmun/loanwords-fst
deno add jsr:@gukhanmun/loanwords-fst
bun  add     @gukhanmun/loanwords-fst

@gukhanmun/loanwords-cdb provides the same data in CDB format. Either loader returns a dictionary source that goes into the dictionaries array, before the preset dictionaries:

import { load } from "@gukhanmun/wasm";
import { loanwordsFst } from "@gukhanmun/loanwords-fst";
import { stdictFst } from "@gukhanmun/stdict-fst";

const g = await load({
  preset: "ko-kr",
  dictionaries: [await loanwordsFst(), await stdictFst()],
});
g.convert("香港"); // "홍콩"

Installing a package does not change conversion by itself; the loader has to be in dictionaries. The data's reuse terms are recorded in each package's ATTRIBUTION.md and are separate from the software license.

The extraction, licensing review, and weekly update pipeline are in #18 and #26, and enabling the dictionary by default for ko-kr is #31.

Other changes

• The bundled Standard Korean Dictionary snapshot is now the 2026-10-12 dump and the Open Korean Dictionary snapshot is the 2026-10-04 dump, with matching provenance and notices. (Update bundled Standard Korean Dictionary and Open Korean Dictionary data automatically #29, Automate Standard Korean Dictionary and Open Korean Dictionary updates #30)
• A weekly workflow now checks both NIKL dictionary sources, regenerates the data, and opens a pull request against the newest *.*-maintenance branch that ships the dictionary, keeping a separate pull request for main. Until a dictionary has shipped in a stable release it updates main only. (Bundle NIKL loanword spelling examples as a dictionary #18, Bundle NIKL loanwords with weekly data updates #26, Update bundled Standard Korean Dictionary and Open Korean Dictionary data automatically #29, Automate Standard Korean Dictionary and Open Korean Dictionary updates #30)
• Engine streaming boundaries changed slightly: a sole non-block root Close with pending text is emitted at the next token, at flush(), or at finish(), and finish() now supplies end-of-file evidence that flush() deliberately does not. Particle lookahead retains four extra source scalars beyond the dictionary tail allowance so a chunk boundary cannot change the chosen reading. (Filter dictionary readings using following Korean particles #20, Use Korean particles to select readings #25)
• Dictionary-wide homophone marking includes the selected candidate's suffix reading, including suffixes that come from custom MapDictionary entries. (Preserve multiple dictionary readings and separate reading selection #19, Preserve competing readings until conversion #22)
• gukhanmun-dict-extract gained loanword collection and offline extraction commands behind the loanwords-collect and loanwords-cli features, with provenance reports and an audit mode. The commands need NIKL_LOANWORD_API_KEY only for live collection; ordinary builds use the pinned snapshot. (Bundle NIKL loanword spelling examples as a dictionary #18, Bundle NIKL loanwords with weekly data updates #26)
• CLI archives now include NOTICE.md, so the attribution for bundled dictionary data travels with the prebuilt binary. (Bundle NIKL loanword spelling examples as a dictionary #18, Bundle NIKL loanwords with weekly data updates #26)
• The playground gained a hanja variant selector and a loanword dictionary toggle. (Variant-character (異體字) normalization #11, Normalize Han character variants without guessing #12, Prefer loanword spellings in the ko-kr preset #31)

Upgrading and installing

Upgrading from 0.2.x

The ko-kr preset now includes the loanword dictionary, which changes output for foreign names written in hanja: 香港 converts to 홍콩 rather than 샹강. To keep the previous answers, pass --no-loanwords on the CLI, call Builder::no_bundled_loanwords() in Rust, or keep the JavaScript binding as it is, since its presets still load no dictionary data implicitly.

Rust builds that use --no-default-features with the ko-kr preset must now either enable the loanwords feature or call Builder::no_bundled_loanwords(); without the feature, build() fails with Error::Config instead of silently converting without the dictionary.

Dictionary data files have a version. Bundled FST and CDB dictionaries that carry candidates or suffix readings are written as v2, and a reader from before this release rejects them rather than misreading them. Upgrade @gukhanmun/wasm or @gukhanmun/napi together with @gukhanmun/stdict-fst, @gukhanmun/stdict-cdb, and the Open Korean Dictionary packages. Scalar custom dictionaries built with gukhanmun-mkdict are still emitted as v1 and load in older readers.

Beyond those, the defaults are unchanged: punctuation normalization stays opt-in, the hanja variant set defaults to as-dictionary, and the CLI --numerals default remains smart.

CLI

Install a prebuilt binary with mise:

mise use -g aqua:dahlia/gukhanmun

On Windows, use winget:

winget install HongMinhee.Gukhanmun

Or build from source with Cargo:

cargo install gukhanmun-cli

Prebuilt archives for Linux, macOS, and Windows are attached to the 0.3.0 release on GitHub.

Rust

cargo add gukhanmun

See the Rust installation guide for feature flag options, and the CHANGES.md section for the complete list of API additions.

JavaScript/TypeScript

# WebAssembly (browsers, Node.js, Deno, Bun):
pnpm add @gukhanmun/wasm @gukhanmun/stdict-fst

# Node-API (Node.js, Deno, Bun; faster on server):
pnpm add @gukhanmun/napi @gukhanmun/stdict-fst

# Optional: loanword spellings and Open Korean Dictionary data:
pnpm add @gukhanmun/loanwords-fst @gukhanmun/opendict-fst

See the JavaScript installation guide for npm, yarn, Bun, and Deno equivalents.