v0.4.2

Changes

• Fixed swapped dash/quote punctuation normalization.
• Misc fixes, including docs.

v0.4.0

Changes

• Embedded-dictionary features restructured - japanese-ipadic-neologd-lindera, japanese-ipadic-lindera, and japanese-unidic-lindera no longer exist - use japanese-lindera plus one (or more) of japanese-lindera-embed-ipadic, japanese-lindera-embed-ipadic-neologd, japanese-lindera-embed-unidic. Likewise, chinese-lindera/korean-lindera alone no longer bundle a dictionary - add chinese-lindera-embed/korean-lindera-embed for the old always-embedded behavior.
• full no longer embeds anything - It now enables japanese-lindera/chinese-lindera/korean-lindera in their new runtime-path-loading form, not the old always-bundled ipadic-neologd/cc-cedict/ko-dic. If you relied on full producing a working CJK tokenizer out of the box, add the matching *-embed-* features or call set_dictionary_path.
• Note that embedding dictionaries is heavy, especially ipadic-neologd.
• Dictionaries can now be loaded from disk at runtime - instead of only from an embedded copy: set_dictionary_path(algorithm, Some(path)). This is the new default behavior for japanese-lindera/chinese-lindera/korean-lindera - enabling one of them alone no longer bundles anything into the binary (a huge size win: ipadic-neologd alone was adding well over a gigabyte).
• Multiple embedded Japanese dictionaries can coexist. - japanese-lindera-embed-ipadic, -ipadic-neologd, and -unidic are independent and additive now, not mutually exclusive. Which one is actually used is a runtime choice via the new set_japanese_embedded_dictionary(JapaneseDictionaryKind) - auto-picks the best one compiled in (ipadic-neologd > unidic > ipadic) if never called.
• New Error::NoDictionary variant: returned when a CJK language is tokenized with no path set and no matching dictionary compiled in, instead of panicking.

v0.3.1

Changes

• Bumped deps.

0.3.0

Changes

• Changed MatchResult to use struct values instead of tuples.
• Changed Token and MatchResult to use u32 offsets and lengths.
• Updated edition to 2024 and MSRV to 1.88.0.

0.2.0

Changes

• Change MatchMode values to use structs instead of tuples