Skip to content

v0.5.1 — CJK support: dictionary-backed Chinese/Japanese/Korean tokenization

Choose a tag to compare

View all tags
@github-actions github-actions released this 26 Aug 11:20
· 71 commits to main since this release
v0.5.1
0754757

Highlights

LadyM now understands Chinese, Japanese, and Korean. CJK text tokenizes out of the box (fully offline), and optional dictionary-backed word segmentation upgrades recall quality — downloaded on demand from the console, baked into Docker images, or compiled into the binary. LadyM never downloads anything on its own.

This release also introduces automated multi-platform releases: every v* tag now publishes prebuilt tarballs for Linux/macOS × amd64/arm64 with a SHA256SUMS file.

What's New

🀄 CJK tokenization (Chinese / Japanese / Korean)

The native hashing tokenizer previously matched only [A-Za-z0-9_] — CJK queries tokenized to an empty set and always scored similarity 0. Now:

  • Han runs segment by dictionary word via go-ego/gse (the jieba algorithm, pure Go); kana and hangul tokenize per character with adjacent bigrams.
  • Zero-dictionary fallback: without any dictionary, CJK still tokenizes per character — recall keeps working, nothing to configure, nothing to download.
  • The default binary stays lean (50MB): gse's embedded dictionaries only link in when explicitly requested (see below).

📦 The dictionary system

Four downloadable variants, sha256-pinned to gse v1.0.2:

Variant Content Download
zh (default) Chinese, simplified + traditional 8.2 MB
zh_s Chinese, simplified only 4.9 MB
zh_t Chinese, traditional only 3.4 MB
jp Japanese, kanji + kana 22.6 MB

Design guarantees:

  • User-triggered only — downloads happen via the console or the admin-gated POST /api/cjk_dict/download; the tokenization path never touches the network (enforced by a contract test).
  • Integrity first — every file is sha256-verified before anything is written; failed or tampered downloads leave the existing dictionary untouched.
  • Hot reload — downloads and variant switches take effect immediately, no restarts; switching variants cleans up the old files.
  • Internal mirrors — {"mirror_base": "..."} points downloads at an air-gapped internal mirror.

🖥️ Console: Settings → Memory

A new admin Settings page manages the dictionary: variant picker, download/switch/remove, live status (source, version, size, directory). Works in both the personal embedded console and the enterprise ladymconsole service.

🐳 Dict-baked images (Docker)

One command builds and runs every role with the dictionary baked into the image — no shared volume, no downloads, no internet at runtime:

docker compose -f docker-compose.dev.yml -f docker-compose.dev.dict.yml up -d --build
docker compose -f docker-compose.enterprise.yml -f docker-compose.enterprise.dict.yml up -d --build

The dictionary ships as a thin data layer (/opt/ladym/dict) on top of the regular image: plain docker build . stays dictionary-less, the layer is architecture-independent, air-gapped builds work by placing the pinned files under dict/, and Dockerfile.dict --build-arg BASE=<image> stacks the layer onto any already-released image.

For fleets without baked images, the shared-volume pattern remains: point every instance at one directory (LADYM_DICT_DIR), download once on any instance, and all instances follow within ~30 seconds.

⚙️ Dictionary directory configuration

The dictionary directory (default ~/.ladyM/dict) is now configurable on all three daemon commands, with the usual priority chain:

ladym serve --http :8080 --dict-dir /data/cjk-dict   # flag
LADYM_DICT_DIR=/data/cjk-dict ladym serve --http :8080  # env
# ladym.toml: dict_dir = "/data/cjk-dict"             # toml

📚 Downstream Go consumers

Three ways to get the embedded dictionary — the module version never changes:

  1. Runtime download — trigger via console/API, or call storage.DownloadCJKDict().
  2. Side-effect import — import _ "github.com/ProjAnvil/LadyM/storage/fulldict" (no build-script changes).
  3. Build tag — -tags fulldict (what the ladym-personal-fulldict-* assets below use).

There is deliberately no v0.5.1-fulldict module version: semver reads that suffix as a pre-release of v0.5.1 and it would confuse go get.

Upgrading

No breaking changes; no migration needed. Optional:

  • For better CJK recall quality, download a dictionary: console Settings → Memory, or curl -X POST -H 'Content-Type: application/json' -d '{"dict":"zh"}' http://localhost:8080/api/cjk_dict/download (admin auth applies).
  • Docker users can switch to dict-baked images with the compose overrides above.

Release Assets

Built automatically by .github/workflows/release.yml (test gates → 4-platform matrix → upload):

Asset What it is
ladym-personal-<ver>-<os>-<arch>.tar.gz Personal edition (dictionary downloadable at runtime)
ladym-personal-fulldict-<ver>-<os>-<arch>.tar.gz Personal edition with the zh dictionary compiled in (~+31MB)
ladym-enterprise-<ver>-<os>-<arch>.tar.gz Enterprise edition (ladym + ladymconsole, Postgres-only)
SHA256SUMS Checksums for every tarball

Platforms: linux/amd64, linux/arm64, darwin/amd64, darwin/arm64. Static binaries (CGO_ENABLED=0).

Full changelog: v0.5.0...v0.5.1

Assets 15
Loading