Background
Lance currently depends on an older Lindera release for the optional Japanese tokenizer path. Lindera 3.x changes both the runtime token API and the dictionary/configuration formats, so keeping Lance on the old integration path makes future tokenizer maintenance and dependency updates harder.
Why upgrade
Upgrading to Lindera 3.0.7 keeps Lance aligned with the current upstream tokenizer API and verifies that our Rust and Python FTS paths can load real Lindera models with the new dictionary format.
The upgrade also improves dependency hygiene. The new dependency graph drops the old encoding and bincode paths that were previously responsible for stale security advisory ignores, reducing audit noise and making future security review more direct.
Expected benefits
- Keep the Japanese tokenizer integration compatible with the current Lindera API.
- Refresh the checked-in Lindera test dictionaries and examples to the current config/dictionary format.
- Remove stale cargo-deny advisory ignores that are no longer needed after the dependency graph changes.
- Reduce future maintenance cost for FTS tokenizer updates by moving away from the old Lindera 0.x surface.
Related work
Background
Lance currently depends on an older Lindera release for the optional Japanese tokenizer path. Lindera 3.x changes both the runtime token API and the dictionary/configuration formats, so keeping Lance on the old integration path makes future tokenizer maintenance and dependency updates harder.
Why upgrade
Upgrading to Lindera 3.0.7 keeps Lance aligned with the current upstream tokenizer API and verifies that our Rust and Python FTS paths can load real Lindera models with the new dictionary format.
The upgrade also improves dependency hygiene. The new dependency graph drops the old
encodingandbincodepaths that were previously responsible for stale security advisory ignores, reducing audit noise and making future security review more direct.Expected benefits
Related work