v0.7.0

CAUTION

The v0.7.* is intended as an intermediate release series before the v1.
Please pin the exact version when using this series, as breaking behavioral changes may be introduced even in patch releases.

Dictionary Binary Format Update: V1

This release introduced new dictionary format V1.
Because of this change, you need to update both of the system dictionary and your user dictionaries.
See migration guide for details.

API Change

This release introduced many API update including some breaking changes.
See migration guide and CHANGELOG.

Full Changelog: v0.6.11...v0.7.0

v0.6.11

Highlights

• Support CPython 3.14 and 3.14t (free-threaded) ( #307 )
• Add Linux arm64 wheel builds ( #302 )
• Support Apple embedded platforms (iOS, tvOS, watchOS, visionOS) ( #308 )

Other

• Update PyO3 to v0.27 ( #307 )
• Update dependencies and lockfiles for the 0.6.11 release series ( #303, #311 )

v0.6.10

Highlights

• Support py3.13t (wheels are provided only for linux-amd64 and macos-univarsal2) (#284)
• Broken source distribution in v0.6.9 is fixed (#294)

Fixed

• Dictionary.__repr__ for no-user dict case is fixed (#282)

Other

• Update build/test workflows (#293, #295)
• Code refactoring (#283)
• Update python-support-version badge to python3.9+ (#292)

v0.6.9

Highlights

• Support python3.13
  • py3.13t is not supported yet
• Remove Python 3.7 and 3.8 support as it reaches its end of life (https://devguide.python.org/versions/) (#249, #281)

rust

Added

• freebsd support (#222 by @KonstantinDjairo, #251)
• Add option for embedded config and fallback resources (#262 by @Kuuuube)

Changed

• fetch_dictionary.sh targets latest dictionary by default (#240)
• Migrate from structopt to clap (#248 by @tkhshtsh0917)

python

Added

• Allow string literals as SplitMode (#245)
• Add sudachipy.Config and sudachipy.errors.SudachiError to default import (#260)

Changed

• -s (system dictionary path) of sudachi ubuild command is now required (#239)
• -d option of sudachi cli (which is no-op) now warns (#278)
• Update the output of sudachi dump subcommand (#277)

see rust changelog and python changelog for more.

v0.6.8

Highlights

• Produce builds for Python 3.12 (#236)
• Add a simple configuration API
• Add surface projections (#230)

Surface Projections

• For chiTra compatibility SudachiPy can now directly produce different tokens in the surface field.
• Original surface is accessible via Morheme.raw_surface() method
• It is possible to customize projection dictionary-wise, via Config object, passing it on a dictionary creation, or for a single pre-tokenizer.
  • Config API
  • Pretokenizer API

0.6.7

Highlights

• Provide binary wheels for Python 3.11
• Add Dictionary.lookup() method which allows you to enumerate morphemes from the dictionary without performing analysis.

0.6.6

Highlights

• Add boundary matching mode to regex oov handler
• macOS binary builds are now unversal2 (arm+x64)

MacOS

• Binary builds are universal2
• Caveat: we don't run tests on arm because there are no public arm instances, so builds may be broken without any warning

0.6.5

Highlights

• Fixed invalid POS tags which appeared when using user-defined POS tags both in user dictionaries and OOV handlers. You are not affected by this bug if you did not use user-defined POS in OOV handlers.

Version 0.6.4

Highlights

• Remove Python 3.6 support which reached end-of-life status on 2021-12-23
• OOV handler plugins support user-defined POS, similar to Java version
• Added Regex OOV handler

Regex OOV Handler

• For details, see Java version changelog
• In Rust/Python Regexes do not support backtracking and backreferences
• maxLength setting defines maximum length in unicode codepoints, not in utf-8 bytes as in Java (will be changed to codepoints later)

0.6.3

Highlights

• Fixed path resolution algorithm for resources. They are now resolved in the following order (first existing file wins):
  1. Absolute paths stay as they are
  2. Relative to "path" value of the config file
  3. Relative to "resource_dir" parameter of the config object during creation
     • For SudachiPy it is the parameter of Dictionary constructor
  4. Relative to the location of the configuration file
  5. Relative to the current directory

Python

• Dictionary now has __repr__() function which displays absolute paths to dictionaries in use.
• Dictionary now has pos_of() function which returns a POS tuple for a given POS id.
• PosMatcher supports set operations
  • union (m1 | m2)
  • intersection (m1 & m2)
  • difference (m1 - m2)
  • negation (~m1)