Skip to content

3.7.0

Choose a tag to compare

@vthorsteinsson vthorsteinsson released this 14 Jul 01:11
357692a

Changes

  • Native (C++) token/terminal matching — parsing is up to 2.2x faster.
    The majority of token/terminal match decisions are now made natively in the
    C++ Earley core instead of via Python callbacks, roughly doubling parsing
    speed on fresh text (~89% of matching callbacks eliminated). Matching
    results are bit-exact with the previous Python matcher, verified by a
    query-level parity mode (GREYNIR_MATCHING_PARITY=1) and an extensive
    test corpus. Derived parsers that override token wrapping (such as
    GreynirCorrect) automatically continue to use Python matching.

  • Python >= 3.10 is now required. Python 3.9 is end-of-life; users on
    3.9 should stay on GreynirEngine 3.6.x. Binary wheels are now built as
    cp310 abi3 for CPython, plus PyPy 3.11 wheels.

  • Robust interprocess locking and atomic grammar writes. The binary
    grammar file is written atomically (temp file + rename), so an
    interrupted process can never leave a truncated grammar behind. The
    hand-rolled lock in glock.py has been replaced by the cross-platform
    filelock package; the lock file now lives next to the binary grammar
    instead of in /tmp, lock acquisition times out with a clear error
    instead of hanging, and no lock is taken at all when the binary grammar
    is already up to date.

  • Hardened parser core. The C++ core now validates the binary grammar
    file fully on load, catches out-of-memory conditions at the C ABI
    boundary instead of crashing, uses race-free diagnostic counters, and
    bounds the token matching cache (~125 MB) in long-running processes.

  • Greynir() now raises TypeError on unrecognized keyword options
    instead of silently ignoring them. Note that e.g. max_sent_tokens is a
    parameter of the parse methods, not of the constructor.

  • Modernized tooling. The project now uses uv with a committed
    lockfile and PEP 735 dependency groups; PEP 639 license metadata;
    mypy-clean type checking as a CI gate; and CI testing on Windows and
    macOS in addition to Linux (CPython 3.10-3.14 and PyPy 3.11).