Skip to content

1.4.0

Choose a tag to compare

View all tags
@vthorsteinsson vthorsteinsson released this 20 Sep 14:35
· 61 commits to main since this release
1.4.0
2c86e9f

New: libbin — the C++ code as a standalone library

The C++ that backs islenska has moved into a self-contained library, libbin/, with a C API (libbin/include/libbin/bin.h). It is now the single implementation of BÍN lookup and of the compound word splitter, used both by the Python package (via CFFI) and directly by native C/C++ programs.

  • Results are returned as structs rather than JSON; handles are immutable and thread-safe (the DAWG's mutable node cache is gone).
  • The compound split policy of Bin lives in the library, and the Python code calls it.
  • Builds standalone with CMake (libbin::bin, plus libbin_objects for single-archive consumers) and has its own smoke test. See libbin/README.md.

New: compact build of compressed.bin

About half of the word forms in BÍN belong to compounds, such as bókahilla, whose inflection is exactly that of their last component (hilla) with the other components (bóka) glued in front. tools/binpack.py --compact produces a compressed.bin that leaves those forms out and lets the compound word algorithm regenerate them. The dropped compounds are restored transparently on lookup, with their original BÍN ids, subcategories and KRISTINsnid fields.

full compact
compressed.bin 94.8 MB 55.4 MB
lemmas in the trie 358,274 194,925
distinct word forms 3,770,484 1,822,169

tools/parity.py compares every word form and every BÍN id between the two files: 0 differences.

The islenska package on PyPI still ships the full file. To use a compact file instead, pass its name to BinCompressed(fname) or point the ISLENSKA_BIN_FILE environment variable at it.

python tools/binpack.py --compact -o compressed-compact.bin --report compact.tsv
python tools/parity.py src/islenska/resources/compressed.bin compressed-compact.bin

Other

  • The data format is now Greynir 05.00.00 (one new header field; compact builds add flagged lemma records and an index of the dropped lemmas). compressed.bin files from earlier versions are not compatible and need to be regenerated with tools/binpack.py.
  • tools/binpack.py has a proper command line: --compact, --output, --keep, --report, --procs.
  • macOS wheels are built for a 10.13 deployment target (libbin is C++17).

Backward compatible for Python users — the existing Bin API and its results are unchanged.

Full diff: #26

Assets 2
Loading