Skip to content

Releases

v0.4.3

Choose a tag to compare

@misterekkk misterekkk released this 14 Nov 12:27

Security

  • Two more crafted files found by the nightly fuzzers. An XGBoost model with a huge num_class
    allocated gigabytes before anything else was checked (666,666,666 classes: 2.7 GB); classes are now
    limited to 65,535, as for LightGBM. A CatBoost .bin file whose trees' leaves did not follow each
    other passed the checks but made prediction read past the leaves; the leaves must now be laid out as
    lignum writes them.

v0.4.2

Choose a tag to compare

@misterekkk misterekkk released this 10 Nov 16:28

Security

  • Damaged .bin files could make prediction read outside the file. An element count in the
    header with high bits set (e.g. the number of leaves plus 2^61) wrapped around when multiplied by
    the element size, so every size check passed and every stored index looked in range. Every count
    is now checked against the file size before it is used, also in the binned sections and for the
    categorical tables' offsets.
  • Crafted XGBoost JSON / UBJSON files could crash or exhaust memory. A tree deeper than about
    20,000 levels overflowed the stack, and a node shared by two parents was unfolded into every
    path (29 nodes, 2 KB of JSON: 2^28 leaves). Trees are now limited to 1024 levels, like the
    LightGBM parsers, and a node may have only one parent.
  • Large categories in a small file allocated gigabytes. A LightGBM split on category
    4294967295 allocated a 512 MB bitset, and every further split as much again. Categories that no
    input can match (LightGBM: from 2^31 on; XGBoost: outside [0, 2^24) or negative) are dropped,
    which changes no prediction, and the category sets of a model may take at most about 256 MB.
  • The fallback cache of compiled libraries was shared. Without HOME (and without
    LIGNUM_CACHE_DIR), compile() cached libraries in the temporary folder, where another user of
    the machine could place a library under the expected name. The fallback is now a folder per user,
    lignum-cache-<uid>, used only when it is owned by this user and nobody else can write to it.
  • Further memory errors on crafted files, found by the new fuzzers. An XGBoost multi-class
    model with a single base_score read freed memory; a .bin file with a node shared by two parents
    made predict hang (every node now has one parent, and unknown header flags are rejected); a UBJSON
    object key was read after the output buffer moved; a .cbm file with empty parameters passed a
    null pointer to memcpy. All are rejected or handled, with regression tests.

Added

  • More objectives, bit-exact: LightGBM models trained with reg_sqrt=True (regression,
    regression_l1, fair, quantile, mape: predict squares the score back, transform
    "signed_square"), XGBoost binary:hinge ("hinge": 0 or 1), and CatBoost's Focal,
    MultiLogloss and MultiCrossEntropy (sigmoid per output, as predict_proba).
    tests/python/test_objectives.py trains every objective lignum accepts and compares.

Changed

  • Unknown objectives are rejected at load time. Before, an objective lignum did not know got
    no output transform, so predict silently differed from the library's (reg_sqrt,
    binary:hinge, RMSEWithUncertainty, MultiLogloss, Focal did). Every objective in the
    documentation of LightGBM 4, XGBoost 3 and CatBoost is now known and tested, except CatBoost's
    RMSEWithUncertainty, QueryCrossEntropy (GPU only) and CatBoost models trained with a Python
    objective, which are rejected with a message. LightGBM and XGBoost
    models trained with a Python objective predict raw scores, as before and as the libraries do.
  • .bin format 0.9. It adds the categories of training; files of format 0.8 (lignum 0.4.0,
    0.4.1) still load. Files written by 0.4.2 cannot be read by lignum 0.4.1 or older.

Fixed

  • pandas category columns get the codes of training. LightGBM (pandas_categorical) and
    XGBoost 3 (cats) store the categories of training, and their predict(df) maps a DataFrame's
    categories onto them. lignum used the codes as they came, so a frame whose categories were in
    another order or a subset (e.g. made from the current batch with astype("category")) predicted
    differently without an error (80% of the rows in a test), string categories failed to convert,
    and integer categories gave their values instead of codes. lignum now reads the categories and
    maps pandas category columns and Arrow dictionary columns (pyarrow, polars) the way the library
    does, bit for bit: a category not in training is missing (LightGBM) or a ValueError (XGBoost),
    and other category columns than in training are a ValueError, as in the libraries. One lookup
    per category and an array read per row; the table is cached per categories object.
  • CatBoost models with more than 65,535 float and one-hot features read the wrong column. Their
    index was cut to 16 bits, so a split on feature 65,541 used feature 5. Such models are now
    rejected; up to 65,535 load as before (and their .bin files load too).
  • XGBoost categories of 2^32 and more were wrapped to small categories by a 32-bit cast (2^32 + 2
    became category 2). They can never be matched and are dropped.
  • A model without splits crashed Arrow prediction with binned=False on x86-64 (an integer
    division by zero).
  • A LightGBM dump_model() JSON with a feature named learner was read as XGBoost and failed.
    JSON files are recognised by their first key. Malformed LightGBM and XGBoost JSON files now give
    errors that name the library and the file, instead of only simdjson's error code; a category that
    is not an integer is an error instead of an exception from std::stoul.

v0.4.1

Choose a tag to compare

@misterekkk misterekkk released this 30 Oct 12:10

Changed

  • New benchmarks. benchmarks/suite.py replaces the earlier comparisons: the same models on every
    machine (500 rounds; LightGBM num_leaves=63, XGBoost lossguide with max_leaves=63, CatBoost
    depth 6), seven data sets, batches of 1 to 1,000,000 rows on one thread and all threads, p50 / p99 /
    p99.9 latencies, preparation (compile) times and a bit-exactness check, against LightGBM, XGBoost,
    CatBoost, lleaves and TL2cgen, from numpy and from Arrow. Single rows now differ on every call:
    the earlier latencies predicted the same row again and again, which the branch predictor learns,
    so they were too optimistic for every engine. With that, the new models and another server, one
    California row on x86 went from 2.9 to 11 µs for lignum and from 7.8 to 24.6 µs for lleaves. The benchmarks page, the README and the home page show only the new
    results (Apple M4 and a Hetzner CCX33 with AMD EPYC-Milan); the old reports are removed.

Fixed

  • compile() on macOS compiles in parallel again. The generated files were compiled from
    several threads through system(), which macOS serialises with a global lock, so they ran one
    after another. The compiler is now started with posix_spawn on Linux and macOS. Apple M4,
    500 rounds with 63 leaves: California 8.4 -> 2.1 s, Diabetes 7.0 -> 1.7 s; Covertype
    (3,500 trees) 63 -> 13 s. Linux (glibc) already ran them in parallel; Windows is unchanged.

v0.4.0

Choose a tag to compare

@misterekkk misterekkk released this 27 Oct 21:14

Changed

  • Breaking: column names are checked. When a model was trained on named columns and the input
    has column names (pandas DataFrame, pyarrow, polars or other Arrow data), predict checks that
    the columns start with the model's feature_names, in the same order, and raises ValueError
    otherwise. Before, a reordered or missing column silently gave other predictions. numpy arrays,
    frames without column names, models without names and the C API are unchanged (by position).
    predict(..., validate_features=False), load(..., validate_features=False) or
    model.validate_features = False turn the check off; it costs 0-5 µs per call (Apple M4).

  • Breaking: .bin format 0.8. It adds the feature names, the library the model was trained
    with, its objective and the lignum version that wrote the file. Files written by lignum 0.3 are rejected: save
    them again from the original model.

  • Compiled-library files (Model.compile()): without library=, libraries go to a per-user
    cache (~/.cache/lignum, ~/Library/Caches/lignum, %LOCALAPPDATA%\lignum\Cache, or
    LIGNUM_CACHE_DIR) instead of lignum_compiled in the temp directory, and the code is built in a
    temporary folder that is always removed: no .c, .o or log files are left behind (a failed
    build keeps only its log, named in the error together with the compiler's last lines).
    library= writes the library exactly there, always compiling it and bypassing the cache.

  • Breaking: compile() has no work_dir any more (set LIGNUM_CACHE_DIR to move the cache),
    and its options after flags are keyword-only. In the C API the fourth argument of
    LignumModelCompile is now the cache folder (NULL: the per-user cache).

Added

  • Model inspection: Model.n_trees, Model.source ("lightgbm", "xgboost" or "catboost"),
    Model.objective (the library's name, e.g. "binary", "multi:softprob", "Logloss") and
    Model.info(), a dict that adds the output transform, the number of leaves, the kernel in use,
    the compiled library and, for .bin files, the lignum version that wrote them. repr(model)
    shows the source, objective, trees and features. C API: LignumModelNumTrees.
  • Model.feature_names: the names of the training columns (LightGBM, XGBoost and CatBoost files,
    kept in .bin), or None for models trained on a matrix.
  • lignum.cache_dir(), lignum.clear_cache() and Model.compiled_library (the path of the
    library in use).
  • compile(library=folder) writes lignum_<model fingerprint>_<platform>.so (.dll on Windows)
    into the folder, and load(..., compiled=folder) picks the file for the model and platform, so one
    folder can hold the libraries of several models and platforms.

Fixed

  • Processes compiling the same model at once no longer write the same files: each builds in its own
    folder and the library is moved into place with an atomic rename.
  • The cache key includes the CPU architecture: x86-64 (e.g. under Rosetta) and arm64 processes no
    longer load each other's libraries. The key is also stable across compilers (FNV-1a instead of
    std::hash), and a cached library that fails to load is rebuilt.
  • Errors when loading a library built for another platform name the platform that is needed.
  • XGBoost models that were predicted wrongly without an error are now rejected: linear boosters
    (booster="gblinear", which loaded as an empty forest predicting base_score) and DART models
    that dropped trees (rate_drop > 0: XGBoost weights each tree, lignum ignored the weights). DART
    files from XGBoost before 3.4 (trees nested under gbtree) used to load as an empty forest too;
    they now predict correctly when no tree was dropped. LightGBM DART models were and are exact.

The old lignum_compiled folder in the temp directory is no longer used and can be deleted.

v0.3.1

Choose a tag to compare

@misterekkk misterekkk released this 27 Oct 11:37

A small follow-up to 0.3.0. No change to predictions or the API.

pip install --upgrade pylignum

Fixed

  • License notices. NOTICE now includes the notices of third-party code lignum derives from, as their licenses require for source and binary copies: CityHash (Copyright (c) 2011 Google, Inc., MIT License; CatBoost's category hash) and fmath by herumi (BSD 3-Clause; the exp CatBoost uses on x86). The 0.3.0 wheels lacked them.
  • Two GCC warnings (-Wall -Wextra).
  • The CatBoost benchmark no longer asks CatBoost to read polars 2.x Categorical columns, which CatBoost does not support (on Windows it could crash on them). lignum was not affected.

Full changelog: https://github.com/misterekkk/lignum/blob/main/CHANGELOG.md

v0.3.0

Choose a tag to compare

@misterekkk misterekkk released this 27 Oct 02:38

lignum 0.3.0 adds CatBoost and moves to the Apache License 2.0.

pip install --upgrade pylignum   # imported as `import lignum`

Highlights

  • CatBoost models, bit-identical to CatBoost. Symmetric trees, float features, categorical features as one-hot or CTRs (Borders, Buckets, Counter, BinarizedTargetMeanValue) including combinations of categories; binary, regression, Poisson, Tweedie, MultiClass and MultiClassOneVsAll. Reads .cbm and JSON.
  • Categories straight from your data: strings or integers from pandas, numpy object arrays, lists, pyarrow and polars. C and C++ callers pass category hashes (LignumCatBoostHash), or Arrow batches that lignum hashes itself.
  • Fast: a dedicated kernel for symmetric trees, built for both generic x86-64 and AVX2 (chosen at run time), on Linux, macOS and Windows.

Performance

CatBoost 1.2.10 / lignum, 500 trees of depth 6, C API, one thread:

x86-64 (AMD EPYC 9V45) ARM (Neoverse-N2)
Adult (categorical) 0.424 / 0.213 s 0.941 / 0.339 s
Diabetes 130-US (categorical) 1.624 / 0.735 s 2.926 / 1.209 s
Covertype (numeric) 0.287 / 0.119 s 0.818 / 0.265 s
One row, Adult 20.1 / 2.3 µs 50.7 / 4.9 µs

2-3x faster than CatBoost on batches and 7-14x on single rows. The smallest margin is on purely numeric models on x86 (California housing: 0.071 vs 0.061 s). Full results, including Apple M4, Windows and synthetic data: https://misterekkk.github.io/lignum/benchmarks/

Changed

  • License: Apache License 2.0. Versions up to 0.2.0 remain available under the MIT License.
  • The .bin format is now version 0.7. Files written by 0.2.0 are rejected with a clear error: export them again from the original model.
  • A 1-D array or flat list is one row for every model.

Fixed

  • predict accepts read-only arrays (pandas 3.1).

Not supported yet

CatBoost Depthwise / Lossguide trees and text or embedding features are rejected at load time.

Also new: a changelog, a security policy and a contributing guide. Using lignum? Add yourself to ADOPTERS.md.

Documentation: https://misterekkk.github.io/lignum/

v0.2.0

Choose a tag to compare

@misterekkk misterekkk released this 22 Oct 23:42

First release on PyPI:

pip install pylignum   # imported as `import lignum`

lignum predicts with LightGBM and XGBoost models, and its results are bit-identical to booster.predict(...).

Highlights

  • Exact semantics. Covers LightGBM and XGBoost thresholds, NaN and zero-as-missing, categorical splits, multiclass models and XGBoost's float32 arithmetic. Every test checks bit-exact parity against the real libraries.
  • Two engines. A binned interpreter handles any batch size. The compiled mode (model.compile()) turns the trees into native code with the system C compiler in a few seconds:
    • save the result as a library next to the model and load it in about 2 ms without a compiler;
    • model.autotune(X) picks the interpreter or the compiled code per batch size.
  • Formats. LightGBM model.txt and JSON, XGBoost JSON and .ubj, detected from the file content. model.save("model.bin") writes a binary file that loads by memory mapping in under a millisecond.
  • Arrow input. pyarrow tables and batches, polars data frames and any object with the Arrow PyCapsule interface are read without copies. Dictionary columns are treated as categories.
  • C API for other languages, including Arrow input and the compiled mode.
  • Wheels for Linux x86_64 and aarch64 (manylinux_2_28), macOS 15+ (arm64 and x86_64) and Windows x64. Python 3.10, 3.11, and abi3 for 3.12+.

Performance

LightGBM, 500 trees, Hetzner AMD EPYC-Milan (4 dedicated vCPU). Whole data set, all threads:

lignum compiled lleaves LightGBM
California housing 0.014 s 0.021 s 0.140 s
Covertype 0.265 s 0.269 s 2.401 s
Diabetes 130-US 0.151 s 0.803 s 1.830 s

On one thread, a single row takes 2.9-3.2 µs (lleaves 7.8-20 µs). The benchmarks page has full results for x86-64, ARM and Apple M4, including the cases where lignum is not the fastest.

Not supported

LightGBM linear trees and random forests, and multi-target / vector-leaf models. These models are rejected at load time.

Documentation: https://misterekkk.github.io/lignum/

v0.1.0

Choose a tag to compare

@misterekkk misterekkk released this 10 Jun 21:21

Initial release of the package