Repository navigation
Releases
v0.4.3
Security
- Two more crafted files found by the nightly fuzzers. An XGBoost model with a huge
num_class
allocated gigabytes before anything else was checked (666,666,666 classes: 2.7 GB); classes are now
limited to 65,535, as for LightGBM. A CatBoost.binfile whose trees' leaves did not follow each
other passed the checks but made prediction read past the leaves; the leaves must now be laid out as
lignum writes them.
v0.4.2
Security
- Damaged
.binfiles could make prediction read outside the file. An element count in the
header with high bits set (e.g. the number of leaves plus 2^61) wrapped around when multiplied by
the element size, so every size check passed and every stored index looked in range. Every count
is now checked against the file size before it is used, also in the binned sections and for the
categorical tables' offsets. - Crafted XGBoost JSON / UBJSON files could crash or exhaust memory. A tree deeper than about
20,000 levels overflowed the stack, and a node shared by two parents was unfolded into every
path (29 nodes, 2 KB of JSON: 2^28 leaves). Trees are now limited to 1024 levels, like the
LightGBM parsers, and a node may have only one parent. - Large categories in a small file allocated gigabytes. A LightGBM split on category
4294967295 allocated a 512 MB bitset, and every further split as much again. Categories that no
input can match (LightGBM: from 2^31 on; XGBoost: outside [0, 2^24) or negative) are dropped,
which changes no prediction, and the category sets of a model may take at most about 256 MB. - The fallback cache of compiled libraries was shared. Without
HOME(and without
LIGNUM_CACHE_DIR),compile()cached libraries in the temporary folder, where another user of
the machine could place a library under the expected name. The fallback is now a folder per user,
lignum-cache-<uid>, used only when it is owned by this user and nobody else can write to it. - Further memory errors on crafted files, found by the new fuzzers. An XGBoost multi-class
model with a singlebase_scoreread freed memory; a.binfile with a node shared by two parents
madepredicthang (every node now has one parent, and unknown header flags are rejected); a UBJSON
object key was read after the output buffer moved; a.cbmfile with empty parameters passed a
null pointer tomemcpy. All are rejected or handled, with regression tests.
Added
- More objectives, bit-exact: LightGBM models trained with
reg_sqrt=True(regression,
regression_l1,fair,quantile,mape: predict squares the score back,transform
"signed_square"), XGBoostbinary:hinge("hinge": 0 or 1), and CatBoost'sFocal,
MultiLoglossandMultiCrossEntropy(sigmoid per output, aspredict_proba).
tests/python/test_objectives.pytrains every objective lignum accepts and compares.
Changed
- Unknown objectives are rejected at load time. Before, an objective lignum did not know got
no output transform, sopredictsilently differed from the library's (reg_sqrt,
binary:hinge,RMSEWithUncertainty,MultiLogloss,Focaldid). Every objective in the
documentation of LightGBM 4, XGBoost 3 and CatBoost is now known and tested, except CatBoost's
RMSEWithUncertainty,QueryCrossEntropy(GPU only) and CatBoost models trained with a Python
objective, which are rejected with a message. LightGBM and XGBoost
models trained with a Python objective predict raw scores, as before and as the libraries do. .binformat 0.9. It adds the categories of training; files of format 0.8 (lignum 0.4.0,
0.4.1) still load. Files written by 0.4.2 cannot be read by lignum 0.4.1 or older.
Fixed
- pandas
categorycolumns get the codes of training. LightGBM (pandas_categorical) and
XGBoost 3 (cats) store the categories of training, and theirpredict(df)maps a DataFrame's
categories onto them. lignum used the codes as they came, so a frame whose categories were in
another order or a subset (e.g. made from the current batch withastype("category")) predicted
differently without an error (80% of the rows in a test), string categories failed to convert,
and integer categories gave their values instead of codes. lignum now reads the categories and
maps pandascategorycolumns and Arrow dictionary columns (pyarrow, polars) the way the library
does, bit for bit: a category not in training is missing (LightGBM) or aValueError(XGBoost),
and othercategorycolumns than in training are aValueError, as in the libraries. One lookup
per category and an array read per row; the table is cached per categories object. - CatBoost models with more than 65,535 float and one-hot features read the wrong column. Their
index was cut to 16 bits, so a split on feature 65,541 used feature 5. Such models are now
rejected; up to 65,535 load as before (and their.binfiles load too). - XGBoost categories of 2^32 and more were wrapped to small categories by a 32-bit cast (2^32 + 2
became category 2). They can never be matched and are dropped. - A model without splits crashed Arrow prediction with
binned=Falseon x86-64 (an integer
division by zero). - A LightGBM
dump_model()JSON with a feature namedlearnerwas read as XGBoost and failed.
JSON files are recognised by their first key. Malformed LightGBM and XGBoost JSON files now give
errors that name the library and the file, instead of only simdjson's error code; a category that
is not an integer is an error instead of an exception fromstd::stoul.
v0.4.1
Changed
- New benchmarks.
benchmarks/suite.pyreplaces the earlier comparisons: the same models on every
machine (500 rounds; LightGBMnum_leaves=63, XGBoostlossguidewithmax_leaves=63, CatBoost
depth 6), seven data sets, batches of 1 to 1,000,000 rows on one thread and all threads, p50 / p99 /
p99.9 latencies, preparation (compile) times and a bit-exactness check, against LightGBM, XGBoost,
CatBoost, lleaves and TL2cgen, from numpy and from Arrow. Single rows now differ on every call:
the earlier latencies predicted the same row again and again, which the branch predictor learns,
so they were too optimistic for every engine. With that, the new models and another server, one
California row on x86 went from 2.9 to 11 µs for lignum and from 7.8 to 24.6 µs for lleaves. The benchmarks page, the README and the home page show only the new
results (Apple M4 and a Hetzner CCX33 with AMD EPYC-Milan); the old reports are removed.
Fixed
compile()on macOS compiles in parallel again. The generated files were compiled from
several threads throughsystem(), which macOS serialises with a global lock, so they ran one
after another. The compiler is now started withposix_spawnon Linux and macOS. Apple M4,
500 rounds with 63 leaves: California 8.4 -> 2.1 s, Diabetes 7.0 -> 1.7 s; Covertype
(3,500 trees) 63 -> 13 s. Linux (glibc) already ran them in parallel; Windows is unchanged.
v0.4.0
Changed
-
Breaking: column names are checked. When a model was trained on named columns and the input
has column names (pandas DataFrame, pyarrow, polars or other Arrow data),predictchecks that
the columns start with the model'sfeature_names, in the same order, and raisesValueError
otherwise. Before, a reordered or missing column silently gave other predictions. numpy arrays,
frames without column names, models without names and the C API are unchanged (by position).
predict(..., validate_features=False),load(..., validate_features=False)or
model.validate_features = Falseturn the check off; it costs 0-5 µs per call (Apple M4). -
Breaking:
.binformat 0.8. It adds the feature names, the library the model was trained
with, its objective and the lignum version that wrote the file. Files written by lignum 0.3 are rejected: save
them again from the original model. -
Compiled-library files (
Model.compile()): withoutlibrary=, libraries go to a per-user
cache (~/.cache/lignum,~/Library/Caches/lignum,%LOCALAPPDATA%\lignum\Cache, or
LIGNUM_CACHE_DIR) instead oflignum_compiledin the temp directory, and the code is built in a
temporary folder that is always removed: no.c,.oor log files are left behind (a failed
build keeps only its log, named in the error together with the compiler's last lines).
library=writes the library exactly there, always compiling it and bypassing the cache. -
Breaking:
compile()has nowork_dirany more (setLIGNUM_CACHE_DIRto move the cache),
and its options afterflagsare keyword-only. In the C API the fourth argument of
LignumModelCompileis now the cache folder (NULL: the per-user cache).
Added
- Model inspection:
Model.n_trees,Model.source("lightgbm","xgboost"or"catboost"),
Model.objective(the library's name, e.g."binary","multi:softprob","Logloss") and
Model.info(), a dict that adds the output transform, the number of leaves, the kernel in use,
the compiled library and, for.binfiles, the lignum version that wrote them.repr(model)
shows the source, objective, trees and features. C API:LignumModelNumTrees. Model.feature_names: the names of the training columns (LightGBM, XGBoost and CatBoost files,
kept in.bin), orNonefor models trained on a matrix.lignum.cache_dir(),lignum.clear_cache()andModel.compiled_library(the path of the
library in use).compile(library=folder)writeslignum_<model fingerprint>_<platform>.so(.dllon Windows)
into the folder, andload(..., compiled=folder)picks the file for the model and platform, so one
folder can hold the libraries of several models and platforms.
Fixed
- Processes compiling the same model at once no longer write the same files: each builds in its own
folder and the library is moved into place with an atomic rename. - The cache key includes the CPU architecture: x86-64 (e.g. under Rosetta) and arm64 processes no
longer load each other's libraries. The key is also stable across compilers (FNV-1a instead of
std::hash), and a cached library that fails to load is rebuilt. - Errors when loading a library built for another platform name the platform that is needed.
- XGBoost models that were predicted wrongly without an error are now rejected: linear boosters
(booster="gblinear", which loaded as an empty forest predictingbase_score) and DART models
that dropped trees (rate_drop > 0: XGBoost weights each tree, lignum ignored the weights). DART
files from XGBoost before 3.4 (trees nested undergbtree) used to load as an empty forest too;
they now predict correctly when no tree was dropped. LightGBM DART models were and are exact.
The old lignum_compiled folder in the temp directory is no longer used and can be deleted.
v0.3.1
A small follow-up to 0.3.0. No change to predictions or the API.
pip install --upgrade pylignumFixed
- License notices.
NOTICEnow includes the notices of third-party code lignum derives from, as their licenses require for source and binary copies: CityHash (Copyright (c) 2011 Google, Inc., MIT License; CatBoost's category hash) and fmath by herumi (BSD 3-Clause; theexpCatBoost uses on x86). The 0.3.0 wheels lacked them. - Two GCC warnings (
-Wall -Wextra). - The CatBoost benchmark no longer asks CatBoost to read polars 2.x
Categoricalcolumns, which CatBoost does not support (on Windows it could crash on them). lignum was not affected.
Full changelog: https://github.com/misterekkk/lignum/blob/main/CHANGELOG.md
v0.3.0
lignum 0.3.0 adds CatBoost and moves to the Apache License 2.0.
pip install --upgrade pylignum # imported as `import lignum`Highlights
- CatBoost models, bit-identical to CatBoost. Symmetric trees, float features, categorical features as one-hot or CTRs (
Borders,Buckets,Counter,BinarizedTargetMeanValue) including combinations of categories; binary, regression, Poisson, Tweedie,MultiClassandMultiClassOneVsAll. Reads.cbmand JSON. - Categories straight from your data: strings or integers from pandas, numpy object arrays, lists, pyarrow and polars. C and C++ callers pass category hashes (
LignumCatBoostHash), or Arrow batches that lignum hashes itself. - Fast: a dedicated kernel for symmetric trees, built for both generic x86-64 and AVX2 (chosen at run time), on Linux, macOS and Windows.
Performance
CatBoost 1.2.10 / lignum, 500 trees of depth 6, C API, one thread:
| x86-64 (AMD EPYC 9V45) | ARM (Neoverse-N2) | |
|---|---|---|
| Adult (categorical) | 0.424 / 0.213 s | 0.941 / 0.339 s |
| Diabetes 130-US (categorical) | 1.624 / 0.735 s | 2.926 / 1.209 s |
| Covertype (numeric) | 0.287 / 0.119 s | 0.818 / 0.265 s |
| One row, Adult | 20.1 / 2.3 µs | 50.7 / 4.9 µs |
2-3x faster than CatBoost on batches and 7-14x on single rows. The smallest margin is on purely numeric models on x86 (California housing: 0.071 vs 0.061 s). Full results, including Apple M4, Windows and synthetic data: https://misterekkk.github.io/lignum/benchmarks/
Changed
- License: Apache License 2.0. Versions up to 0.2.0 remain available under the MIT License.
- The
.binformat is now version 0.7. Files written by 0.2.0 are rejected with a clear error: export them again from the original model. - A 1-D array or flat list is one row for every model.
Fixed
predictaccepts read-only arrays (pandas 3.1).
Not supported yet
CatBoost Depthwise / Lossguide trees and text or embedding features are rejected at load time.
Also new: a changelog, a security policy and a contributing guide. Using lignum? Add yourself to ADOPTERS.md.
Documentation: https://misterekkk.github.io/lignum/
v0.2.0
First release on PyPI:
pip install pylignum # imported as `import lignum`lignum predicts with LightGBM and XGBoost models, and its results are bit-identical to booster.predict(...).
Highlights
- Exact semantics. Covers LightGBM and XGBoost thresholds, NaN and zero-as-missing, categorical splits, multiclass models and XGBoost's float32 arithmetic. Every test checks bit-exact parity against the real libraries.
- Two engines. A binned interpreter handles any batch size. The compiled mode (
model.compile()) turns the trees into native code with the system C compiler in a few seconds:- save the result as a library next to the model and load it in about 2 ms without a compiler;
model.autotune(X)picks the interpreter or the compiled code per batch size.
- Formats. LightGBM
model.txtand JSON, XGBoost JSON and.ubj, detected from the file content.model.save("model.bin")writes a binary file that loads by memory mapping in under a millisecond. - Arrow input. pyarrow tables and batches, polars data frames and any object with the Arrow PyCapsule interface are read without copies. Dictionary columns are treated as categories.
- C API for other languages, including Arrow input and the compiled mode.
- Wheels for Linux x86_64 and aarch64 (manylinux_2_28), macOS 15+ (arm64 and x86_64) and Windows x64. Python 3.10, 3.11, and abi3 for 3.12+.
Performance
LightGBM, 500 trees, Hetzner AMD EPYC-Milan (4 dedicated vCPU). Whole data set, all threads:
| lignum compiled | lleaves | LightGBM | |
|---|---|---|---|
| California housing | 0.014 s | 0.021 s | 0.140 s |
| Covertype | 0.265 s | 0.269 s | 2.401 s |
| Diabetes 130-US | 0.151 s | 0.803 s | 1.830 s |
On one thread, a single row takes 2.9-3.2 µs (lleaves 7.8-20 µs). The benchmarks page has full results for x86-64, ARM and Apple M4, including the cases where lignum is not the fastest.
Not supported
LightGBM linear trees and random forests, and multi-target / vector-leaf models. These models are rejected at load time.
Documentation: https://misterekkk.github.io/lignum/