You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Damaged .bin files could make prediction read outside the file. An element count in the
header with high bits set (e.g. the number of leaves plus 2^61) wrapped around when multiplied by
the element size, so every size check passed and every stored index looked in range. Every count
is now checked against the file size before it is used, also in the binned sections and for the
categorical tables' offsets.
Crafted XGBoost JSON / UBJSON files could crash or exhaust memory. A tree deeper than about
20,000 levels overflowed the stack, and a node shared by two parents was unfolded into every
path (29 nodes, 2 KB of JSON: 2^28 leaves). Trees are now limited to 1024 levels, like the
LightGBM parsers, and a node may have only one parent.
Large categories in a small file allocated gigabytes. A LightGBM split on category
4294967295 allocated a 512 MB bitset, and every further split as much again. Categories that no
input can match (LightGBM: from 2^31 on; XGBoost: outside [0, 2^24) or negative) are dropped,
which changes no prediction, and the category sets of a model may take at most about 256 MB.
The fallback cache of compiled libraries was shared. Without HOME (and without LIGNUM_CACHE_DIR), compile() cached libraries in the temporary folder, where another user of
the machine could place a library under the expected name. The fallback is now a folder per user, lignum-cache-<uid>, used only when it is owned by this user and nobody else can write to it.
Further memory errors on crafted files, found by the new fuzzers. An XGBoost multi-class
model with a single base_score read freed memory; a .bin file with a node shared by two parents
made predict hang (every node now has one parent, and unknown header flags are rejected); a UBJSON
object key was read after the output buffer moved; a .cbm file with empty parameters passed a
null pointer to memcpy. All are rejected or handled, with regression tests.
Added
More objectives, bit-exact: LightGBM models trained with reg_sqrt=True (regression, regression_l1, fair, quantile, mape: predict squares the score back, transform "signed_square"), XGBoost binary:hinge ("hinge": 0 or 1), and CatBoost's Focal, MultiLogloss and MultiCrossEntropy (sigmoid per output, as predict_proba). tests/python/test_objectives.py trains every objective lignum accepts and compares.
Changed
Unknown objectives are rejected at load time. Before, an objective lignum did not know got
no output transform, so predict silently differed from the library's (reg_sqrt, binary:hinge, RMSEWithUncertainty, MultiLogloss, Focal did). Every objective in the
documentation of LightGBM 4, XGBoost 3 and CatBoost is now known and tested, except CatBoost's RMSEWithUncertainty, QueryCrossEntropy (GPU only) and CatBoost models trained with a Python
objective, which are rejected with a message. LightGBM and XGBoost
models trained with a Python objective predict raw scores, as before and as the libraries do.
.bin format 0.9. It adds the categories of training; files of format 0.8 (lignum 0.4.0,
0.4.1) still load. Files written by 0.4.2 cannot be read by lignum 0.4.1 or older.
Fixed
pandas category columns get the codes of training. LightGBM (pandas_categorical) and
XGBoost 3 (cats) store the categories of training, and their predict(df) maps a DataFrame's
categories onto them. lignum used the codes as they came, so a frame whose categories were in
another order or a subset (e.g. made from the current batch with astype("category")) predicted
differently without an error (80% of the rows in a test), string categories failed to convert,
and integer categories gave their values instead of codes. lignum now reads the categories and
maps pandas category columns and Arrow dictionary columns (pyarrow, polars) the way the library
does, bit for bit: a category not in training is missing (LightGBM) or a ValueError (XGBoost),
and other category columns than in training are a ValueError, as in the libraries. One lookup
per category and an array read per row; the table is cached per categories object.
CatBoost models with more than 65,535 float and one-hot features read the wrong column. Their
index was cut to 16 bits, so a split on feature 65,541 used feature 5. Such models are now
rejected; up to 65,535 load as before (and their .bin files load too).
XGBoost categories of 2^32 and more were wrapped to small categories by a 32-bit cast (2^32 + 2
became category 2). They can never be matched and are dropped.
A model without splits crashed Arrow prediction with binned=False on x86-64 (an integer
division by zero).
A LightGBM dump_model() JSON with a feature named learner was read as XGBoost and failed.
JSON files are recognised by their first key. Malformed LightGBM and XGBoost JSON files now give
errors that name the library and the file, instead of only simdjson's error code; a category that
is not an integer is an error instead of an exception from std::stoul.