Skip to content

v0.4.0

Choose a tag to compare

@misterekkk misterekkk released this 27 Oct 21:14
· 36 commits to main since this release

Changed

  • Breaking: column names are checked. When a model was trained on named columns and the input
    has column names (pandas DataFrame, pyarrow, polars or other Arrow data), predict checks that
    the columns start with the model's feature_names, in the same order, and raises ValueError
    otherwise. Before, a reordered or missing column silently gave other predictions. numpy arrays,
    frames without column names, models without names and the C API are unchanged (by position).
    predict(..., validate_features=False), load(..., validate_features=False) or
    model.validate_features = False turn the check off; it costs 0-5 µs per call (Apple M4).

  • Breaking: .bin format 0.8. It adds the feature names, the library the model was trained
    with, its objective and the lignum version that wrote the file. Files written by lignum 0.3 are rejected: save
    them again from the original model.

  • Compiled-library files (Model.compile()): without library=, libraries go to a per-user
    cache (~/.cache/lignum, ~/Library/Caches/lignum, %LOCALAPPDATA%\lignum\Cache, or
    LIGNUM_CACHE_DIR) instead of lignum_compiled in the temp directory, and the code is built in a
    temporary folder that is always removed: no .c, .o or log files are left behind (a failed
    build keeps only its log, named in the error together with the compiler's last lines).
    library= writes the library exactly there, always compiling it and bypassing the cache.

  • Breaking: compile() has no work_dir any more (set LIGNUM_CACHE_DIR to move the cache),
    and its options after flags are keyword-only. In the C API the fourth argument of
    LignumModelCompile is now the cache folder (NULL: the per-user cache).

Added

  • Model inspection: Model.n_trees, Model.source ("lightgbm", "xgboost" or "catboost"),
    Model.objective (the library's name, e.g. "binary", "multi:softprob", "Logloss") and
    Model.info(), a dict that adds the output transform, the number of leaves, the kernel in use,
    the compiled library and, for .bin files, the lignum version that wrote them. repr(model)
    shows the source, objective, trees and features. C API: LignumModelNumTrees.
  • Model.feature_names: the names of the training columns (LightGBM, XGBoost and CatBoost files,
    kept in .bin), or None for models trained on a matrix.
  • lignum.cache_dir(), lignum.clear_cache() and Model.compiled_library (the path of the
    library in use).
  • compile(library=folder) writes lignum_<model fingerprint>_<platform>.so (.dll on Windows)
    into the folder, and load(..., compiled=folder) picks the file for the model and platform, so one
    folder can hold the libraries of several models and platforms.

Fixed

  • Processes compiling the same model at once no longer write the same files: each builds in its own
    folder and the library is moved into place with an atomic rename.
  • The cache key includes the CPU architecture: x86-64 (e.g. under Rosetta) and arm64 processes no
    longer load each other's libraries. The key is also stable across compilers (FNV-1a instead of
    std::hash), and a cached library that fails to load is rebuilt.
  • Errors when loading a library built for another platform name the platform that is needed.
  • XGBoost models that were predicted wrongly without an error are now rejected: linear boosters
    (booster="gblinear", which loaded as an empty forest predicting base_score) and DART models
    that dropped trees (rate_drop > 0: XGBoost weights each tree, lignum ignored the weights). DART
    files from XGBoost before 3.4 (trees nested under gbtree) used to load as an empty forest too;
    they now predict correctly when no tree was dropped. LightGBM DART models were and are exact.

The old lignum_compiled folder in the temp directory is no longer used and can be deleted.