Catboost vs ONNX runtime : very large RAM usage. Expected ?

[FlorianIragne]

Hi,

i have a model from catboost. It weights ~ 100MB on disk (dumped by joblib) and around 250MB in RAM when loaded "natively" with catboost. If i save it as onnx format using catboost save_model() function, it weights ~ 500MB on disk and when loaded in an InferenceSession, it takes ~ 5.5GB in ram ! So roughly 22 times the RAM usage of native catboost.

Is it expected ?
Btw, there's also performance degradation using ort : 10k predictions take 32s with catboost, but ~55s with ort

Doing this CPU only, with python 3.11, catboost 1.1.1 and onnxruntime 1.14.1

Am i missing something ?

Thanks for any advice

[cbourjau]

Technically, this would appear to be an onnxruntime issue, but it is also reflected in the standard. I could imagine that you are using many categorical features? The TreeEnsemble operators in the standard do not have a node mode for sets which would be primarily used for categorical features. Instead, the test for set membership is "rolled out" over several nodes each testing a single value with equality. This is less efficient to serialize on disk as well as to compute at runtime. That said, onnxruntime could be improved to recognize and optimize trees of this kind.

[FlorianIragne]

The model is a CatBoostRegressor from catboost with 25k trees. I don't know what is considered as "many" categorical features, but i think it may qualifies as this.
The performance degradation of predict is not really an issue yet, but the RAM usage definitely is. Do you have any advice for getting back to a manageable amount of RAM ? Or is it just the way it works right now with this kind of model ?
Anyway, thanks for your answer !

[cbourjau]

RAM-wise, I wonder if the onnxruntime can handle DAGs instead of trees. The standard is underspecified and you may be able to build a graph where each respective BRANCH_EQ node of a set points to the same downstream nodes rather than copy all its descendants.

@xadupre has worked on those operators in the past and might have some further insights.

[xadupre]

As @cbourjau said, there is no rule (x in set(...)) in onnx, these rule are unfolded into many rule ==. That would explain the memory increase. This is true for lightgbm. The function doing that is https://github.com/onnx/onnxmltools/blob/main/onnxmltools/convert/lightgbm/operator_converters/LightGbm.py#L673. However, the catboost exporter is implemented by catboost. I did not look what they do in that specific case. onnxruntime can handle DAGs.

onnxruntime does not support sparse but unless you run a big batch, it should not explain the huge memory increase.

It could be possible to create new features based on nodes using a rule (x in set(...)). This should reduce the memory increase but it could lead to the addition of many features.

You may want to use the latest nightly build. I made some modifications last week to reduce the memory footprint by 60% for TreeEnsemble in onnxruntime.