tensor_search_helpers.cpp

[Kirill-Wwizarddd-Vladimirovich]

Problem: _catboost.CatBoostError: catboost/private/libs/algo/tensor_search_helpers.cpp:99: This should be unreachable
catboost version: 1.1.1
Operating System: Windows 11
CPU: appnode (360GB)
GPU: -
При запуске процесса подбора гиперпараметров на 3-5 итерации процесс обучения останавливается. Параметры из конфига представлены ниже
tunning:
cv_params:
bagging_temperature:
max: 1000
min: 0
optional: true
param_suggestion: suggest_float
bootstrap_type:
param_suggestion: suggest_categorical
values:
- Bayesian
- Bernoulli
- MVS
- 'No'
border_count: 254
depth:
max: 10
min: 4
param_suggestion: suggest_int
early_stopping_rounds: 20
grow_policy:
param_suggestion: suggest_categorical
values:
- SymmetricTree
- Depthwise
- Lossguide
iterations: 1000
l2_leaf_reg:
max: 10
min: 0.01
param_suggestion: suggest_loguniform
learning_rate:
max: 1
min: 0.001
param_suggestion: suggest_loguniform
loss_function:
param_suggestion: suggest_categorical
values:
- RMSE
- MAE
min_data_in_leaf:
max: 1000
min: 50
param_suggestion: suggest_int
one_hot_max_size:
max: 20
min: 2
param_suggestion: suggest_int
random_seed: 42
random_strength:
max: 10
min: 0
param_suggestion: suggest_float
subsample:
max: 1
min: 0.1
optional: true
param_suggestion: suggest_float
thread_count: 8
direction: maximize
fold_id_column: fold_id
model_class_name: CatBoostSoloModel
model_module_name: uplift_model.model.solo_model
n_trials: 100
target_column: fm_plu
train_data_table: kirill_ivanovich_stm_single_20240517_card1598771_dataset
Полный лог ошибки:
0: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 5.88s remaining: 1h 37m 52s
1: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 11.6s remaining: 1h 36m 16s
2: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 17.4s remaining: 1h 36m 30s
3: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 23.4s remaining: 1h 37m 5s
4: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 29.5s remaining: 1h 37m 52s
5: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 35.6s remaining: 1h 38m 14s
6: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 41.3s remaining: 1h 37m 43s
7: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 47.3s remaining: 1h 37m 47s
8: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 53.2s remaining: 1h 37m 42s
9: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 59.1s remaining: 1h 37m 32s
10: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 5s remaining: 1h 37m 33s
11: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 10s remaining: 1h 37m 14s
12: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 16s remaining: 1h 37m 2s
13: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 22s remaining: 1h 36m 44s
14: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 28s remaining: 1h 36m 54s
15: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 34s remaining: 1h 36m 39s
16: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 40s remaining: 1h 36m 41s
17: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 46s remaining: 1h 36m 39s
18: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 52s remaining: 1h 36m 32s
19: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 1m 57s remaining: 1h 36m 17s
20: learn: 0.0000000 test: -307523.2127502 best: -307523.2127502 (0) total: 2m 3s remaining: 1h 36m 8s
Stopped by overfitting detector (20 iterations wait)

bestTest = -307523.2128
bestIteration = 0

Shrink model to first 1 iterations.[I 2024-05-15 13:31:13,591] Trial 1 finished with value: -362739.55468349456 and parameters: {'bootstrap_type': 'No', 'depth': 8, 'grow_policy': 'SymmetricTree', 'l2_leaf_reg': 0.05752738804460068, 'learning_rate': 0.06903620003566195, 'loss_function': 'RMSE', 'min_data_in_leaf': 518, 'one_hot_max_size': 14, 'random_strength': 0.4181026889484596}. Best is trial 0 with value: -362696.8346348881.
[W 2024-05-15 13:31:20,423] Trial 2 failed with parameters: {'bootstrap_type': 'Bayesian', 'depth': 4, 'grow_policy': 'Depthwise', 'l2_leaf_reg': 0.03418701506102692, 'learning_rate': 0.023356861235220554, 'loss_function': 'RMSE', 'min_data_in_leaf': 708, 'one_hot_max_size': 8, 'random_strength': 9.80591372473501, 'bagging_temperature': 753.3913606426174} because of the following error: CatBoostError('catboost/private/libs/algo/tensor_search_helpers.cpp:99: This should be unreachable').
Traceback (most recent call last):
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 200, in _run_trial
value_or_values = func(trial)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model_selection/param_optimizer.py", line 140, in optuna_func
return self.objective(trial, cv_folds)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model_selection/param_optimizer.py", line 103, in objective
common_uplift_flag=self.common_uplift_flag,
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model_selection/cv.py", line 64, in cv
optimize_params=True,
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model/utils.py", line 36, in wrapper
return func(self, *args, **kwargs)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model/solo_model.py", line 73, in fit
use_best_model=True
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/catboost/core.py", line 2513, in fit
save_snapshot, snapshot_file, snapshot_interval, init_model, callbacks, log_cout, log_cerr)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/catboost/core.py", line 2360, in _fit
train_params["init_model"]
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/catboost/core.py", line 1759, in _train
self._object._train(train_pool, test_pool, params, allow_clear_pool, init_model._object if init_model else None)
File "_catboost.pyx", line 4623, in _catboost._CatBoost._train
File "_catboost.pyx", line 4672, in _catboost._CatBoost._train
_catboost.CatBoostError: catboost/private/libs/algo/tensor_search_helpers.cpp:99: This should be unreachable
[W 2024-05-15 13:31:20,425] Trial 2 failed with value None.

Traceback (most recent call last):
File "uplift_model/pipelines/optimize_params.py", line 87, in
feature_config_path=args.feature_config,
File "uplift_model/pipelines/optimize_params.py", line 66, in optimize_params
optimizer.search_params()
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model_selection/param_optimizer.py", line 151, in search_params
n_trials=self.tunning_params['n_trials'],
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/optuna/study/study.py", line 451, in optimize
show_progress_bar=show_progress_bar,
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 76, in _optimize
progress_bar=progress_bar,
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 163, in _optimize_sequential
frozen_trial = _run_trial(study, func, catch)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 251, in _run_trial
raise func_err
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 200, in _run_trial
value_or_values = func(trial)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model_selection/param_optimizer.py", line 140, in optuna_func
return self.objective(trial, cv_folds)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model_selection/param_optimizer.py", line 103, in objective
common_uplift_flag=self.common_uplift_flag,
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model_selection/cv.py", line 64, in cv
optimize_params=True,
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model/utils.py", line 36, in wrapper
return func(self, *args, **kwargs)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/uplift_model/model/solo_model.py", line 73, in fit
use_best_model=True
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/catboost/core.py", line 2513, in fit
save_snapshot, snapshot_file, snapshot_interval, init_model, callbacks, log_cout, log_cerr)
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/catboost/core.py", line 2360, in _fit
train_params["init_model"]
File "/home/local/X5/kirill.ivanovich/cvm-uplift-model/venv/lib/python3.7/site-packages/catboost/core.py", line 1759, in _train
self._object._train(train_pool, test_pool, params, allow_clear_pool, init_model._object if init_model else None)
File "_catboost.pyx", line 4623, in _catboost._CatBoost._train
File "_catboost.pyx", line 4672, in _catboost._CatBoost._train
_catboost.CatBoostError: catboost/private/libs/algo/tensor_search_helpers.cpp:99: This should be unreachable
Command exited with non-zero status 1
Command being timed: "python uplift_model/pipelines/optimize_params.py --config=presets/run_stm-single-20240517-card1598771.yml --feature_config=presets/run_stm-single-20240517-card1598771_features.yml --appnode=True"
User time (seconds): 2747.20
System time (seconds): 162.51
Percent of CPU this job got: 105%
Elapsed (wall clock) time (h:mm:ss or m:ss): 45:55.25
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 16276200
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 74214636
Voluntary context switches: 310238
Involuntary context switches: 183553
Swaps: 0
File system inputs: 296
File system outputs: 3496
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 1

[ek-ak]

Hello!
You are using an old version of the CatBoost package, please try the latest version.