Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug with Cross Validation #2221

Closed
olegkorshunov opened this issue Nov 10, 2022 · 2 comments
Closed

Bug with Cross Validation #2221

olegkorshunov opened this issue Nov 10, 2022 · 2 comments
Assignees

Comments

@olegkorshunov
Copy link

olegkorshunov commented Nov 10, 2022

Problem: Bug with Cross Validation ( 'task_type':'GPU')
catboost version: 1.1.1
Operating System: Linux
CPU: Intel(R) Xeon(R) CPU @ 2.00GHz
GPU: Tesla T4

after I train catboost CV in 1st time everything is okay, but if I try train CV in second time and change "fold_count" I got an error

Training on fold [0/10]
---------------------------------------------------------------------------
CatBoostError                             Traceback (most recent call last)
[<ipython-input-6-75efc1527f1c>](https://localhost:8080/#) in <module>
      9   'early_stopping_rounds':3,
     10 } 
---> 11 cv_data = cv(**cv_params)

[/usr/local/lib/python3.7/dist-packages/catboost/core.py](https://localhost:8080/#) in cv(pool, params, dtrain, iterations, num_boost_round, fold_count, nfold, inverted, partition_random_seed, seed, shuffle, logging_level, stratified, as_pandas, metric_period, verbose, verbose_eval, plot, plot_file, early_stopping_rounds, save_snapshot, snapshot_file, snapshot_interval, metric_update_interval, folds, type, return_models, log_cout, log_cerr)
   6649         if not return_models:
   6650             return _cv(params, pool, fold_count, inverted, partition_random_seed, shuffle, stratified,
-> 6651                     metric_update_interval, as_pandas, folds, type, return_models)
   6652         else:
   6653             results, cv_models = _cv(params, pool, fold_count, inverted, partition_random_seed, shuffle, stratified,

_catboost.pyx in _catboost._cv()

_catboost.pyx in _catboost._cv()

CatBoostError: catboost/cuda/methods/boosting_progress_tracker.cpp:171: Saved model's params are different from current model's params

image
image

@andrey-khropov andrey-khropov self-assigned this Apr 23, 2023
robot-piglet pushed a commit that referenced this issue Apr 25, 2023
…onger rely on snapshots for folds on GPU so do not enable them and do not require AllowWriteFiles. This also makes some cludges unnecessary... #1893.. #2221
@Master-sniffer
Copy link

I have the same error even though I don't change folds. This problem seems to happen only with GPU and it is unclear if the bug was fixed

@andrey-khropov
Copy link
Member

I have the same error even though I don't change folds. This problem seems to happen only with GPU and it is unclear if the bug was fixed

Yes, it should be fixed even if you don't change folds. The fix will be included in the next release, you can also try to build from source from master and check if it works in your case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants