Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: pycaret compare_models MAE increase when add installed tpot #3983

Open
2 of 3 tasks
showkeyjar opened this issue Apr 25, 2024 · 1 comment
Open
2 of 3 tasks
Labels
bug Something isn't working

Comments

@showkeyjar
Copy link

showkeyjar commented Apr 25, 2024

pycaret version checks

Issue Description

I installed pycaret in conda env, and try to compare models with my training data for a regression task,
but it had different results every time, even set session_id and random seed.
It means the best result can not be reproduced.

so, I continue installed tpot to check result, but when I ran pycaret again, the MAE grew 5x than before.

I feel very confused, I'm running pycaret in jupyterlab, and the console has no error.

If I create new env only install pycaret, there is no error any more.

Reproducible Example

`pip install pycaret[full]`

from pycaret.regression import RegressionExperiment

exp.setup(df_samples[ml_sel_cols], target='y', session_id=123)

best = exp.compare_models(exclude=['lr', 'lar', 'llar', 'lasso', 'omp', 'knn', 'huber', 'par', 'ridge', 'dummy', 'catboost'])

`conda install -c conda-forge tpot`

then run upper code again

Expected Behavior

the MAE of two times must be same or near.
and the best result must be reproduced.

Actual Results

but the first best MAE is 0.98

and the second best MAE is 4.99

Installed Versions

python 3.11
conda 22.90
pycaret 3.3.1 and 3.3.0 both tested
tpot 0.12.2
jupyterlab 4.1.5

@showkeyjar showkeyjar added the bug Something isn't working label Apr 25, 2024
@showkeyjar showkeyjar changed the title [BUG]: when add install tpot, then pycaret compare_models MAE increased [BUG]: pycaret compare_models MAE increased when add installed tpot Apr 25, 2024
@showkeyjar showkeyjar changed the title [BUG]: pycaret compare_models MAE increased when add installed tpot [BUG]: pycaret compare_models MAE increase when add installed tpot Apr 25, 2024
@showkeyjar
Copy link
Author

and I found new bugs, maybe is the same of above: when comparing_models multiple times and testing again, the pipline of the previous version of the model may be used.

it happend in jupyterlab, I used two different datas(data1, data2) to compare_models twice, but the second time predict raise error: no columns XXX found. those missing columns are in data1

it works when I saved model and restart kerenal and rerun it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant