-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AutoMLs Benchmark. Why is Auto-sklearn so bad? #923
Comments
Thanks for bringing this up. We'll definitely have a look into this, but we're currently a bit swamped so it might take a bit for a definite answer. I just had a very brief look at your code and found the following things:
We're usually working with the following AutoML benchmark: https://openml.github.io/automlbenchmark/ . Could you please let us know what's the difference to your benchmark? As your results show that LightGBM and CatBoost are strong competitors, we'll probably add them to the OpenML AutoMLBenchmark then. |
Thanks for the answer.
Sometimes in the process of optimization the algorithm won without predict_proba (which is very strange), so I had to implement such a construction.
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs for the next 7 days. Thank you for your contributions. |
I'm closing this as it's very out-dated and a lot has changed by now. I don't see a good reason to keep this open. |
Any explanation as to why it generates poorer results than practically every other framework in the benchmark? Knowing that Auto-Sklearn has won numerous Kaggle championships, etc. that perplexed me, @eddiebergman @mfeurer @Alex-Lekov, may we ask to try with the latest 0.14.7 version, maybe? see if improvements have occurred solving this GitHub issue right after. Cheers |
For sure, feel free to benchmark it how you wish :) We still use the automl-benchmark as our go to set of 39 datasets to test on. We can't speak for the validity of custom benchmarks but we use this one as it's been peer reviewed and often used for comparison. I took a look at the source code and the setup for AutoSklearn and others. There seems to be no control for CPU allocation in which the default for AutoGluon is to use as much as it thinks it needs while for AutoSklearn, it just uses 1. It also seems that I took a look at the tabular results as well and it seems most frameworks are basically equal to within 0.01-0.02, meaning the ranking is a lot more susceptible to noise. As also pointed out CatBoost/LightGBM is included in It is interesting that auto-sklearn failed on the If you have other ideas as to why it might have performed so badly by their ranking system, we would love some insight. Best, |
That is obviously a comprehensive examination, @eddiebergman. Many thanks for your response and prompt answer!!🥳 I would say that the automl-benchmark peer-reviewed paper/method is significantly superior to AutoML Alex in terms of fair-comparison and reliability, as you also mentioned. Nevertheless, it was intriguing to see AutoML Alex bring that up. However, when I attempted to operate this system, nothing worked well, therefore I doubt that it is a good comparative benchmark, as you primarily asserted via various criteria. On the other hand, in my current use case, I was doing lengthy Auto-Sklearn runs on the cloud and obtaining consistently bad results (~not better than random), whereas with Auto-Gluon in a single, brief run, I was obtaining results that were far superior to random finds. This caught my eye because Auto-Sklearn has won multiple competitions and the creators (including you) continue to work on it actively! And this was a result of my AutoSklearn implementation not requesting the appropriate metric to optimise on! This may also indicate why it was badly represented in the AutoML Alex benchmark! Auto-Sklearn seems to place a greater emphasis on selecting the optimal optimisation metric than the majority of other AutoML frameworks I have examined (being yet beginner to it). (Note: I do not have much time to delve into the code of AutoML alex, but I am curious about the metrics he chose to optimise on (using Auto-Sklearn) and whether he chose the same metrics for every framework he benchmarked.) Regardless, I believe I have the correct metric now to ask my Auto-Sklearn pipeline to optimise on, although I will post a new question that you may be able to answer easily, since I was curious about something on this particular subject (i.e, the chosen metric to optimise-on by the AutoML system). See you on the other side. Thanks, once more, so much for your investigation, @eddiebergman! Wonderful day☀️ |
Regarding the metric, we don't optimize for the best metric, we use sklearn's Best, |
Now it makes more sense! Given the metric or type of target function, it can alter anyway as it is different metric and goal behind them, but it is odd that the change would vary so much indeed as you said. Perhaps my data is quite intricate, I do not know. Let's see how goes our other discussion on the metric, and I will provide you with further information on the code so that you may verify it. Thanks so much! |
In that case @Alex-Lekov would you consider a retest of AutoML, since there has been too many advancements in GB-like models? It is already possible to hot-plug the relevants algorithms into Auto-sklearn, and |
This is not a criticism - I really want to understand.
I made Benchmark AutoML libs, and Auto-sklearn showed very poor results, even worse than the usual CatBoost with standard parameters!
https://github.com/Alex-Lekov/AutoML-Benchmark/
I run the benchmark in docker - so you can easily reproduce it
here is the code from the benchmark:
Is the code correct? (I do not adjust the advanced parameters, since AutoML, in theory, should pick everything up by itself, that's why it is AutoML)
Tell me what am I doing wrong? or is this a real result and the library is really that bad?
The text was updated successfully, but these errors were encountered: