-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance Metrics & Fitness Functions #33
Comments
Yes. I think we should eventually support allowing the user to pass arbitrary scoring functions, similar to how sklearn does it.
As in, use one scoring functions for optimization then a different scoring metric for final model selection? Interesting idea. I'm currently working on a version of TPOT that allows it to optimize on multiple criteria simultaneously, so perhaps that will help in this regard.
That's what we currently do with accuracy. I think it makes sense to do the same with other measures. |
Would passing in a keyword for the specific metric work for now?
Yeah, I was thinking that and/or adding additional metrics to the report of the final model.
I'm happy to contribute by adding simple support for F1/Precision/Recall in the same vein as the accuracy. I think it should be relatively straightforward. |
At first thought, I think it'd be better/easier to simply allow the user to pass an arbitrary scoring function. Otherwise, we have to choose what scoring functions to support, write a special case for each one, etc. Not very scalable from a coding point of view. Of course, we'd have to also clarify that the user should provide a scoring function that's appropriate for their data.
Is there a specific use case for this that you can think of?
Would you be interested in making an attempt at the implementation that allows the passing of arbitrary scoring functions discussed above? Perhaps we could also provide some example snippets in the docs of how to expand F1/precision/recall/etc. to support multiple classes, which can then be passed as the arbitrary scoring function. |
I see your point about making the effort now to support the arbitrary functions. I'll make an attempt at it and provide examples for at least F/P/R. |
I imagine that it would be useful to use TPOT to find models that optimize alternate performance metrics like precision, recall, etc. As such, I've come up with the following brainstorming questions:
There are plenty of other questions, but I figured this would be a decent place to start. Let me know if I'm totally misguided in proposing this -- I won't profess to be an expert in genetic programming.
The text was updated successfully, but these errors were encountered: