[FR] Add class_probability to the input "eval_df" of "custom_metrics" function in recipes #10323
Open
1 of 22 tasks
Labels
area/recipes
MLflow Recipes, Recipes APIs, Recipes configs, Recipe Templates
enhancement
New feature or request
Willingness to contribute
Yes. I would be willing to contribute this feature with guidance from the MLflow community.
Proposal Summary
For developing some metrics, one would need to use the output probability.
I propose to add class_probability to the input "eval_df" of "custom_metrics" function.
Motivation
For some classification algorithms, I'd like to use precision_recall_auc as the primary metric however I cannot implement it, as it requires the output probability.
In many imbalanced datasets, these metrics are the suggested metrics in the community.
For the estimators that do not support "predict_prob" these columns could be removed or set to None in the "eval_df" DF.
Currently, the precision_recall_auc or roc_auc values are automatically computed in the "evaluation" step if the estimator has some specifications, a similar approach can be used in the recipes.
Details
No response
What component(s) does this bug affect?
area/artifacts
: Artifact stores and artifact loggingarea/build
: Build and test infrastructure for MLflowarea/docs
: MLflow documentation pagesarea/examples
: Example codearea/gateway
: AI Gateway service, Gateway client APIs, third-party Gateway integrationsarea/model-registry
: Model Registry service, APIs, and the fluent client calls for Model Registryarea/models
: MLmodel format, model serialization/deserialization, flavorsarea/recipes
: Recipes, Recipe APIs, Recipe configs, Recipe Templatesarea/projects
: MLproject format, project running backendsarea/scoring
: MLflow Model server, model deployment tools, Spark UDFsarea/server-infra
: MLflow Tracking server backendarea/tracking
: Tracking Service, tracking client APIs, autologgingWhat interface(s) does this bug affect?
area/uiux
: Front-end, user experience, plotting, JavaScript, JavaScript dev serverarea/docker
: Docker use across MLflow's components, such as MLflow Projects and MLflow Modelsarea/sqlalchemy
: Use of SQLAlchemy in the Tracking Service or Model Registryarea/windows
: Windows supportWhat language(s) does this bug affect?
language/r
: R APIs and clientslanguage/java
: Java APIs and clientslanguage/new
: Proposals for new client languagesWhat integration(s) does this bug affect?
integrations/azure
: Azure and Azure ML integrationsintegrations/sagemaker
: SageMaker integrationsintegrations/databricks
: Databricks integrationsThe text was updated successfully, but these errors were encountered: