Add API for storing trained model metadata #881
Labels
feature
Feature requests or pull request implementing a new feature
help-wanted
An issue currently lacks a contributor
Is your feature request related to a problem? Please describe.
There are many existing libraries and platforms providing amazing metrics tracking capabilities, e.g. MLFLow's auto logging and Comet.ml's hosted solution with very nice UI.
In a typical ML workflow, the data science team in the model development phase, may use an experimentation management platform to produce lots of trained models and then go through a model selection process where they will be comparing different models and decide which ones work the best. In that scenario, tools like MLFlow and Comet.ml can give the user lots of insight into the training process and what are the algorithms or model architecture that works for their specific problem.
On the other hand, BentoML is a tool focusing on serving and deploying trained models. Once the data scientist has selected a model via the model development phase, they can use BentoML to productionize the trained model and make it ready for production-grade deployment.
In a sense, models that are saved to BentoML's model registry YataiService are the "golden models", they are produced by a production training pipeline and ready to be tested and shipped to serve production traffic.
Hoever, when looking at BentoML's saved bundle before deploying it, the user may still want to see some context information about the trained model. For example, the version of the training dataset used, the training parameters used, the experimentation/training job ID in their experimentation management platform, or the evaluation results(precision, recall, inception score, BPC, etc).
This information makes it a lot easier for users to trace back if they run into an issue with a new version of their BentoML Service, and to make decisions about which service to deploy.
Describe the solution you'd like
Here is to propose an API for adding context information in a loosely typed JSON blob, and allow viewing this information from CLI and Web UI.
The user added metadata will be inserted in the
bentoml.yml
file under the saved bundle directory:Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: