add genai metrics endpoint in UI for model overview metrics #2517
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imatiach-msft
requested review from
tongyu-microsoft,
xuke444,
RubyZ10 and
vinuthakaranth
as code owners
kartik727
approved these changes
Jan 29, 2024
imatiach-msft
added a commit
to imatiach-msft/responsible-ai-toolbox
that referenced
this pull request
Jan 30, 2024
imatiach-msft
added a commit
to imatiach-msft/responsible-ai-toolbox
that referenced
this pull request
Jan 30, 2024
imatiach-msft
added a commit
that referenced
this pull request
Jan 30, 2024
3 tasks
imatiach-msft
added a commit
that referenced
this pull request
Feb 2, 2024
* Added info about required packages * Update responsibleaidashboard-question-answering-model-debugging.ipynb * show example prediction * Update responsibleaidashboard-question-answering-model-debugging.ipynb * add genai metrics Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Add genai task type Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * add genai task type (#2494) * Added info about required packages * Update responsibleaidashboard-question-answering-model-debugging.ipynb * show example prediction * Update responsibleaidashboard-question-answering-model-debugging.ipynb * add genai metrics Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Add genai task type Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> --------- Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * remove target_column requirement for genai tasks Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * genai dev demo notebook Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Added info about required packages * Update responsibleaidashboard-question-answering-model-debugging.ipynb * add genai metrics Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Add genai task type Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * remove target_column requirement for genai tasks (#2501) Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * add demo notebook (#2502) * remove target_column requirement for genai tasks Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * genai dev demo notebook Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> --------- Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * added generative text explainer for OpenAI models Signed-off-by: Mohsin Shah <websitemohsinshah@gmail.com> * update UI to support genai task for RAI text dashboard (#2504) * generative ai explanations for all rows of dataset Signed-off-by: Mohsin Shah <websitemohsinshah@gmail.com> * move metrics to utils submodule Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * error analysis support for genai text Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * error analysis support (#2510) * move metrics to utils submodule Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * error analysis support for genai text Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> --------- Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * fix some linting issues Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * more linting fixes Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Kartikch/add genai metrics (#2513) * update UI to support genai task for RAI text dashboard (#2504) (#2508) * move metrics to utils submodule Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * error analysis support for genai text Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * fix some linting issues Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * more linting fixes Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> --------- Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> Co-authored-by: Ilya Matiach <ilmat@microsoft.com> * add genai metrics endpoint in UI for model overview metrics (#2517) * Add socketio event handler for generative text metrics Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * add converter from genai task to model type, fix endpoint results (#2518) * Add support for evaluation model in RAITextInsights Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * fix linting issues Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Remove commented out code for generative text tasks Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * update genai metric compute function for new ml wrapper interface Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * add output examples in genai metric prompts Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * fix merge errors Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * add methods and constants for genai task type Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * add missing files for genai metrics Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * update copyright information Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Fix target_column assignment in ErrorAnalysisManager Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Remove duplicate import and refactor if condition in RAITextInsights class Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Refactor rating examples in genai_metrics scripts Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Refactor debug_ml method to handle missing text_cols attribute Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Remove unnecessary code for generative text models Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> * Rearranged import statements in _compute.py Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> --------- Signed-off-by: Kartik Choudhary <kartikchoudh@umass.edu> Signed-off-by: Mohsin Shah <websitemohsinshah@gmail.com> Co-authored-by: Mohsin Shah <websitemohsinshah@gmail.com> Co-authored-by: Ilya Matiach <ilmat@microsoft.com> Co-authored-by: Mohsin Shah <106512207+mohsinposts@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
add genai metrics endpoint in UI for model overview metrics
Checklist