Explainability - Phase 1

[andrewballantyne]

• UX Effort: Explainability sig-ml-developer-experience#29

Target Branch
f/mserving-metrics

Foundation Work

• Explainability: Fairness and Bias Metrics (Phase 0) #1001
• Backend proxy for TrustyAI Service Client #1151
• TrustyAI Service Client #1152

P0: Ability to navigate to a tab dedicated to model bias reporting for each served modelConfigured bias reports are unique to a served model.

• Explainability: Fairness and Bias Metrics (Phase 0) #1001

P0: Ability to opt in to and configure bias detection for models already in production

Display message indicating model bias reporting is not configured
Provide option to configure bias metric reporting
Give users list of available fairness metrics
User to select a metric to monitor over the lifetime of the versioned model
Configure selected fairness metric, calculation and bias detection
User inputs the sensitive feature/group/label/column/facet and corresponding values applicable to their model
Default to Trusty-provided mathematically sensible baseline defaults and thresholds
Give user the ability to update thresholds to their preferred values
Configure metric resolution
User provided name for monitoring job
Allow user to schedule metric calculation job cadence
User has ability to update default interval
Button to save or cancel configuration
Store configuration state for use in dashboard
Configured dashboards persist over the lifetime of the served model version
Return to main page, ability to add new configuration
Allow user to configure multiple facets for any of their selected metrics For example, F1 and F2 for SPD, but only F1 for DIR

• Explainability: Fairness and Bias Metrics (Phase 0) #1001
• [SPIKE] Metric registration #1153
• [SPIKE] Update metric registrations #1161
• Bias metric configuration list screen - visual elements #1269
• Display configured bias metrics from TrustyAI #1275
• Indicate when no bias metrics are configured on model server #1267
• Custom empty state for bias tab #1154
• Configure bias metric form #1274

P0: Configured bias detection graphs over time

Clearly state the bias metric in the graph title
User can select which group/label/column to view from dropdown
User friendly description of monitoring job
Display calculated metrics over dates and time
Render baseline and threshold values and clearly indicate threshold violations
Ability to update the graph’s refresh rate
Adjust viewable time range, for example: 30 minutes to 3 days

• Explainability: Fairness and Bias Metrics (Phase 0) #1001
• Multi-metric display on model bias screen #1273
• Display user defined threshold values on charts #1163
• Threshold violation alerting #1158
• Configurable chart refresh rate #1159
• Dynamic fetching of chart descriptions #1160

P0: Ability to quickly reference bias detection documentation within product

Hover for description of metric and it’s usage

• Dynamic fetching of chart descriptions #1160
• Possibly requires one more ticket when mocks available.

P0: Ability to update report as needed

User can update, add or remove metric configurations
User can add a metric configurations, same as above
User can remove/delete a configuration. Configuration instance will no longer be calculated nor displayed
User can update variables used in configuration instance. E.G.
User can update calculation resolution - 5000 -> 1000 observations
User can change threshold
Updates will recalculate and display new report
Current configuration persisted and used going forward

• [SPIKE] Metric registration #1153
• [SPIKE] Update metric registrations #1161
• Deletion of metric registrations #1162
• Display user defined threshold values on charts #1163
• Configure bias metric form #1274

Additional Requirement [19/05/2023]:

User can add TrustyAI service per Data Science Project
User can edit TrustyAI service details
User can delete TrustyAI service per Data Science Project

• Adds support for TrustyAI Operator #1276

Supplementary work

• Update Trusty AI client to handle API changes #1336
• [BUG]: Configure bias metric form does not work on submission #1359
• Enhance validation and defaults of bias metric config form #1383
• Feature flag for Bias #1384
• Minor enhancements to bias chart #1386
• [BUG] Bias charts auto-refresh with stale data #1403
• Retrieve bias metrics from thanos querier #1472

Additional issues raised

• Explainability Phase 1: Incubation refinements #1447
• [Story]: Support Metrics on the Backend #1511
• [BUG] When enabling TrustyAI on a project, the dashboard immediately shows success notification #1531
• [BUG] When trying to uninstall TrustyAI from a project the dashboard shows error "Could not find TrustyAI service", even though the TrustyAI service is installed  #1532
• [BUG] Bias metric chart selector shows a selected, non-existent chart when metrics are deleted by direct call to the TrustyAI API #1533
• [Bug] Error message on enabling TrustyAI via UI are not descriptive #1996
• [Story]: Add project name to the session storage key for the selected metrics chart #2022
• [Story]: Improve error flow when TrustyAI CR fetch times out  #2023
• [Story]: Provide a better experience when on the project settings tab when BiasMetrics feature flag disabled #2024
• [Story]: Update f/mserving-metrics branch to work with new Area concept #2088
• [Story]: Hide metrics when project is kserve enabled #2253

After incubation begins:

• Microcopy review #1526
• [Story]: Create integration test tickets #2087

Dependent on:

• [Tracker]: Migrate metrics to UWM #1854

[alexcreasy]

Update: a PR has been raised for UWM (#1854) it is awaiting review and unblocks development of any remaining tickets for Explainability.

[alexcreasy]

Update: development is in progress on the remaining issues.

[alexcreasy]

Update: Still working through the last couple of issues, we have one new ticket before incubation, which is #2088

[dgutride]

Migrated to https://issues.redhat.com/browse/RHOAIENG-243

[dgutride]

All subtasks migrated - moving tracker to closed