Feat/multiclass metrics

[IanAtCredo]

Describe your changes

Setting up model types to enable multiclass classification metric routing.

Checklist before requesting a review

• I have performed a self-review of my code
• I have built basic tests for new functionality (particularly new evaluators)
• If new libraries have been added, I have checked that readthedocs API documentation is constructed correctly
• Will this be part of a major product update? If yes, please write one phrase about this update.

Extra-mile Checklist

• I have thought expansively about edge cases and written tests for them

[IanAtCredo]

To do:

• add multiclass classification metrics to list
• Route a multiclass classification model (or dataset?) to the appropriate metrics
  • find_metrics has a metric_category attribute that we can probably use. We'd need a helper function to go from a list of artifacts -> metric category
• better messaging if metrics fail

Questions:

• How do we want to deal with models that we can't identify? For instance, if the model is non-sklearn we won't know what kind of classification it is. Options:
  • We could update the type based on what we see in the dataset (there is a type_of_target function which could inform things). That may be a nice thing to do!
  • Classification type routes to binary classification and we have a good warning?

[fabrizio-credo]

• Completed update of MULTICLASS_CLASSIFICATION_FUNCTIONS
• Created our own version for TPR, TNR

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
credoai
__init__.py	3	0	100%
credoai/artifacts
__init__.py	7	0	100%
credoai/artifacts/data
__init__.py	0	0	100%
base_data.py	107	13	88%	55, 136, 155, 158, 173, 180, 187, 191, 195, 199, 211, 214, 221
comparison_data.py	63	13	79%	53, 60, 71, 76, 81, 90, 96, 100, 105, 114, 147, 153, 156
tabular_data.py	40	6	85%	52, 73, 77, 96, 98, 105
credoai/artifacts/model
__init__.py	0	0	100%
base_model.py	36	2	94%	56, 88
classification_model.py	23	1	96%	44
comparison_model.py	11	0	100%
constants_model.py	2	0	100%
regression_model.py	11	4	64%	43–45, 48
credoai/evaluators
__init__.py	15	0	100%
data_fairness.py	147	12	92%	83–90, 205, 260–261, 287, 311, 334–340, 356
data_profiler.py	34	2	94%	57, 60
deepchecks.py	40	3	92%	113–122
equity.py	153	31	80%	78, 181–184, 204, 230–257, 281–296, 307–309, 358–359
evaluator.py	70	8	89%	50, 58, 61, 80, 106, 126, 174, 181
fairness.py	147	12	92%	117, 238, 246–251, 312–321, 323, 335–338
feature_drift.py	59	1	98%	66
identity_verification.py	112	2	98%	144–145
model_profiler.py	74	12	84%	128–131, 145–148, 165, 182–183, 192–193, 231
performance.py	119	14	88%	110, 137–143, 232–241, 243, 260–263
privacy.py	118	4	97%	410, 447–449
ranking_fairness.py	134	14	90%	136–137, 157, 178, 184–185, 382–404, 409–439
security.py	96	1	99%	297
shap.py	87	14	84%	119, 127–128, 138–144, 170–171, 253–254, 284–292
survival_fairness.py	67	50	25%	29–33, 36–48, 53–64, 67–78, 81–99, 102, 105, 108
credoai/evaluators/utils
__init__.py	3	0	100%
fairlearn.py	18	2	89%	46, 59
utils.py	8	1	88%	9
validation.py	80	28	65%	14, 34–35, 37–39, 46, 67–74, 80–86, 89, 95–98, 105, 108, 111, 114–115, 119–121
credoai/governance
__init__.py	1	0	100%
credoai/lens
__init__.py	2	0	100%
lens.py	189	12	94%	173–174, 210–215, 272, 314, 338, 420, 435, 439, 451
pipeline_creator.py	60	12	80%	20–21, 37, 79–91
utils.py	39	28	28%	20–27, 49–52, 71–82, 99, 106–109, 128–135
credoai/modules
__init__.py	3	0	100%
constants_deepchecks.py	2	0	100%
constants_metrics.py	19	0	100%
constants_threshold_metrics.py	3	0	100%
metric_utils.py	24	18	25%	15–30, 34–55
metrics.py	61	7	89%	62, 66, 69–70, 73, 83, 120
metrics_credoai.py	134	62	54%	43–72, 92–101, 106–108, 131–159, 175–178, 205, 229–230, 293–295, 371–377, 413–414, 484–485
stats.py	39	28	28%	11–14, 17–22, 25–27, 30–35, 38–52, 55–60
stats_utils.py	5	3	40%	5–8
credoai/utils
__init__.py	5	0	100%
common.py	102	40	61%	55, 68–69, 75, 84–91, 96–104, 120–126, 131, 136–141, 152–159, 186
constants.py	2	0	100%
dataset_utils.py	61	35	43%	23, 26–31, 50, 54–55, 88–119
logging.py	55	13	76%	10–11, 14, 19–20, 23, 27, 44, 58–62
model_utils.py	30	11	63%	14–19, 29–30, 35–40
version_check.py	11	1	91%	16
TOTAL	2731	520	81%

[fabrizio-credo]

Merged into feat/test_expansion 166d214, we can close this.