Multidimensional subset scanning (MDSS) for bias in classifiers #238
Conversation
…ias in classifiers. Includes Bernoulli, Poisson and Berk Jones scoring functions, and example notebooks. Signed-off-by: Victor Akinwande <victor.akinwande1@ibm.com>
…ias in classifiers. Includes Bernoulli, Poisson and Berk Jones scoring functions, and example notebooks. Signed-off-by: Victor Akinwande <victor.akinwande1@ibm.com>
|
This pull request introduces 2 alerts when merging d4dd16c into 746e763 - view on LGTM.com new alerts:
|
Signed-off-by: Victor Akinwande <victor.akinwande1@ibm.com>
Signed-off-by: Victor Akinwande <victor.akinwande1@ibm.com>
Signed-off-by: Victor Akinwande <victor.akinwande1@ibm.com>
|
|
||
| def bisection_q_mle(score_function: ScoringFunction, observed_sum: float, probs: np.array, **kwargs): | ||
| """ | ||
| Computes the q which maximizes score (q_mle). |
There was a problem hiding this comment.
The alternative hypothesis which the MDSS optimization finds evidence for is that there is a multiplicative factor q in the odds for a subgroup of the data. For a subset of records, we can obtain the MLE of q (which is defined uniquely depending on the scoring function) and use this to score that subset.
| from sklearn.metrics import make_scorer as _make_scorer, recall_score | ||
| from sklearn.metrics import multilabel_confusion_matrix | ||
| from sklearn.neighbors import NearestNeighbors | ||
| from sklearn.utils import check_X_y | ||
| from sklearn.exceptions import UndefinedMetricWarning | ||
|
|
||
| from aif360.sklearn.utils import check_groups | ||
|
|
||
| from aif360.metrics.mdss.ScoringFunctions import Bernoulli |
There was a problem hiding this comment.
Why only Bernoulii is imported? What about Berk Jones and Poisson? Are they not incorporated yet?
There was a problem hiding this comment.
The original paper on Bias Scan of classifiers used Bernoulli. For a straightforward application of MDSS as a classification metric Bernoulli makes the most sense. Advanced users will be able to use other scoring functions as they please.
Signed-off-by: Victor Akinwande <victor.akinwande1@ibm.com>
Signed-off-by: Victor Akinwande <victor.akinwande1@ibm.com>
Signed-off-by: Victor Akinwande <victor.akinwande1@ibm.com>
Multidimensional subset scanning (MDSS) for bias in classifiers
This PR includes implementation of bias scan: Identifying Significant Predictive Bias in Classifiers" https://arxiv.org/abs/1611.08292 as a classification metric and example notebooks.
The underlying optimization method MDSS can be used with both parametric (eg. Bernoulli and Poisson) and non-parametric (eg. Berk Jones) scoring functions. All three are implemented in this PR.