DOC fix warnings in documentation build

[adrinjalali]

There are warnings during doc build which need to be taken care of:

Details

Output from /home/adrin/Projects/fairlearn/fairlearn/examples/plot_new_metrics.py                                                                                                       
/home/adrin/Projects/fairlearn/fairlearn/examples/plot_new_metrics.py:91: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  X_raw['race'] = X_raw['race'].map(race_transform).fillna('Other').astype('category')
['Black' 'Other' 'White']
Output from /home/adrin/Projects/fairlearn/fairlearn/examples/plot_new_metrics.py
/home/adrin/Projects/fairlearn/fairlearn/examples/plot_new_metrics.py:125: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  A['Credit Score'] = col_credit
/home/adrin/Projects/fairlearn/fairlearn/examples/plot_new_metrics.py:126: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  A['Loan Size'] = col_loan_size
Output from /home/adrin/Projects/fairlearn/fairlearn/examples/plot_new_metrics.py
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
Output from /home/adrin/Projects/fairlearn/fairlearn/examples/plot_new_metrics.py
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
Found 36 subgroups. Evaluation may be slow
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/miniforge3/envs/fairlearn/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1495: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:: WARNING: py:class reference target not found: default=True
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:: WARNING: py:class reference target not found: default: None
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:: WARNING: py:class reference target not found: default=False
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:: WARNING: py:class reference target not found: default=False.
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_adult:33: WARNING: py:class reference target not found: sklearn.utils.Bunch
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:: WARNING: py:class reference target not found: default=True
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:: WARNING: py:class reference target not found: default: None
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:: WARNING: py:class reference target not found: default=False
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:: WARNING: py:class reference target not found: default=False.
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_bank_marketing:39: WARNING: py:class reference target not found: sklearn.utils.Bunch
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: default=True
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: default: None
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: default=False
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: default=False.
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: boolean
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:: WARNING: py:class reference target not found: default=True.
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/datasets/__init__.py:docstring of fairlearn.datasets.fetch_boston:64: WARNING: py:class reference target not found: sklearn.utils.Bunch
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame:: WARNING: py:class reference target not found: callable
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame:: WARNING: py:class reference target not found: dict of 1d arrays
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame:: WARNING: py:class reference target not found: dict of 1d arrays
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.difference:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.difference:: WARNING: py:class reference target not found: pandas.core.frame.DataFrame
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.group_max:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.group_max:: WARNING: py:class reference target not found: pandas.core.frame.DataFrame
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.group_min:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.group_min:: WARNING: py:class reference target not found: pandas.core.frame.DataFrame
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.group_min:: WARNING: py:class reference target not found: typing.Any pandas.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.ratio:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/_metric_frame.py:docstring of fairlearn.metrics.MetricFrame.ratio:: WARNING: py:class reference target not found: pandas.core.frame.DataFrame
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.count:: WARNING: py:class reference target not found: array_like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.count:: WARNING: py:class reference target not found: array_like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.demographic_parity_difference:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.demographic_parity_difference:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.demographic_parity_difference:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.demographic_parity_ratio:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.demographic_parity_ratio:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.demographic_parity_ratio:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.equalized_odds_difference:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.equalized_odds_difference:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.equalized_odds_difference:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.equalized_odds_ratio:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.equalized_odds_ratio:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.equalized_odds_ratio:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_negative_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_negative_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_negative_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_negative_rate:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_negative_rate:: WARNING: py:class reference target not found: scalar
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_negative_rate:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_positive_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_positive_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_positive_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_positive_rate:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_positive_rate:: WARNING: py:class reference target not found: scalar
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.false_positive_rate:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.make_derived_metric:: WARNING: py:class reference target not found: callable
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.make_derived_metric:: WARNING: py:class reference target not found: callable
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.mean_prediction:: WARNING: py:class reference target not found: array_like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.mean_prediction:: WARNING: py:class reference target not found: array_like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.mean_prediction:: WARNING: py:class reference target not found: array_like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.selection_rate:: WARNING: py:class reference target not found: array_like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.selection_rate:: WARNING: py:class reference target not found: array_like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.selection_rate:: WARNING: py:class reference target not found: Scalar
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.selection_rate:: WARNING: py:class reference target not found: array_like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_negative_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_negative_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_negative_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_negative_rate:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_negative_rate:: WARNING: py:class reference target not found: scalar
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_negative_rate:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_positive_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_positive_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_positive_rate:: WARNING: py:class reference target not found: array-like
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_positive_rate:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_positive_rate:: WARNING: py:class reference target not found: scalar
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/metrics/__init__.py:docstring of fairlearn.metrics.true_positive_rate:: WARNING: py:class reference target not found: optional
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:1: WARNING: py:class reference target not found: sklearn.base.MetaEstimatorMixin
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: estimator object implementing 'predict' and possibly 'fit'
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: default='demographic_parity'
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: default='accuracy_score'
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: default=1000
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: default=False
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: default=False
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: {'auto'
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: 'predict_proba'
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: 'decision_function'
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: 'predict'            }
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer:: WARNING: py:class reference target not found: default='auto'
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer.predict:: WARNING: py:class reference target not found: RandomState instance
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer.predict:: WARNING: py:class reference target not found: default=None
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer.predict:: WARNING: py:class reference target not found: Scalar
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/postprocessing/_threshold_optimizer.py:docstring of fairlearn.postprocessing.ThresholdOptimizer.predict:: WARNING: py:class reference target not found: vector as numpy.ndarray
writing output... [ 16%] api_reference/fairlearn.preprocessing .. api_reference/index
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_moments/bounded_group_loss.py:docstring of fairlearn.reductions.BoundedGroupLoss:1: WARNING: py:class reference target not found: fairlearn.reductions.ConditionalLossMoment
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_moments/utility_parity.py:docstring of fairlearn.reductions.DemographicParity:8: WARNING: py:meth reference target not found: signed_weights
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_moments/utility_parity.py:docstring of fairlearn.reductions.EqualizedOdds:20: WARNING: py:meth reference target not found: signed_weights
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_moments/utility_parity.py:docstring of fairlearn.reductions.ErrorRateParity:16: WARNING: py:meth reference target not found: signed_weights
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_exponentiated_gradient/exponentiated_gradient.py:docstring of fairlearn.reductions.ExponentiatedGradient:1: WARNING: py:class reference target not found: sklearn.base.MetaEstimatorMixin
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_exponentiated_gradient/exponentiated_gradient.py:docstring of fairlearn.reductions.ExponentiatedGradient:: WARNING: py:class reference target not found: estimator
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_exponentiated_gradient/exponentiated_gradient.py:docstring of fairlearn.reductions.ExponentiatedGradient.predict:: WARNING: py:class reference target not found: RandomState instance
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_exponentiated_gradient/exponentiated_gradient.py:docstring of fairlearn.reductions.ExponentiatedGradient.predict:: WARNING: py:class reference target not found: default=None
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_exponentiated_gradient/exponentiated_gradient.py:docstring of fairlearn.reductions.ExponentiatedGradient.predict:: WARNING: py:class reference target not found: Scalar
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_exponentiated_gradient/exponentiated_gradient.py:docstring of fairlearn.reductions.ExponentiatedGradient.predict:: WARNING: py:class reference target not found: vector
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_moments/utility_parity.py:docstring of fairlearn.reductions.FalsePositiveRateParity:20: WARNING: py:meth reference target not found: signed_weights
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_grid_search/grid_search.py:docstring of fairlearn.reductions.GridSearch:1: WARNING: py:class reference target not found: sklearn.base.MetaEstimatorMixin
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_grid_search/grid_search.py:docstring of fairlearn.reductions.GridSearch:: WARNING: py:class reference target not found: estimator
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_grid_search/grid_search.py:docstring of fairlearn.reductions.GridSearch.fit:: WARNING: py:class reference target not found: for now
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/__init__.py:docstring of fairlearn.reductions.Moment.load_data:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/__init__.py:docstring of fairlearn.reductions.Moment.load_data:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/__init__.py:docstring of fairlearn.reductions.Moment.load_data:: WARNING: py:class reference target not found: array
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/_moments/utility_parity.py:docstring of fairlearn.reductions.TruePositiveRateParity:26: WARNING: py:meth reference target not found: signed_weights
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/__init__.py:docstring of fairlearn.reductions.UtilityParity.load_data:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/__init__.py:docstring of fairlearn.reductions.UtilityParity.load_data:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/../fairlearn/reductions/__init__.py:docstring of fairlearn.reductions.UtilityParity.load_data:: WARNING: py:class reference target not found: pandas.core.series.Series
/home/adrin/Projects/fairlearn/fairlearn/docs/faq.rst:128: WARNING: py:class reference target not found: MetricFrame
/home/adrin/Projects/fairlearn/fairlearn/docs/faq.rst:138: WARNING: py:class reference target not found: ExponentiatedGradient
/home/adrin/Projects/fairlearn/fairlearn/docs/faq.rst:138: WARNING: py:class reference target not found: GridSearch
/home/adrin/Projects/fairlearn/fairlearn/docs/faq.rst:138: WARNING: undefined label: _bounded_group_loss (if the link has no caption the label must precede a section header)
/home/adrin/Projects/fairlearn/fairlearn/docs/user_guide/migrating_versions/migrating_to_v0_5_0.rst:12: WARNING: py:class reference target not found: sklearn.utils.Bunch
/home/adrin/Projects/fairlearn/fairlearn/docs/user_guide/migrating_versions/migrating_to_v0_5_0.rst:21: WARNING: py:class reference target not found: MetricFrame

[romanlutz]

I want to add that any contribution doesn't need to tackle all of these. Even just one is a step forward!

[alliesaizan]

Hi there, I joined the Discord recently and I'd like to address these warnings. @romanlutz is anyone currently working on this?

[romanlutz]

Hi @alliesaizan ! Thanks for taking a look at this! Nobody else is currently investigating these.

You can try this out by following the steps in this guide: https://fairlearn.org/main/contributor_guide/contributing_documentation.html
The "simple" doc build using python -m sphinx -v -b html -n -j auto docs docs/_build/html or make doc should be sufficient, since the "full" doc build doesn't provide the same output verbosity.

The majority of remaining errors seem to be of the following kind:

\fairlearn\datasets\__init__.py:docstring of fairlearn.datasets._fetch_adult.fetch_adult:: WARNING: py:class reference target not found: default=True
\fairlearn\datasets\__init__.py:docstring of fairlearn.datasets._fetch_adult.fetch_adult:: WARNING: py:class reference target not found: default=None
\fairlearn\datasets\__init__.py:docstring of fairlearn.datasets._fetch_adult.fetch_adult:: WARNING: py:class reference target not found: default=False
\fairlearn\datasets\__init__.py:docstring of fairlearn.datasets._fetch_adult.fetch_adult:: WARNING: py:class reference target not found: default=False

meaning our docstrings have types that aren't recognized. This isn't entirely surprising since we sometimes use free text to describe the types rather than actual type names. The corresponding lines would be:

• fairlearn/fairlearn/datasets/_fetch_adult.py

  Line 30 in 3211828

  cache : bool, default=True

• fairlearn/fairlearn/datasets/_fetch_adult.py

  Line 33 in 3211828

  data_home : str, default=None

• fairlearn/fairlearn/datasets/_fetch_adult.py

  Line 38 in 3211828

  as_frame : bool, default=False

• fairlearn/fairlearn/datasets/_fetch_adult.py

  Line 46 in 3211828

  return_X_y : bool, default=False

... and a few dozen others. I don't think these are really worth fixing since there's no solution that I'm aware of ( @fairlearn/fairlearn-maintainers please correct me if I'm wrong!). I haven't checked every single one, though, so there may very well be a few that are fixable.

Other errors that should show when you run the doc build command include:

• Output from C:\Users\rolutz\git\fairlearn\examples\plot_quickstart.py y
  C:\Users\rolutz\Anaconda3\lib\site-packages\pandas\plotting_matplotlib\core.py:1616: MatplotlibDeprecationWarning: normalize=None does not normalize if the sum is less than 1 but this behavior is deprecated since 3.3 until two minor releases later. After the deprecation period the default value will be normalize=True. To prevent normalization pass normalize=False

  I'm not entirely sure what this affects, but you can try passing it and see if it changes anything compared to the existing quickstart. Ideally the plots should remain the same (unless you can improve them in some way, of course).

• fairlearn\postprocessing_interpolated_thresholder.py:115: UserWarning: The value of prefit is True, but check_is_fitted raised NotFittedError on the base estimator.

  This is actually a warning we're creating from Fairlearn and not yet resolvable (waiting on a fix in scikit-learn).

• Anaconda3\lib\site-packages\sklearn\metrics_classification.py:1496: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use zero_division parameter to control this behavior.
  average, "true nor predicted", 'F-score is', len(true_sum)

  shows up a lot in this example. Would be good to figure out how we can fix that. It's just a contrived example to show the functionality of Fairlearn's metrics capabilities, so we can certainly change the data if it really has "no true nor predicted samples" in some cases, or change the zero_division param. Again, haven't looked into it so there may be multiple good options. Suggestions are welcome! ( @riedgar-ms FYI since you wrote that example originally)

@alliesaizan if you have issues, questions or would like to look into something different just reach out on Discord and we discuss! Again, welcome to the community.

[alliesaizan]

@romanlutz I started with the last warning:

Anaconda3\lib\site-packages\sklearn\metrics_classification.py:1496: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use zero_division parameter to control this behavior.
average, "true nor predicted", 'F-score is', len(true_sum)

And the culprit appears to be line 234 in the examples >> plot_new_metrics.py file, fbeta_06 = functools.partial(skm.fbeta_score, beta=0.6). Adding a zero_division=1 parameter and re-running lines 234 & 236 appears to resolve the warning without changing the results as they appear on the blog post. Please let me know how you'd like to proceed!

[riedgar-ms]

@alliesaizan that's fine by me. The particular function isn't important; it's about showing how to use metric functions in MetricFrame.

[romanlutz]

@alliesaizan that's awesome! You can create a PR with this fix. The smaller the better and the faster it typically gets merged.

[alliesaizan]

@romanlutz I'm looking at this error now:

Output from C:\Users\rolutz\git\fairlearn\examples\plot_quickstart.py y
C:\Users\rolutz\Anaconda3\lib\site-packages\pandas\plotting_matplotlib\core.py:1616: MatplotlibDeprecationWarning: normalize=None does not normalize if the sum is less than 1 but this behavior is deprecated since 3.3 until two minor releases later. After the deprecation period the default value will be normalize=True. To prevent normalization pass normalize=False

It appears to be caused by the following code chunk:

metric_frame.by_group.plot(
kind="pie",
subplots=True,
layout=[3, 3],
legend=False,
figsize=[12, 8],
title="Show all metrics in pie",
)

The pie chart in plot_quickstart.py looks like this:

For a MetricFrame with the following values:

I get that the idea is to show users how to show how the metrics compare across subgroups, but I am not sure the pie chart achieves this goal. The values across Male and Female don't add up to 1 in most cases, so I don't think this is the correct use of a pie chart. Here's what the pie chart looks like when I pass a normalize=True parameter to the plot function:

In the charts above, each gender's metric is scaled so that the sum across equals 1 (if the metrics are less than zero), so it's showing the share of the metric distributed across genders. Personally, I'd recommend removing the pie charts because I don't think either explanation is accurate to what the quickstart is trying to show, but adding that parameter gets rid of the warning. I am not sure if there's a better fix. Please let me know what you think!

[romanlutz]

Thanks for the thorough assessment @alliesaizan !
The filename is a bit of a misnomer since it was originally built for the quickstart, but it's used in other places, too. If you check the webpage's quickstart, you'll find that the pie charts don't show up for the same reasons you describe. To my knowledge, they're only used in the user guide: https://fairlearn.org/main/user_guide/assessment.html#customize-plots-kind
The point there is to show that one can customize the kind parameter. I agree that these metrics don't lend themselves to this representation, perhaps with the exception of count. Perhaps it makes sense to show the pie chart only for count?