feat: overhaul error analysis (roadmap 1.2, 3.1-3.3) - #41
Merged
Conversation
Fix ErrorAnalyzer bugs and make it a first-class feature:
- rank_errors/merge_errors/analyze_target now return plain DataFrames,
dropping the redundant {'values', 'idx'} wrapper.
- require X/y explicitly instead of reaching into the estimator (which
never stored them); rank_errors computes out-of-fold predictions.
- regression errors use a configurable quantile threshold (error_quantile)
instead of the global +/-1 std band; consistent for multioutput.
- permutation importances are computed over the Poniard CV folds instead
of a random 80/20 split.
- analyze_target returns error_count/target_count/error_rate per
class/bin; categorical features get an error-rate table.
- add analyze() orchestrator returning a single structured report.
- fix latent multiclass label upcast and standalone feature-types bugs.
- add 33 tests covering rank/merge/target/feature analysis and the
report; document ErrorAnalyzer in the README.
Add deterministic standalone tests where errors are confined to a known class / category / numeric range, asserting analyze_target and analyze_features correctly surface them via error_rate and group means. Also assert permutation-importance feature selection is deterministic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Overhauls
ErrorAnalyzerto make it a first-class feature (roadmap items 1.2 — fix bugs in differentiated modules, 3.1 — test coverage, 3.2 — data passing, 3.3 — documentation).Changes
Bug fixes
analyze_featurescalled_train_test_split_from_cv()with noX, y(wouldNameError); now passes them explicitly.analyze_features: feature selection is now unified on positional indices; a positional int is never compared to a column name._compute_predictions/rank_errors/analyze_targetreached intoself._poniard.X/ywhich were never stored on the estimator; they now requireX/yexplicitly (ErrorAnalyzer stays stateless)._rank_errors_continuous_multioutputbroadcast bug (np.std(y, axis=0)returned a Series → pandas alignment failure).row['y']to float →KeyError: 'proba_2.0'.analyze_featuresiterated.feature_typesdict without.items().Redesign
rank_errors/merge_errorsreturn plain DataFrames (dropped the{"values", "idx"}wrapper).merge_errorsaccepts a dict of DataFrames or a single DataFrame.|residual| > (1 - error_quantile)quantile threshold (default worst 10%), replacing the global ±1 std band. Per-target thresholds for multioutput.analyze_targetreturnserror_count/target_count/error_rateper class/bin (always a DataFrame;as_ratio/wrt_targetremoved). Categorical features inanalyze_featuresget an error-rate-per-category table.analyze(X, y, ...)returns a single report withranked_errors,merged_errors,summary,by_target,by_feature.Tests & docs
tests/test_error_analysis.pywith 33 tests (all target types, error paths, report structure, quantile threshold).ErrorAnalyzersection added to README.Breaking changes
rank_errors/merge_errorsreturn DataFrames instead of{"values", "idx"}dicts.analyze_targetdropsas_ratio/wrt_target; now always returns a DataFrame witherror_rate.analyze_featuresrequiresX(andywhenestimator_nameis used).rank_errorsrequiresXandywhen built viafrom_poniard.Tests
pytest— 128 passed (95 existing + 33 new).