Skip to content

CRIT-AID v1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@kovariati kovariati released this 26 Aug 17:37
v1.0.0
dfea6c5

CRIT-AID v1.0.0

Canonical code, results, and reproducibility release for the peer-reviewed article:

Kovari, A. (2026). Stable Discrimination Can Hide Reliability Failures in AI Decision Support Under Distribution Shift and Changing Target Definitions. Computers, 15(9), 560. https://doi.org/10.3390/computers15090560

Publisher article: https://www.mdpi.com/2073-431X/15/9/560

CRIT-AID is an executable empirical reliability-audit framework for AI decision support under distribution shift. It tests whether probability mappings, selection/abstention rules, and conformal uncertainty outputs retain their intended statistical meaning when source-derived rules are transported to target domains, when target definitions change, or when decision-relevant evidence is degraded.

Key scientific findings

  • Stable discrimination does not imply stable decision-support reliability.
  • On identical ACS 2024 records, changing the income target definition left AUROC nearly unchanged while ECE differed by 0.083; prevalence/intercept alignment reduced the ECE contrast to approximately −0.004.
  • Across 27 primary 90% OOD/stress conformal conditions, label-conditional calibration improved worst-class coverage in 18 and worsened it in 9, while usually enlarging prediction sets.
  • A fixed cross-domain LightGBM sensitivity analysis changed absolute performance without eliminating the mismatch among discrimination, calibration, operating points, and uncertainty.

What this release contains

The tagged source repository contains the analysis implementation, exact dependency versions, deterministic cohort/split/provenance manifests, compact machine-readable results, result registries, figures, citation metadata, and source/results validation tools.

The attached CRIT_AID_large_results_v1.0.0.zip supplies the three large generated output files intentionally excluded from Git history:

  • acs_risk_selection_curves.csv.gz
  • oulad_risk_selection_curves.csv.gz
  • acs_primary_target_predictions.csv.gz

Raw third-party datasets are not redistributed. Exact public sources and expected archive names are documented in DATA_AVAILABILITY.md, THIRD_PARTY_DATA.md, and manifests/raw_file_map.json.

Reproduce or validate

Fast repository audit:

python scripts/audit_repository.py --mode source
python tools/validate_repository.py .

Complete shipped-results audit after extracting the large-results asset into outputs/:

python scripts/audit_repository.py --mode results

See REPRODUCTION_LEVELS.md and README_RUNNING.md for prepared-table and cold-regeneration workflows.

Citation

If the scientific method or findings contribute to your work, cite the peer-reviewed article:

Kovari, A. (2026). Stable Discrimination Can Hide Reliability Failures in AI Decision Support Under Distribution Shift and Changing Target Definitions. Computers, 15(9), 560. https://doi.org/10.3390/computers15090560

If you directly reuse or modify the software, also cite the CRIT-AID v1.0.0 software release and follow the repository license guidance.

Repository: https://github.com/kovariati/CRIT-AID
Release: https://github.com/kovariati/CRIT-AID/releases/tag/v1.0.0
Article DOI: https://doi.org/10.3390/computers15090560
Author ORCID: https://orcid.org/0000-0003-3521-4757

Scientific identity

This publication-day metadata/discovery refresh does not change the canonical numerical result set. It aligns the public research object with the published version of record in Computers and strengthens citation, discovery, reproducibility, and public-release metadata.

Assets 4
Loading