0.4.1

Install: pip install concept-benchmark==0.4.1. The paper's results and paper-results.zip on v0.4.0 are unchanged.

Fixed

• Sudoku board images no longer reveal the label through their layout. The cells drawn in the printed font were chosen with a seed equal to the weighted digit sum of the board, which is the same number for every valid board (each row and column sums to 45), so all valid boards shared one printed/handwritten layout and invalid boards did not; a logistic regression on the layout alone told them apart with 97% accuracy without reading a digit (reported by a user who trained an image model on the Hub dataset). The seed is now a hash of the board, so every board gets its own layout and the same board still gets the same one. Present in the images of every release before 0.4.1. No result in the paper reads pixels: every model is trained on the true digits and evaluated on the digits the recognizer reads cell by cell, so the layout could not reach them. The paper's boards are those of v0.4.0 (git checkout v0.4.0); RENDER_VERSION is 3, image dataset folders carry it (…_seed171_r3), and the Hub dataset juliannski/sudoku was regenerated (the 0.4.0 images remain at revision e0a31cfb).

0.4.0

Install: pip install concept-benchmark==0.4.0

paper-results.zip (attached) holds every file the paper's table and figure scripts read; unpack it and pass the folder as --results-root (see scripts/paper/README.md).

Changed

• The default robot label rule is now balanced. No single concept decides the label and both classes are equally likely. The previous rule is label_rule="sparse" (CLI: --label-rule sparse). Datasets generated with default settings differ from 0.3.x; a config saved by 0.3.x needs label_rule: sparse.
• plot_alignment_comparison takes a table with one row per run (concepts, model, accuracy_before, optional accuracy_after) instead of a dict of gains.
• plot_intervention_curve averages several runs, draws a standard-error band and takes group= to draw one line per model or concept set.
• ECBM follows the authors' code; the settings ecbm_emb_size, ecbm_inference_steps and ecbm_inference_lr are gone and must be deleted from a config saved by 0.3.x.
• set_deterministic_seed also turns on PyTorch's deterministic algorithms (with warnings instead of errors).

Added

• DatasetGenerator.generate_splits(): the train/validation/test split of the paper.
• ROBOT_LABEL_RULES, label_rule: the two label rules and the training foot subtypes that go with each.
• Plots: plot_intervention_heatmap, plot_automation, plot_concept_report, plot_answer_reliance, plot_confidence.
• Metrics: abstention_threshold, decision_threshold, selective_at (the paper's selective-classification protocol), classwise_thresholds, intervention_metrics.
• experiments.evaluate: train_cbm, train_dnn, intervention_table, automation_table, coverage_at_target, predict_labels, predict_proba_positive; the pipelines and the examples are built from these blocks.
• plot_intervention_curve and plot_automation draw the paper's two panels.
• Robot pipeline: --label-rule, --intervention-encoding, --intervention-sources self, --dump-interventions, the noisy_human_concepts concept source, and all four architectures on every concept source.
• Sudoku pipeline: ECBM, the optional diagnose stage, and abstention thresholds fitted with the rule that applies them.
• experiments/real_data.py: interventions on Derm7pt and CUB.
• scripts/paper/: the scripts behind every table, figure and test of the paper. They read one results folder (--results-root): the paper's results, a separate download on the releases page, or your own runs copied with collect_pipeline_runs.py.
• LICENSE, concept_benchmark.__version__.

Fixed

• Sudoku automation follows conceptual safeguards as published: the label probability is Platt-scaled on validation (PlattScaling, calibrate in the sudoku config, calibrate= in automation_table and coverage_at_target), the model predicts a valid board above 0.5 and abstains on [t, 1 - t] with t fitted once on validation. The tuned decision cut that the pipeline refit at every budget is gone. Uncalibrated, the propagated probability of a valid board is about 0.3 (27 concepts at 0.96 each), and the safeguard deferred every valid board at 18 px.
• Every sudoku board is written in its own handwriting (the board index seeds the glyphs; before, a digit in a cell was drawn the same way on every board, 729 glyphs in all), and the digit recognizer is trained on the training boards only, so the validation boards (where the thresholds are fitted) and the test boards carry handwriting it has never seen. Detection errors are now real: about a fifth of the 50 px boards carry a misread digit.
• Sudoku boards at any cell_px are the 50 px board resampled: printed starters, candidate bubbles and lines keep their proportions (at 18 px the starters used to overflow their cells and the bubbles covered the digits). The drawing at 50 px is unchanged; the handwriting (next entry) changes every board.
• The impaired-detection condition of the sudoku benchmark is 10 px per cell (the paper's second resolution; --cell-px 10). Rendered correctly, 18 px does not impair detection: the recognizer reads 99.5% of the cells at 50, 18 and 12 px alike, and only at 10 px does it fall (99.3% per cell, a misread digit on half the boards instead of a fifth).
• opencv-python-headless is a declared dependency: the sudoku image generator and the digit recognizer import cv2, which uv sync did not install.
• The sudoku digit recognizer's best epoch is saved as a copy. It used to be saved as state_dict() tensors moved with .cpu(), which on a CPU- or MPS-trained model are the live weights, so the last epoch was saved under the best epoch's accuracy; when the last epoch diverged (one run in a fresh clone on macOS went from 99.95% to 80% cell accuracy at epoch 5), the recognizer misread a fifth of the digits and the CBM evaluated on them predicted at chance. CUDA runs, including the paper's, copy on .cpu() and were not affected.
• The robot text pipeline runs again: it read a config field removed with the render-space revert (group_split_by_semantic_id; the split groups by robot row as in the paper), and its concept detector has a predict_proba for the model API that ConceptBasedModel now calls.
• The sudoku digit recognizer trains until its validation cell accuracy stops improving (up to ocr_epochs = 5, stopping after ocr_patience = 3 epochs without improvement, in the sudoku config) and keeps the best epoch. It used to train for two epochs with no check: on one seed in ten it stopped at 97.6% cell accuracy (the others 99.6–99.9%), which put a misread digit on 84% of that seed's boards and halved the coverage of every model on it.
• Sudoku: corruptions that cancelled out left about 1% of the "invalid" boards valid; the generator now redraws until a concept is violated. Datasets regenerated with 0.4.0 differ from 0.3.x in those boards.
• Abstention follows one rule, min(p, 1 - p) >= t (abstention_mask), in the threshold fit, selective_at and the intervention strategies (coverage and selective_accuracy take a confidence and keep confidence >= threshold; selective_kept keeps min(p, 1 - p) <= threshold); the two inequalities used before could disagree by one floating-point ulp.
• decision_threshold returns a threshold inside a run of tied best thresholds and the accuracy at that threshold.
• A concept probability of exactly 0.5 counts as present everywhere, and a label logit of exactly 0 predicts the negative class, as LogisticRegression.predict does.

Removed

• concept_benchmark.metrics.compute_selective_metric and experiments.metrics (use selective_at, which applies the same abstention band).

• experiments.utils keeps only run_alignment: determine_device, get_loader_config, compute_accuracy and patch_macos_dataloader live in concept_benchmark.utils, and experiments.utils.train_dnn is replaced by experiments.evaluate.train_dnn, which returns the trained model instead of a test accuracy.

• plot_regime_comparison (use plot_intervention_heatmap).