Pricing Incompleteness in Equivalent Clean Rows

What does a missing value cost? Answer it in rows, and you can compare it with collecting more.

---

The problem

Studies comparing ways to fill in missing values report the difference in units nobody can act on - reconstruction error, or a change in AUC of a few thousandths. Neither tells a data owner whether the problem is worth fixing, or how it compares against collecting more data.

So measure the cost in rows instead. If a model trained on damaged data scores the same as one trained on m complete rows, the damage cost you the difference. Now it is comparable with the price of acquiring more.

To do that, fit a learning curve of error against training size and invert it:

err(n) = a·n^(-b) + c          fit
n(err) = (a/(err - c))^(1/b)   price

The experiment only works on data that starts complete. You cannot measure the cost of a gap you did not create, because you would not know what was in it. So the study uses the one fully complete dataset available and punches the holes itself - ground truth by construction.

Results

Seven strategies x three mechanisms x three rates x two training sizes = 126 conditions. 118 priced, 5 refused by the instrument, 3 left too little data to fit at all.

Deleting incomplete rows is a failure mode, not a safe default

Strategy	MCAR	MAR	MNAR
complete-case deletion	0.727	0.640	0.586
median	0.098	0.107	0.138
mean	0.058	0.077	0.114
k-NN	0.046	0.076	0.086
iterative (MICE)	0.057	0.142	0.103
native (histogram GBM)	0.091	0.125	0.143
missingness indicator	0.058	0.114	0.141

Fraction of the training set lost, pooled over rate and size.

Deletion costs 59-73% of the training set - four to sixteen times any imputation strategy. At the highest missing rate it left fewer than 50 complete rows out of 22,500 in three conditions: not enough to fit a model. Those are reported as unusable rather than dropped, since excluding the worst outcome would flatter the method.

But the ranking between imputers is not a result

This is the part worth reading before quoting any number above.

The estimator was tested on questions whose answers were already known - and got them wrong by up to 57%

Between two clean training sizes, the true gap is just their difference. No estimation needed. So ask the instrument to recover it:

true gap	estimated	error
5,625	2,413	-57%
11,250	5,639	-50%
14,626	10,874	-26%
5,625	3,225	-43%
9,001	8,460	-6%
3,376	5,235	+55%

Every imputation cost in the absolute table lies between 382 and 2,616 rows - inside the error the instrument makes where the answer is known.

The intervals are the honest part: they cover the known answer in 54 of 55 clean anchor pairs. It is the point estimates that the rankings are built on.

The signal is below the noise floor of the metric it comes from

At MCAR 10% and n = 22,500, the six imputation strategies span 0.00157 AUC on the fixed 7,500-row test set. The Hanley-McNeil standard error of AUC at that value, with 1,659 positives, is 0.00702 - 4.5x larger than the spread being ranked.

Three of the six degraded models score above the clean model, which is why some prices come out negative. The bootstrap cannot propagate test-set variance: there is one test set in the study.

86 of the 108 non-deletion conditions have a 95% interval containing zero, and the median interval is 53% of the training size. No imputation strategy here has been shown to cost anything at all.

The zero point had to be measured, not assumed

A cost is n minus the damaged price only if the instrument values undamaged data at exactly n. It does not:

clean training set	instrument says	error in n - price
9,000 rows	9,313	-313
22,500 rows	18,418	+4,082

The gap between those two errors - 4,395 rows - is pure spurious size dependence, and an earlier draft reported it as a finding.

curve.filter_anchors keeps only anchors carrying the modal seed count, and so discards five, including both sizes priced at. Two of them, n=4,500 and n=22,500, are discarded for carrying nine seeds against a modal eight - better measured than the anchors retained, and rejected for it. Restoring those two gives b = 0.130 rather than 0.096 and prices the clean 22,500-row set at 20,114. Both figures are reported rather than choosing after seeing them.

The fix is a paired estimator: price(clean at n) - price(damaged), differenced inside each bootstrap replicate so the fit uncertainty common to both cancels.

What survives all of it

The deletion result. It is ten times larger than the measurement noise, and no resampling makes a 59-73% loss comparable to a 5-14% one.

The three mechanisms

The reason a value is missing changes what filling it in can achieve.

	What decides the gap	Visible in the data?
MCAR	nothing - pure chance	n/a
MAR	another column you still hold	yes
MNAR	the missing value itself	no

MNAR is the dangerous one: it adds bias, not noise, and more data does not remove bias. It is also not identifiable from the data - proving the missing values differ would require knowing them. That is a limit of logic, not of methods, and it is why the practical recommendation here is about robustness across mechanisms rather than optimality within one.

The controlled comparison

The central methodological claim, and the one most easily got wrong.

For seven strategies to be comparable, each must see the same holes - not the same number of holes, the identical cells. The mask is drawn once from (mechanism, rate, seed) and handed to all seven. Verified: the NaN masks are bit-identical across all seven strategies for every one of the nine mechanism-rate pairs.

An earlier version drew a separate mask per strategy. A strategy that looked cheaper might simply have been handed less informative columns to destroy, so the comparison confounded strategy with damage. Fixing it changed the answer.

One axis is controlled; two are not. Each mechanism and each rate damages a different nine of the 23 columns - the three sharing a median of one - so comparisons across mechanism or rate confound the variable with the columns that happened to be hit. Stated in the report rather than left implicit.

Setup

pip install -r requirements.txt

Python 3.13, CPU only.

Reproducing everything

python data.py            # download + characterise the dataset
python runner2.py         # smoke test: one condition
python grid2.py           # the full ablation grid  (long, resumable)
python price2.py          # prices, tables and both figures
python export_results.py  # parquet cache -> results_all.csv

Every experiment is cached as one Parquet file keyed by a hash of its configuration, so grid2.py skips completed work and is safe to interrupt.

Layout

File	Purpose
ablate.py	the damage - MCAR/MAR/MNAR injection and the seven imputers
price2.py	the pricing - every condition, both tables, both figures
curve.py	the instrument: fit, bootstrap, invert, refuse
data.py	dataset loading and characterisation
runner.py / runner2.py	trains one model, writes one result row
grid2.py	sweeps the ablation grid
export_results.py	consolidates the Parquet cache into results_all.csv
paper/	the report and its figures
results/results_all.csv	every experiment, committed
priced_conditions.csv	every condition with its price, interval and status

This repository is self-contained - data.py, runner.py and curve.py are part of it, not imported from elsewhere.

Reproducing from a fresh clone. results/parquet/ is gitignored, so a clone has only results_all.csv and load_results() falls back to it. Every point estimate in the report reproduces exactly from that CSV. Interval endpoints do not: the bootstrap resamples each anchor's observations in the order it receives them, and the CSV is sorted while the Parquet cache is read in filesystem order, so a fixed RNG_SEED still yields different draws. Endpoints move by about 1%. Reproduce intervals from the Parquet cache, or accept the difference. It is a one-line fix, deliberately not applied, because it would have moved published numbers at the point where they most needed to stop moving.

Dataset

Default of Credit Card Clients - 30,000 rows, 23 numeric features, from the UCI ML Repository, CC BY 4.0, fetched via ucimlrepo.

Chosen for one reason: it is the only genuinely complete dataset among those surveyed. The method requires starting from data with no gaps, so the gaps can be created and their cost measured against a known baseline.

Design decisions

• The test set is split once with a constant seed and never resampled. Subsampling touches the training pool only.
• Every imputer is fitted inside the pipeline, on the training subsample alone. It never sees the test set.
• Missing values are injected into training features only. The test set stays complete, which is what the instrument requires - and means this prices the learning cost of incompleteness, not the inference-time cost.
• Mechanisms are compared at equal total missingness. Probabilities are rescaled so the realised marginal rate matches the nominal one, so mechanism is not confounded with amount.

Known limitations

• One dataset, one learner, two training sizes, two seeds per condition.
• The between-strategy signal is below what the instrument can resolve. The ranking is reported as measured, not established.
• Mechanisms are simulated. Real missingness need not resemble the three idealised patterns.
• The mechanism is treated as known throughout, which is exactly what a practitioner does not have.
• A pre-registered prediction - that a missingness indicator would be cheapest under MNAR - cannot be tested by this design at all. The test set is never degraded, so at inference every indicator column is identically zero. Found afterwards, and recorded.
• For probabilistic PCA, Ipsen and Hansen show analytically that missing data reduces the signal-to-noise ratio rather than the sample size. Their setting is unsupervised, linear and Gaussian, so it does not transfer directly - but the equivalent-row reduction should not be assumed correct in general.

Licence

The dataset is CC BY 4.0 from UCI and is cited in the report.