goldenmatch v3.3.0
Added
-
Negative evidence on Fellegi-Sunter (
type: probabilistic) matchkeys
(Formulation B, EM-learned):negative_evidencewas previously silently
ignored on probabilistic matchkeys (weighted/exact only), which meant every
Splink-converted config — exactly one FS matchkey — had no defense against
the fan-out/homonym snowball (two distinct people sharing name+city merging
because name evidence dominates). Each NE field now joinstrain_emas a
constrained EM-learned dimension contributinglog2(m_fired/u_fired)when
it FIRES (both values present +scorer(a, b) < threshold, strict<) and
exactly 0 otherwise — the fired-else-zero clamp is what makes it negative
evidence rather than a regular scored field.NegativeEvidenceFieldgets a
newpenalty_bits(log2 LLR fixed override, probabilistic-only,abs()
applied) alongside the existingpenalty(still required on weighted/exact,
now rejected on probabilistic — setpenalty_bitsinstead). Guards: native,
fused, and the fast-path scorer all decline NE-bearing FS matchkeys (pure
Python fallback; a future kernel port addsFS_SUPPORTS_NE); the bucket
backend's slim-projection keep-list was extended so an NE-only field (e.g.
phone, never a regular matchkey field) survives the default
GOLDENMATCH_BUCKET_SLIM_PROJECTION.EMResult.validate_fornow requires
match_weights["__ne__<field>"]for every NE field withoutpenalty_bits,
so a model trained (or a Splink model imported) before this feature fails
loudly instead of silently scoring NE at weight 0. An unregistered/unknown
NE scorer on FS fails loud at train/score time (score_fieldraises on
unknown scorers; no_NE_BROKENswallow on FS) — unlike weighted's
swallow-and-warn fallback, this is intentional. Continuous/Winkler-path
(train_em_continuous) NE is out of scope and rejected with a clear error.
Supersedes the deferral indocs/superpowers/specs/2026-05-21-ne-fs-investigation.md
(Wave D): that investigation judged the Bayesian-factor formulation correct
but deferred it believingP(disagree_NE | match)needed labeled pairs —
stale, since EM already estimates match-conditional probabilities for every
regular FS field without labels, and the same machinery estimates them for
NE dimensions. -
Splink migration upgrade pass (
goldenmatch import-splink SETTINGS.json --upgrade DATA.parquet --model-out MODEL.json, or
gm.upgrade_splink_conversion(conversion, data)from Python): a data-aware
pass over a converted Splink config that applies three independent levers —
term-frequency tables computed from the data (Splink model exports don't
carry them, so converted tf fields were inert), distance thresholds
re-derived from measured string lengths (the converter assumes length 10),
and link/review thresholds calibrated from the blocked-pair score
distribution. Writes the upgraded config/model to--output/--model-out
with the faithful baseline alongside as*.baseline.*, plus a
baseline-vs-upgraded delta table;--splink-clusters/--labelstake
reference cluster mappings (first column id, second cluster_id) for
agreement / truth F1 measurement. Measured on the wild-config dogfood bench
(defaults-vs-defaults, pairwise F1 vs truth): real_time_settings/fake_1000
0.482 → 0.633 (native Splink: 0.601), saved_model_from_demo/fake_1000
0.677 → 0.766 (Splink: 0.699), model_h50k/historical_50k
0.707 → 0.740 (Splink: 0.686) — the upgraded conversion beats native
Splink on all three pairs. -
Fan-out / negative-evidence upgrade lever (
fan_out, in the default
import-splink --upgradelever set, betweendistance_thresholdsand
calibration): detects unused identity-grade columns (phone/email/id-named,
high-cardinality, non-matchkey, non-blocking) whose disagreement contradicts
pairs the imported model would confidently merge (posterior >= 0.9), and —
when the contradiction rate clears the risk gate (>= 2%, >= 10 firing
pairs) — adds the column asnegative_evidencewith posterior-weighted
EM-shape weights written into the upgraded model (__ne__<field>entries).
Also tunesgolden_rules.max_cluster_size = max(10, 2 * reference max cluster size)from--labels(preferred) or--splink-clusters, so
auto_splitcatches mid-size homonym snowballs the static default (100)
ignores on person-shaped data. The calibration lever is now NE-aware
(fs_weight_range+ per-pair NE contributions; its warn+skip tripwire is
removed). Wild-bench: no regressions;model_h50kimproved 0.7396 -> 0.7421
on guard tuning alone. -
Native negative-evidence scoring (
goldenmatch-native >= 0.1.15): the
Rust kernels now score FS negative evidence —score_block_pairs_fsAND the
fusedmatch_fused_fs, which also gained customlevel_thresholdsbanding
(full kernel parity). Detection is capability-gated (FS_SUPPORTS_NE,
FUSED_FS_SUPPORTS_LEVEL_THRESHOLDS), so older wheels keep the pure-Python
fallback with no behavior change; NE-bearing matchkeys previously always
took the pure-Python path. The fused path declinesderive_fromNE (its
raw-columns entry never materializes derived columns).
Fixed
- Threshold calibration on imported Splink models: the calibration lever
now re-estimates the within-block match rate from the model's likelihood
ratios instead of trustingproportion_matched, which on imported models
holds Splink'sprobability_two_random_records_match— a random-pair prior
orders of magnitude below the post-blocking rate the percentile math
expects. Trusting it cut at the extreme top of the score distribution and
collapsed recall (F1 0.482 → 0.157 on the bench pair above). - Tuned
max_cluster_sizesurvives the YAML loader round-trip: the
config loader'sgolden_rulesnormalization sweptmax_cluster_sizeinto
field_rules(the "set programmatically, never via YAML" assumption went
stale when the fan-out lever started writing it), soimport-splink --upgradeoutput failed to reload. It is now a recognized top-level
golden_rules key.