Verification

Auto-Config Verification

New in v1.5.0. Auto-config (auto_configure_df, zero-config dedupe_df / match_df) now runs a preflight pass before returning and a postflight pass after scoring. This turns silent auto-config failures into localized, loggable events and closes a class of runtime bugs where a generated config referenced columns the pipeline never created.

---

The bug this fixes

Before v1.5.0, dedupe_df(df) on bibliographic-style data (DBLP-ACM, Abt-Buy, any text-heavy source) would crash at runtime:

ValueError: Missing required columns: ['__title_key__']

Auto-config emitted matchkeys and blocking keys referencing __title_key__ (a domain-extracted feature) but didn't enable config.domain, so the pipeline never ran the domain-extraction step that produces the column. Preflight now auto-repairs this hand-off by setting config.domain = DomainConfig(enabled=True, mode=<detected>) whenever the config references a domain-extracted column.

---

Preflight — 6 checks before auto_configure_df returns

Check	Severity	Auto-repair
Every referenced column resolves	error	Sets config.domain for domain-extracted refs; otherwise raises ConfigValidationError
Exact matchkey cardinality ≥ 0.99	warning	Drops the matchkey (no pairs possible)
Exact matchkey cardinality ≤ 0.01	warning	Drops the matchkey (degenerate)
Block-size sanity (P99 ≤ 5000, median ≥ 2)	warning	No auto-repair — advisory only
Remote-asset demotion	warning	Drops record_embedding, demotes embedding → ensemble, clears rerank=True
Confidence-gated weight cap	warning	Caps weight at 0.5 for profile confidence < 0.5

Failure semantics: unrepairable errors raise ConfigValidationError with the PreflightReport attached. Warnings never raise — they attach to config._preflight_report.findings.

import goldenmatch as gm

cfg = gm.auto_configure_df(df)
for finding in cfg._preflight_report.findings:
    print(f"[{finding.severity}] {finding.check}: {finding.message}")
    if finding.repaired:
        print(f"  repaired: {finding.repair_note}")

---

Postflight — 4 signals after scoring, before clustering

Runs inside the pipeline. Attached to the result as result.postflight_report.

Signal	What it measures	Adjustment
Score histogram	100-bin distribution + bimodality detection	Auto-nudges threshold to the valley when clearly bimodal
Blocking recall	Gated ≥ 10K rows (currently deferred — returns "deferred" sentinel)	Advisory only
Preliminary cluster sizes	Union-Find percentiles + bottleneck pair for oversized clusters	Advisory only
Threshold-band overlap	% of pairs within threshold ± 0.02	Advisory recommending --llm-auto when overlap > 20%

Strict mode (auto_configure_df(..., strict=True)) suppresses adjustments but keeps signals + advisories — used for deterministic parity runs (DQBench, regression testing).

result = gm.dedupe_df(df)
sig = result.postflight_report.signals

print(f"Scored {sig['total_pairs_scored']} pairs")
print(f"Threshold overlap: {sig['threshold_overlap_pct']:.1%}")
print(f"Oversized clusters: {len(sig['oversized_clusters'])}")

for adj in result.postflight_report.adjustments:
    print(f"adjusted {adj.field}: {adj.from_value} → {adj.to_value} ({adj.reason})")

for adv in result.postflight_report.advisories:
    print(f"advisory: {adv}")

---

Signals schema (stable contract)

PostflightReport.signals is typed as PostflightSignals (TypedDict). Consumers can rely on these exact keys:

Key	Type
score_histogram	{"bins": list[float], "counts": list[int]}
blocking_recall	float | Literal["deferred"]
block_size_percentiles	{"p50": int, "p95": int, "p99": int, "max": int}
threshold_overlap_pct	float
total_pairs_scored	int
current_threshold	float
preliminary_cluster_sizes	{"p50": int, "p95": int, "p99": int, "max": int, "count": int}
oversized_clusters	list[{"cluster_id": int, "size": int, "bottleneck_pair": [int, int]}]

Future versions may add keys non-breakingly. Removals or re-typings require a new contract spec.

---

Offline-safe by default

Auto-config no longer silently downloads embedding models or cross-encoders. Opt in with allow_remote_assets=True:

# Default: preflight demotes embedding / record_embedding / rerank=True
cfg = gm.auto_configure_df(df)

# Explicitly opt in to remote assets
cfg = gm.auto_configure_df(df, allow_remote_assets=True)

This fixes the class of CI failures where offline runners would hit HuggingFace rate limit errors mid-pipeline.

---

Handling ConfigValidationError

from goldenmatch import ConfigValidationError

try:
    cfg = gm.auto_configure_df(df)
except ConfigValidationError as err:
    for f in err.report.findings:
        if f.severity == "error" and not f.repaired:
            print(f"cannot proceed: {f.check}[{f.subject}]: {f.message}")

dedupe_df / match_df zero-config paths let ConfigValidationError propagate unchanged — catch and inspect if you want a partial config.

---

Classifier improvements (v1.5.0)

Preflight is the safety net; these upstream fixes reduce how often the net has to catch something.

• Cardinality guard: columns with unique-value ratio ≥ 0.95 are classified as identifier, not phone/zip/numeric. Fixes NCVR voter_reg_num being misclassified as a phone.
• year col_type: columns named *_year / *_yr or holding 4-digit values in 1900–2100 become blocking candidates, not scoring fields.
• multi_name col_type: comma/semicolon-delimited multi-name fields (≥ 70% delim-containing rows, ≥ 2 delims/row) get routed to token_sort with weight 1.0.
• Low-confidence weight cap: fields with profile confidence < 0.5 cap at weight 0.3 in weighted matchkeys.

---

Examples

• examples/verification_inspection.py — end-to-end walkthrough of preflight findings and postflight signals.
• examples/strict_mode_parity.py — deterministic parity runs with strict=True.

---

Further reading

• Python API — full preflight / postflight signatures and dataclass shapes.
• Configuration — how verification integrates with manual YAML configs.
• Quick Start — the minimum-viable usage pattern.
• Release notes for v1.5.0.