Evaluation

Evaluation

Measure matching accuracy against ground truth and enforce quality gates in CI/CD pipelines.

Ground truth format

A CSV file with two columns identifying matched pairs:

id_a,id_b
1,42
1,108
5,200
5,201
5,203

Each row represents a known true match. Column names default to id_a and id_b but are configurable.

IDs correspond to GoldenMatch's __row_id__ (int64). Ground truth CSVs may have string IDs -- load_ground_truth_csv attempts int conversion automatically.

gt_pairs = gm.load_ground_truth_csv("gt.csv", col_a="id_a", col_b="id_b")
# Returns set of (int, int) tuples

EvalResult

@dataclass
class EvalResult:
    precision: float    # TP / (TP + FP)
    recall: float       # TP / (TP + FN)
    f1: float           # 2 * P * R / (P + R)
    tp: int             # True positives (correct matches)
    fp: int             # False positives (incorrect matches)
    fn: int             # False negatives (missed matches)

    def summary(self) -> dict

Evaluate clusters

Evaluate a cluster dict (as returned by build_clusters). Expands cluster members into pairs for comparison.

import goldenmatch as gm

result = gm.evaluate_clusters(clusters, ground_truth_pairs)
print(result.f1)

Note: run_dedupe() does not return scored_pairs -- use the clusters dict instead.

Evaluation workflow

1. Build ground truth: Use goldenmatch label or create a CSV manually
2. Run evaluation: goldenmatch evaluate --gt gt.csv
3. Iterate: Adjust config (thresholds, scorers, blocking) and re-evaluate
4. Gate CI: Add --min-f1 threshold to your CI pipeline

label pairs --> ground_truth.csv --> evaluate --> adjust config --> repeat
                                         |
                                    CI/CD gate (--min-f1 0.90)

Cluster comparison (CCMS)

Compare two clustering outcomes on the same dataset without ground truth. Based on the Case Count Metric System (Talburt et al., arXiv:2601.02824v1).

import goldenmatch as gm

result = gm.compare_clusters(clusters_a, clusters_b)
print(result.summary())
# {"unchanged": 42, "merged": 3, "partitioned": 5, "overlapping": 1, "twi": 0.92, ...}

Each cluster from run A is classified into one of four cases:

Case	Meaning
Unchanged	Identical cluster in both runs
Merged	Run A cluster absorbed into a larger cluster in run B
Partitioned	Run A cluster split into smaller clusters in run B
Overlapping	Complex reorganization -- members redistributed across clusters

The TWI (Talburt-Wang Index) measures overall clustering similarity, normalized to [0, 1] where 1.0 means identical outcomes.

goldenmatch compare-clusters run_a.json run_b.json --details --case-type merged

Benchmark evaluation tips

• Always use threshold-based pair generation, NOT top-1-per-record (argmax)
• Leipzig benchmark CSVs have invalid UTF-8 -- use pl.read_csv(encoding="utf8-lossy", ignore_errors=True)
• Run benchmarks: python tests/benchmarks/run_leipzig.py