Scoring

Scoring

GoldenMatch provides 10+ scoring methods for comparing record pairs. Scoring runs after blocking and produces (row_id_a, row_id_b, score) tuples.

Fuzzy scoring

Fuzzy matching uses rapidfuzz.process.cdist for vectorized NxN scoring within each block. This is the core scoring engine for weighted matchkeys.

import goldenmatch as gm

score = gm.score_strings("John Smith", "Jon Smyth", "jaro_winkler")
# 0.884

Weighted matchkeys

Each field gets a scorer, weight, and optional transforms. The overall score is a weighted average:

matchkeys:
  - name: fuzzy_person
    type: weighted
    threshold: 0.85
    fields:
      - field: first_name
        scorer: jaro_winkler
        weight: 0.4
        transforms: [lowercase, strip]
      - field: last_name
        scorer: jaro_winkler
        weight: 0.4
      - field: zip
        scorer: exact
        weight: 0.2

overall_score = sum(field_score * weight) / sum(weight)

Pairs with overall_score >= threshold are matched.

Probabilistic scoring (Fellegi-Sunter)

EM-trained m/u probabilities with comparison vectors. Match weights are log-likelihood ratios.

matchkeys:
  - name: fs_match
    type: probabilistic
    em_iterations: 20
    fields:
      - field: first_name
        scorer: jaro_winkler
        levels: 3              # agree / partial / disagree
        partial_threshold: 0.8
      - field: last_name
        scorer: jaro_winkler
        levels: 2              # agree / disagree
      - field: zip
        scorer: exact
        levels: 2

import goldenmatch as gm

em_result = gm.train_em(df, matchkey, n_sample_pairs=10000, blocking_fields=["zip"])
pairs = gm.score_probabilistic(block_df, matchkey, em_result)

Key details:

• u-probabilities estimated from random pairs and fixed during EM (Splink approach)
• Blocking fields must be excluded from training (always agree within blocks)
• Comparison vectors apply field transforms before scoring
• Achieves 98.8% precision, 57.6% recall on DBLP-ACM

Cross-encoder reranking

Re-score borderline pairs with a pre-trained cross-encoder for higher precision.

matchkeys:
  - name: fuzzy_name
    type: weighted
    threshold: 0.85
    rerank: true
    rerank_band: 0.1
    rerank_model: cross-encoder/ms-marco-MiniLM-L-6-v2

Pairs within threshold +/- rerank_band get reranked. Requires pip install goldenmatch[embeddings].

reranked = gm.rerank_top_pairs(pairs, df, matchkey)

Embedding scoring

Requires pip install goldenmatch[embeddings].

Single-field embedding

fields:
  - field: description
    scorer: embedding
    weight: 1.0
    model: all-MiniLM-L6-v2

Record embedding (multi-field)

Concatenate multiple fields with optional per-field weights:

fields:
  - columns: [title, authors, venue]
    scorer: record_embedding
    weight: 1.0
    column_weights: { title: 2.0, authors: 1.0, venue: 0.5 }

Vertex AI embeddings

Use Google Cloud's managed embedding API (no GPU needed):

# Set GOOGLE_APPLICATION_CREDENTIALS, then use embedding scorer
# Vertex AI text-embedding-004 supports inference only (no fine-tuning)