Skip to content

Boat Matching Strategy

Jump to bottom
Greg Wilkins edited this page May 24, 2026 · 5 revisions

Boat Matching Specification (Revised)

Overview

Boat matching determines whether two source records refer to the same physical boat. Matching is deterministic and relies on an existing alias resolution engine (Aliases.java) and data store (DataStore.java).

Key Principles:

  1. No Fuzzy Matching: No Jaro-Winkler or probabilistic scoring. Matches are exact after normalization or via explicit alias rules.
  2. Implicit Normalization: Certain transformations (e.g., country prefix stripping) are handled implicitly by the engine, not requiring explicit YAML entries.
  3. Design Flexibility: Design mismatches are handled via overrides, "ignored" design logic, and time-windowed resolution, not just hard failures.
  4. Centralized Aliases: All alias logic resides in aliases.yaml and Aliases.java. Boat records do not store their own alias lists.

1. Normalisation & Matching Rules

1.1 Sail Number Normalisation

Rules:

  1. Uppercase all letters.
  2. Strip all non-alphanumeric characters (spaces, hyphens, dots).
  3. Country Prefix Handling: The country prefix (e.g., AUS, GBR) is preserved in the normalized string but stripped implicitly during matching.
    • aus-012345 → AUS012345 (Normalized)
    • AUS 1234 → AUS1234 (Normalized)
    • 012345 → 012345 (Normalized)

Matching Logic (Implicit):

  • The matching engine (Aliases.java) automatically treats AUS1234 and 1234 as equivalent.
  • Leading zeros are stripped numerically during comparison.
    • AUS01234 matches 01234 (prefix stripped, leading zero stripped).
    • AUS01234 matches 1234 (prefix stripped, leading zero stripped).
    • AUS01234 matches AUS1234 (prefix stripped, leading zero stripped).
  • Correction: The previous spec incorrectly stated AUS01234 would not match AUS1234. Under the current implementation, they do match because both reduce to 1234.

1.2 Name Normalisation

Rules (IdGenerator.normaliseName):

  1. Strip decorative suffixes after the last - (see below).
  2. Lowercase all letters.
  3. Strip ALL non-alphanumeric characters.
  4. No sponsor-prefix stripping: Sponsor prefixes are preserved unless an explicit alias exists.
    • Komatsu Azzuro → komatsuazzuro
    • Azzuro → azzuro
    • Result: komatsuazzuro ≠ azzuro (Requires explicit alias).

Decorative suffix stripping (IdGenerator.stripStandardSuffixes):

The trailing portion after the last - is dropped when it matches one of the standard Australian handicap-division markers. The comparison is done in fully-normalised space — the portion after the last - is lowercased and non-alphanumerics dropped before being checked against the suffix list — so the raw form can carry any mixture of case and internal whitespace and still match. "Foobar - GM", "Foobar-gm", "Foobar - Under 17", and "Foobar -UNDER 18" all strip cleanly. The strip is iterative (Boat - GM - U18 → Boat). Both the persisted display name and the boat id reflect the stripped form, so a "Foobar - GM" import and a "Foobar" import collapse into a single boat record named "Foobar".

Suffix tokens (normalised) Meaning
l, m, gm, ggm, gggm Grand-Master class markers
u16, u17, ... , u21, under16, ... , under 21 Youth division markers

Add to STANDARD_SUFFIXES in IdGenerator.java when a new convention appears in source data. Entries are exact normalised strings (lowercase, alphanumeric only) — the source data's case and inner spacing are normalised before lookup.

1.2a Name-Equivalence Matching (matchKey)

Two raw names that share the same sail number and design are treated as the same boat when their IdGenerator.nameMatchKey collapses to the same non-empty string. This handles the standard nuisance variants — "Sticky" vs "Sticky II" vs "Sticky 2", "Goat" vs "The Goat", "The Sticky 2 - GM" vs "Sticky", etc. — without requiring a hand-written aliases.yaml entry for each one.

nameMatchKey algorithm (operates on the raw form so word boundaries survive):

  1. Strip decorative suffixes (stripStandardSuffixes).
  2. Lowercase.
  3. If the result starts with "the " (article + space), drop those four characters. The trailing space is required, so "Thelma" and "Thereby" are left alone.
  4. Iteratively trim a trailing whitespace-separated Arabic or Roman numeral token (\s+(\d+|[ivxlcdm]+)). Roman numerals are accepted in either case. The whitespace requirement protects embedded letter runs — "Tivoli" keeps its "li", "Anna" keeps its "a".
  5. Strip non-alphanumerics.
Raw name matchKey Notes
Goat goat
The Goat goat leading "the " dropped
Sticky 2 sticky trailing Arabic dropped
Sticky II sticky trailing Roman dropped
The Sticky 2 - GM sticky all three rules combined
Tivoli tivoli no whitespace before "li" → kept
Thelma thelma no space after "the" → kept

Canonical-name picker (IdGenerator.preferredDisplayName): When an equivalence-class merge happens (either at import time or in the startup repair pass), the canonical display name is built from all observed raw variants:

  1. For each variant, drop standard suffixes, then split into (body, numeral) where numeral is the trailing whitespace-separated Arabic / Roman token.
  2. The body with the greatest character length wins (case-insensitive ties → first input wins, so callers can pass the incoming name first to prefer fresh data).
  3. If any variant supplied an Arabic numeral, use that; otherwise use any Roman numeral; otherwise no numeral. Arabic always beats Roman.
Input variants Canonical Resulting boat id (sail S, no design)
["Goat", "The Goat"] The Goat S-thegoat
["Sticky", "Sticky II"] Sticky II S-stickyii
["Sticky 2", "Sticky II"] Sticky 2 S-sticky2 (Arabic wins)
["Foobar - GM", "Foobar"] Foobar S-foobar
["The Sticky 2 - GM", "Sticky"] The Sticky 2 S-thesticky2

Scope of the rule: Match-key equivalence requires identical sail numbers (after AUS-prefix / leading-zero stripping) AND compatible designs (same designId, or at least one side design-less — same rule as exact-name matching). A name-only match across the database would be far too loose.

1.3 Design Normalisation

Rules:

  1. Lowercase all letters.
  2. Strip all non-alphanumeric characters.
  3. Design Overrides: DataStore.java supports time-windowed design overrides (resolveDesignOverride). If a boat changes design (e.g., refit), the override takes precedence for the specific date range.
  4. Ignored Designs: See Ignored Designs below.

1.3a Ignored Designs

Some "designs" in the source data are not real one-design classes — generic labels like yacht, custom, monohull, or junk text. Treating these as designs fragments a boat's history: the same physical boat appears once as a design-less record and again as a …-yacht record, and neither sees the other's races.

To prevent this, designs can be marked ignored:

  • Storage: the ignored: list in hpf-data/config/design.yaml is the single source of truth. It can also be toggled at runtime via DataStore.setDesignIgnored, which writes back to design.yaml and reloads the design catalogue.
  • Lookup: DataStore.isDesignIgnored(designId) answers whether a design is currently ignored (delegates to designCatalogue.isIgnored).

Effect on matching (findOrCreateBoat):

  1. Incoming record: after design-alias and override resolution, if the resolved design is ignored it is cleared — designId becomes "" and rawDesign becomes null. The incoming boat is matched as design-less.
  2. Candidate records: during the sail+name candidate scan, any existing boat whose stored design is now ignored is skipped entirely. It is treated as a stale orphan so it cannot block or split an otherwise unambiguous match.
  3. Design creation: findOrCreateDesign returns null for an ignored design id, so a new ignored-design record can never be created.

Cascade when a design is newly ignored (setDesignIgnored(id, true)):

cascadeIgnoreDesign rewrites every boat currently carrying that design:

  • The boat id is regenerated without the design suffix (sailNo-name-design → sailNo-name) and designId is cleared.
  • If a boat already occupies the target id, the two are merged — certificates deduplicated, sources unioned, club ids merged, finisher references rewritten from the old id to the target id.
  • Otherwise the boat is simply renamed in place, with finisher references and the club-config boat-id mapping rewritten.
  • An Ignored:<designId> entry is added to the affected boat's sources for traceability.

Un-ignoring a design (setDesignIgnored(id, false)) only flips the flag; boats already de-designed by a previous cascade are not restored — the original design is unrecoverable. Callers must save() to persist entity-level changes.

2. Matching Algorithm (Existing Implementation)

The matching logic is encapsulated in Aliases.java and DataStore.java.

Step 1: Normalize Inputs

Compute normalized sail number and name for both records.

Step 2: Check Direct Match

  1. Name: Must match exactly after normalization.
  2. Sail Number: Match after implicit prefix/leading-zero stripping.
  3. Design:
    • If both have designs: Must match (or resolve via design.yaml aliases).
    • If the incoming record's design is "ignored": treat the incoming record as null (design-less).
    • If a candidate boat's stored design is "ignored": skip that candidate — it is a stale orphan and never matches.
    • If one has null and the other has a design: Match (the design-less boat absorbs the design).
    • Conflict: If both have different, non-ignored designs, check for Design Overrides in DataStore. If no override exists, this is a potential mismatch requiring manual review.

Step 2.5: Check Name-Equivalence Match (matchKey)

If Step 2 found no exact-name candidates, scan again under the same sail+design constraint but compare names via IdGenerator.nameMatchKey (see §1.2a). This catches the standard variants — "Sticky" ≡ "Sticky II" ≡ "Sticky 2", "Goat" ≡ "The Goat" — without requiring a hand-written aliases.yaml entry.

  • The match-key pass runs only when the exact-name pass returned nothing, so it can never widen a unique exact hit into ambiguity.
  • A unique match-key hit becomes the resolved boat. If the canonical name (preferredDisplayName(incoming, candidate)) differs from the candidate's stored name, the candidate is renamed in place: new boatId, rewriteFinisherBoatId on all races, ClubLoader.remapBoatId for club overrides, and the supplanted variants are written to aliases.yaml so subsequent imports of the old form keep resolving here.
  • Multiple match-key hits → ambiguous, logged, returns null.

Step 3: Check Alias Match

If direct match fails, consult aliases.yaml via Aliases.java.

Alias Types Supported:

  • Sail Number Alias: AUS1234 → 5656
  • Name Alias: Raging Bull → rbull
  • Design Alias: sydney36od → sydney36cr (handled in designs: section)
  • Implicit Prefix Alias: AUS1234 ≡ 1234 (Handled automatically, no YAML entry needed).

Alias Resolution Logic:

  • The engine checks for exact matches on normalized keys.
  • If a partial match exists (e.g., sail number matches, name differs), it checks if a name alias bridges the gap.
  • Combinatorics: The engine handles the combination of sail-only and name-only aliases internally to avoid explosion.

Step 4: Source Tracking & Merging

  • Source Tracking: Every import adds a <source>:<rawDesign> tag to Boat.sources for traceability.
  • Merge Logic:
    • If a match is found, the records are merged.
    • Boat ID Upgrade: If a design-less boat acquires a design (via match or override), its boatId changes. DataStore.java automatically rewrites all Finisher.boatId references in historical races (critical load-bearing logic).
    • Orphan Absorption: Aliases.addAliases() cleans up stale alias entries when identities are merged.

2B. Override & Repair Layers

The matching algorithm in §2 is layered with overrides (admin-controlled YAML config) and repair passes (run on every DataStore.start()). These layers exist because boat identity changes over time: an importer's source data can rename a boat, change its sail number, or report a different design, and the system must reconcile those changes against records it created from earlier data. Without the layers below, every such source-data change would create a duplicate record.

2B.1 Order of transformations in findOrCreateBoat

The DataStore call that turns (rawSail, rawName, rawDesign, date) into a canonical Boat runs these steps in order:

  1. Decorative suffix strip — IdGenerator.stripStandardSuffixes(rawName) removes "- GM", "- U18", etc. from the incoming raw name so the persisted display name never carries them.
  2. Input normalisation — IdGenerator.normaliseSailNumber, IdGenerator.normaliseName (which strips the same suffixes internally for ID consistency).
  3. Boat alias lookup — Aliases.lookupBoat(sailNo, name) returns the canonical (sail, name, displayName) if the inputs are an alias of a different identity.
  4. Active stale-record scan — if the alias resolved to a different canonical, scan boats for any orphan record still sitting under the pre-alias (sail, name) and merge / rename it into the canonical id immediately. Mirrors the startup auto-fix-stale pass but runs at import time, so an aliases.yaml edit heals without waiting for a restart.
  5. Design alias resolution — Aliases.resolveDesignAlias(normaliseName(rawDesign)).
  6. Design override resolution — Designs.resolveDesignOverride(canonSail, canonName, date). Keyed by post-alias normSail|normName with optional date range.
  7. Ignored-design check — isDesignIgnored(designId) clears the design so the boat matches as design-less.
  8. Existing-boat search (Phase 1) in boats.values() by post-alias sail+name with the four design-compatibility branches in §2 step 2.
  9. Name-equivalence search (Phase 1.5) — runs only when Phase 1 found nothing. Scans for a unique boat sharing the same sail+design whose name collapses to the same nameMatchKey (§1.2a). On a hit, applyMatchKeyResult renames the candidate to preferredDisplayName(incoming, candidate), absorbs the supplanted variants into aliases.yaml, and returns the renamed boat.
  10. BoatId generation — IdGenerator.generateBoatId(sail, name, design).
  11. BoatId-based club override — ClubCatalogue.resolveBoatIdOverride(boatId).
  12. Sail+name club override fallback — ClubCatalogue.resolveClubOverride(normSail, rawName).

2B.2 Lookup-key reference

Lookup Key
aliases.yaml boat aliases (boats:) canonical normSail-normName with alias entries containing optional sailNumber / name
aliases.yaml design aliases (designs:) canonical normalised design id
design.yaml boatDesignOverrides[].boats[] normSail | normName POST-boat-alias with optional from/until
design.yaml ignored: / excluded: canonical normalised design id
clubs.yaml boatIdOverrides full boatId (post-design)
clubs.yaml legacy sail+name club overrides normSail + rawName
clubs.yaml noclub full boatId

Important: boatDesignOverrides is keyed by the post-alias sail+name. If you have alias 1088 / nortonwhitecorum → 1088 / Corum in aliases.yaml and want a design override, the override entry in design.yaml must use the canonical form 1088 / Corum, not the alias 1088 / nortonwhitecorum.

2B.3 Startup repair passes

DataStore.start() runs the following repair passes after loading boats / designs / clubs / races into memory:

  1. noclub correction (pre-migration) — clears clubIds on any boat both in the noclub config and currently carrying a club.
  2. Design alias correction — boats whose stored designId is itself an alias get migrated to the canonical design id (merging into any existing canonical record).
  3. Design override correction — boats whose stored designId disagrees with an active undated entry in boatDesignOverrides get migrated to the override design (merging where the override target already exists).
  4. Auto-fix stale boats — for each boat, aliases.lookupBoat(sail, name) on the boat's pure normalised name; if it resolves to a different canonical, merge or rename in place. Also absorbs the orphan alias entry from aliases.yaml. This is the safety net for boats created before an alias entry existed.
  5. Design upgrade — merges any design-less boat into a design-bearing sibling with the same sail+name.
  6. Suffix-strip normalisation (Phase A) — re-normalises any boat whose stored display name still carries one of the standard suffixes (§1.2). Records persisted as "Foobar - GM" are renamed to "Foobar" with a new boatId; finisher references and the club-config boat-id mapping are rewritten; the original suffixed name is absorbed into aliases.yaml. Merges into the canonical id if one already exists.
  7. Name-equivalence collapse (Phase B) — groups remaining boats by (sail, designId, nameMatchKey); each multi-boat group is merged into a single canonical boat picked by preferredDisplayName. Absorbs the variants into aliases.yaml so future imports of the old names continue to land on the canonical. Catches pre-existing duplicates that were created before findOrCreateBoat learned about nameMatchKey.
  8. noclub correction (post-migration) — re-runs the noclub check because the design migrations above may have produced new boatIds, and a noclub entry typically targets the post-migration id.
  9. Sanity scan — computeStaleBoatViolations() re-checks every boat against the four known violation patterns (alias-stale name, alias-stale sail, design-override drift, stale noclub assignment, design-alias drift) and emits LOG.error for each residual; the list is cached for findStaleBoatViolations() and the GET /api/health endpoint.

2B.4 Invariants

  • Idempotent re-import: running an importer twice in a row produces no new boats and no merges.
  • Alias edit + restart heals stale records: any boat whose (sail, name) is now an alias of another canonical is auto-merged or auto-renamed by the startup auto-fix-stale pass on the next restart.
  • Alias edit + import (no restart) also heals: the active stale-record scan inside findOrCreateBoat (§2B.1 step 4) catches in-memory orphans at import time so an alias edit takes effect immediately on the next import, without requiring a restart.
  • Name-variant imports do not duplicate: importing "Sticky" then later "Sticky II" under the same sail+design produces a single boat record, not two — the second import triggers Phase 1.5 (§2B.1 step 9), renames the existing record to the longer canonical, and persists the supplanted variant as an alias.
  • Design override edit + restart heals: any boat whose designId disagrees with an active override is migrated by the design-override correction pass on the next restart.
  • noclub propagates to post-migration boatIds: the noclub correction runs both before AND after the design migrations, so a noclub entry targeting the post-migration id is applied even when the migration is triggered by the same startup.
  • GET /api/health is the visibility surface: if it returns {"ok": true, "staleBoats": []}, the dataset is clean; if not, the returned list names every residual violation.

2A. Matching Without a Sail Number (Name + Club Lookup)

The sail number is the primary matching key. Some sources, however, publish results pages with no Sail-No column at all — several TopYacht clubs (e.g. RYCT) list only the boat name. The importer must still attach those finishers to the right boat, but it must never invent a brand-new design-less boat from a bare name, because names are far less unique than sail numbers.

Importer side (TopYachtImporter)

  • Sail-No is treated as an optional column. A division is no longer skipped when the sail column is missing; only name + time (or finish/start) are required.
  • When merging rows across result pages, rows are keyed by sail number when present, and otherwise by "name:" + normalisedName.
  • resolveBoat chooses the lookup path:
    • Sail present → normal DataStore.findOrCreateBoat (sail + name keyed).
    • Sail absent → DataStore.findBoatByNameAndClub(name, organisingClubId). If that returns empty (no match, or ambiguous), the finisher row is skipped with a WARN — no boat is created.

Store side (DataStore.findBoatByNameAndClub)

A deliberately strict, read-only lookup:

  • Scope is restricted to boats already known to belong to the organising club (Boat.hasClub(clubId)). A name-only match across the whole database would be far too loose.
  • Matching is case-insensitive on the normalised boat name.
  • Candidates whose stored design is currently ignored are skipped (mirrors the orphan-tolerance in findOrCreateBoat).
  • Returns the boat only when exactly one candidate matches. Zero matches or two-plus matches both return Optional.empty(), so an ambiguous name never silently picks a boat.

This keeps the no-sail path purely additive: it can link a finisher to an already-known club boat, but it can never fabricate identity from a name alone.

3. Handling Ambiguity (Design Conflicts)

Problem

When a boat with no design matches multiple boats with different designs, or when designs conflict without an override.

Current Behavior (vs. Spec Proposal)

  • No Formal Queue: A formal UnmatchedBoatQueue with UI states (PENDING/RESOLVED) has been descoped.
  • Logging: Ambiguous matches are logged to a file for manual review.
  • Resolution: Admins manually edit aliases.yaml or design.yaml to resolve conflicts, then re-run the importer.
  • Design Overrides: For boats that physically change designs, use designCatalogue.resolveDesignOverride in DataStore to map specific sail/name/date combinations to the correct design.

4. Alias Management

Storage

  • Single Source of Truth: hpf-data/config/aliases.yaml.
  • No Per-Boat Aliases: Boat records do not store List<TimedAlias>. Embedding aliases on boats invites drift.
  • Timestamping: If provenance is needed, add addedAt / createdBy fields to YAML entries, not the boat record.

Workflow

  1. Admin identifies a mismatch or ambiguity.
  2. Admin edits aliases.yaml (boat aliases) or design.yaml (design overrides / ignores) — typically by clicking through the admin UI, which writes the YAML.
  3. Either:
    • Restart the server — startup repair passes (§2B.3) merge any stale records.
    • Or re-run the relevant importer — the active stale-record scan in findOrCreateBoat (§2B.1 step 4) repairs at import time.
  4. DataStore handles ID upgrades and finisher reference rewriting automatically.
  5. Verify clean state with GET /api/health — expect {"ok": true, "staleBoats": []}.

5. Troubleshooting: known stale-record patterns

These are the four failure patterns repeatedly seen in production. Each is documented with the repair pass that owns it. If /api/health reports a violation that none of the documented passes fixes, file a bug — the matching algorithm has a gap.

Stale pattern Cause Repair pass that fixes it
Old-canonical name persists after the canonical was changed (e.g. boat 1088-nortonwhitecorum-farr36 after canonical was flipped to 1088-corum) YAML edit (often via admin merge) demoted the previous canonical to an alias, but the original record on disk was not in memory at that moment Auto-fix stale boats (§2B.3 step 4). The active scan in findOrCreateBoat also catches this at import time.
Old-canonical sail number persists after canonical sail was changed (e.g. 44361AUS4788-okavangodelta-j24 after canonical became 4788) Same as above but the sail number rather than name was changed Auto-fix stale boats (§2B.3 step 4).
Boat predates a newly-added alias entry (e.g. YC868-peachteatsvelocity-... created before the peachteatsvelocity → velocity alias) The boat was imported with the (now-alias) raw name before the alias entry existed Auto-fix stale boats (§2B.3 step 4).
Boat predates a newly-added design override (e.g. MYC12-santoy-radford12 exists, override added pointing to radford12catrig) Design override added after the boat was already created with the raw design Design override correction (§2B.3 step 3).
noclub entry not applied to a boat created before the entry, or to a post-migration boatId noclub keys by full boatId; if the boatId was just rewritten by a design migration, the pre-migration noclub correction missed it noclub correction (post-migration) (§2B.3 step 8).
Boat name still carries a decorative suffix (e.g. "Foobar - GM", boatId …-foobargm) after the suffix list was extended The boat predates the addition of that suffix to STANDARD_SUFFIXES (§1.2) Suffix-strip normalisation (§2B.3 step 6). The original suffixed name is absorbed into aliases.yaml.
Two boats share the same sail+design but differ only in name suffix/prefix ("Sticky" and "Sticky II", "Goat" and "The Goat") Both records predate the addition of name-equivalence matching, or were created by parallel imports before Phase 1.5 was wired in Name-equivalence collapse (§2B.3 step 7). Canonical chosen by preferredDisplayName.

Historical note

Prior to the 2026-05 cleanup, the Auto-fix stale boats pass had a bug (Aliases.lookupBoat was called with normName + "-" + designId instead of just normName), causing it to silently skip every boat that had a designId. The pass was effectively a no-op for the live dataset, so duplicates accumulated across years of imports / admin operations. The fix at DataStore.java:2130 restores the safety net.

Clone this wiki locally