Repository navigation
Processing Pipeline
This document describes the full processing pipeline for the Australian Yacht Racing Elapsed Time Database. The pipeline is divided into two phases: Data Preparation and Reference Network Construction (steps 1–13) and PF Calculation (steps 14–20).
All steps in Phase 1 are implemented and operational. They are run via the admin webapp's
import management page or via scheduled runs configured in admin.yaml.
Fetch the ORC Australian certificate list from:
https://data.orc.org/public/WPub.dll?action=activecerts&CountryId=AUS
For each certificate, fetch the full certificate detail page using dxtID. Parse GPH from the
HTML (regex on filecode="GPH") and convert to TCF via 600 / GPH. Detect certificate variant
from CertType and FamilyName fields:
| CertType | FamilyName | Flags set |
|---|---|---|
| 1 | — | (IRC+ORC international) |
| 2 | ORC Standard | (none — standard international) |
| 3 | ORC Standard | club=true |
| 8 | Double Handed | twoHanded=true |
| 9 | Double Handed |
twoHanded=true, club=true
|
| 10 | Non Spinnaker | nonSpinnaker=true |
| 11 | Non Spinnaker |
nonSpinnaker=true, club=true
|
Creates/updates Boat, Design, and embedded Certificate records. Certificate is stored
with system="ORC" and the dxtID as certificateNumber for idempotency.
Scrape AMS certificate listings from raceyachts.org. Creates Certificate records with
system="AMS". Includes non-spinnaker and two-handed variants.
Scan SailSys race records starting from sailsysNextRaceId (configured in admin.yaml).
Supports local file mode and HTTP API mode. For each completed race (non-null
lastProcessedTime), extracts:
- Race metadata (club, series, date, handicap system, divisions)
- Per-boat entries: sail number, name, elapsed time,
nonSpinnakerflag - Certificate inference from
handicapCreatedFromincalculations[]— the handicap actually used for scoring. Inferred certificates include thenonSpinnakerflag from the race entry. - Boat design from the SailSys boat API, fetched on demand per entry.
Unknown boats are resolved via fuzzy matching (Jaro-Winkler, threshold configurable in
admin.yaml) or created as new Boat instances. Clubs are matched via clubs.yaml seed data.
make/model fields are normalised for design name derivation; ORC Class is preferred.
Conflict avoidance with dedicated importers: CYCA offshore races (Sydney Hobart, Gold Coast)
appear in both SailSys and the dedicated BwpsImporter. When SailSys produces no finishers for
a race and a BwpsImporter record already exists for the same club and date, SailSys runs in
cert-only mode — it extracts measurement certificates from the boat data without creating a
duplicate race record.
Scrape TopYacht HTML result pages from club URLs curated in clubs.yaml. For each result page:
- Parse the group caption to detect handicap system and variant flags. The caption parser
detects
ORC,AMS, andIRCas base system keywords, then scans subsequent tokens:-
NS→nonSpinnaker=true -
WL→windwardLeeward=true -
DH,2HD,SH→twoHanded=true - Underscores are normalised to spaces before splitting (handles
ORC_AP,ORC_WLetc.)
-
- Only measurement systems (IRC, ORC, AMS) have their AHC values stored as inferred certificates; PHS and other systems are stored for elapsed time only.
- Multi-page results (same race with different handicap systems) are merged into a single
Racewith multiple divisions. The merge path usesisMeasurementHandicapSystem()to filter which handicap values to keep.
Inferred certificates include nonSpinnaker, twoHanded, and windwardLeeward flags from
the parsed caption.
A single importer handles all major CYCA offshore race data across three sources:
Phase A — BWPS club series races (Flinders Islet, Tollgate Islands, Newcastle Bass Island,
Cabbage Tree Island, Bird Island): fetched from the BWPS standings site at
bwps.cycaracing.com. IRC-based results providing elapsed times and inferred IRC certificates.
Phase B — Rolex Sydney Hobart Yacht Race (RSHYR): fetched from rolexsydneyhobart.com and
the CYCA feeds API at feeds.cycaracing.com. Results from 2021 onwards. Elapsed times are
computed from the Line Honours feed (finish time minus fixed 13:00 AEDT start on 26 December).
IRC divisions are imported with per-division finisher lists; ORC divisions where available.
Phase C — Noakes Sydney Gold Coast Yacht Race: fetched from goldcoast.cycaracing.com and
the CYCA feeds API. Results from 2018 onwards (no race in 2020 or 2021 due to COVID). Race
date varies by year; elapsed times computed from Line Honours finish times and known start times.
For all three phases, boat design information is supplemented by yacht listing pages on each
race website (/race/{year}/yachts), which provide a sail-number-to-design-type table.
Category (division) IDs for unknown future years are discovered via the
feeds.cycaracing.com/Race/Categories/{raceId} API, falling back to forward-scanning if the
API doesn't return usable results.
Each major race belongs to two series simultaneously: the Blue Water Pointscore {year} series (alongside the club series races) and its own dedicated race series (e.g. Rolex Sydney Hobart Yacht Race).
Build navigation indexes from raw data:
-
boatIdsByDesignId— designId → Set of boatIds -
raceIdsByBoatId— boatId → Set of raceIds -
seriesIdsByBoatId— boatId → Set of seriesIds
These indexes replace back-references that would otherwise be needed on entity records.
HandicapAnalyser.analyseAll() mines all boats for paired handicap observations. For each
boat holding certificates in multiple systems, years, or variants, it emits DataPair records
linking the two TCF values. Pairs are grouped by ComparisonKey (e.g. "IRC spin 2024 → ORC
spin 2024", "IRC spin 2023 → IRC spin 2024").
For each comparison key with enough pairs, a LinearFit is computed via least squares
regression. Outliers beyond a configurable sigma threshold (default 2.5) are trimmed and the
fit recomputed.
ConversionGraph.from(comparisons) builds a directed graph where:
-
Nodes are
ConversionNode(system, year, nonSpinnaker, twoHanded)tuples -
Edges carry the
LinearFit(minimum R² = 0.75 required for inclusion) - The graph enables conversion between any two connected system×year×variant nodes
ReferenceNetworkBuilder.build(store, graph, targetYear) computes a BoatReferenceFactors
for every boat. The target is the IRC node for targetYear (configurable, defaults to the
max issued IRC cert year in the data).
Certificate-based factors (generation 0): For each boat's certificates, perform DFS through the ConversionGraph to find the shortest weighted path from the certificate's node to the target IRC node. Certificate base weight is 1.0, with multiplicative discounts:
- Club certificate: ×
clubCertificateWeight(default 0.9, configurable inadmin.yaml) - Windward/leeward: × 0.8
- Certificate age: degrades by 0.2 per year beyond the target year
- Each graph hop: weight scaled by the edge's
LinearFit.weight()
Where multiple paths exist, the highest-weight path wins. Three factors are computed independently: spinnaker, non-spinnaker, and two-handed.
For boats without reference factors, scan all races where they competed against boats that
do have factors. Estimate an implied factor from elapsed time ratios, weighted by the
reference boat's RF weight. All implied factors across all races are aggregated with
Factor.aggregate.
Repeat step 9 until no new factors are assigned (max 20 iterations). At convergence, every boat reachable via race co-participation has spin and/or nonSpin factors.
After race propagation converges, aggregate all boat-level RFs to design level using
Factor.aggregate. This produces a ReferenceFactors per design from all boats of that
class that have a factor. Running this once after convergence (rather than inside the
loop) ensures the design factors reflect a stable, fully-propagated set of boat RFs.
For every boat whose design has a Reference Factor, aggregate the boat's own per-variant RF
with the design's RF using Factor.aggregate (combineWithDesignFactors()). Evidence
accumulation increases confidence while disagreement penalises weight:
- A boat with a low-weight inferred RF (race propagation only) is pulled strongly toward the design norm — correcting outliers caused by small or unrepresentative race samples.
- A boat with a high-weight direct certificate is barely affected — its own data is authoritative.
- A boat with no RF for a variant but whose design has one adopts the design factor directly (replacing the former design-fallback step).
For any boat that has one variant's IRC-equivalent RF but is missing another, derive the missing variant by traversing the ConversionGraph's cross-variant edges (e.g. nonSpin → spin, spin → twoHanded). Covers boats whose RF came entirely from race propagation or design combination and therefore never had the cert-based cross-variant pass applied to them.
For each race, for each boat entry, compute a factor-corrected elapsed time:
correctedTime = elapsedTime / referenceFactor
Take the weighted median of these values across all boats in the race, weighted by each boat's reference factor weight. This is the initial reference time T₀ for the race — the estimated elapsed time a 1.000 PF boat would have taken to complete the course.
Use weighted median rather than mean because elapsed time distributions are right-skewed — slow outliers due to gear failure, bad wind holes or tactical errors are common and should not distort the baseline.
Record the weighted IQR of the corrected times as a first-pass measure of race dispersion. Races with high IQR relative to T₀ are candidates for down-weighting in subsequent steps.
For each race entry, compute the boat's initial PF estimate for that race:
PF_race = T₀ / elapsedTime
This is the handicap the boat would have needed to equal the median corrected time in that race. Aggregate across all races for a boat using a weighted mean in log space:
log(PF_boat) = Σ( w_r × log(PF_race) ) / Σ(w_r)
Where w_r is the race's aggregate reference weight — the sum of reference factor weights of all boats in that race. Working in log space ensures that being 10% fast and 10% slow are treated symmetrically, which is the correct prior before asymmetric weighting is applied in Step 16.
For each race, compute a race weight based on:
- Total reference factor weight of all boats in the race — more and better-anchored boats give higher weight
- IQR / T₀ ratio — higher dispersion gives lower weight, reflecting a less reliable race
For each boat entry within a race, compute an entry weight based on:
- The boat's reference factor weight
- How far the boat's corrected time deviates from T₀ — entries more than k×IQR from the median are down-weighted (k ≈ 2.0, to be tuned)
- Asymmetry principle: entries where the boat was surprisingly fast relative to T₀ are down-weighted more aggressively than entries where the boat was slow. It is essentially impossible for a boat to genuinely sail faster than its potential; an apparently outstanding result in a small fleet is more likely to reflect poor performance by others or a weather/tide gate than genuine exceptional performance.
The two unknowns are:
- PF for each boat — one value per boat, stable across all races in the optimisation scope
- T for each race — one reference time per race
The objective is to minimise across all entries in scope:
Σ w_entry × ( log(elapsedTime) + log(PF_boat) - log(T_race) )²
plus a regularisation term pulling each boat's PF toward its reference factor:
+ λ × referenceFactor_weight × ( log(PF_boat) - log(referenceFactor) )²
λ is a global tuning parameter controlling the overall strength of the anchor. The per-boat pull is modulated by each boat's individual reference factor weight — boats with high weight stay close to their reference factor value; boats with low weight float more freely toward what the race data suggests.
Alternating steps:
-
A — Fix PF, solve for T per race: Each T_race has a closed-form weighted mean solution:
log(T_race) = Σ( w_entry × (log(elapsedTime) + log(PF_boat)) ) / Σ(w_entry) -
B — Fix T, solve for PF per boat: Each PF_boat has a closed-form solution combining the race evidence with the regularisation pull:
log(PF_boat) = [ Σ( w_entry × (log(T_race) - log(elapsedTime)) ) + λ × referenceFactor_weight × log(referenceFactor) ] / [ Σ(w_entry) + λ × referenceFactor_weight ]
Iterate A and B until convergence — defined as the maximum change in any PF value across an iteration falling below a threshold (e.g. 0.0001).
After convergence of Step 17, recompute entry weights using the residuals from the fitted model:
residual = log(elapsedTime) + log(PF_boat) - log(T_race)
A positive residual means the boat was slower than expected; a negative residual means it was faster. Re-apply the asymmetry principle: large positive residuals are down-weighted moderately; large negative residuals are down-weighted more aggressively.
Flag races where a large fraction of entries have high absolute residuals — this signals a weather/tide gate or other race-level anomaly. In this case, down-weight the entire race rather than individual entries.
Return to Step 17 with the updated weights and re-run to convergence. Repeat until weights stabilise across outer iterations (typically 3–5 outer iterations are sufficient).
The optimisation can be run at several scopes, from narrowest to broadest:
- Single race — useful for inspection and debugging; T is fixed, only PF values float
- Single series — the natural unit for initial handicap allocation; T floats per race, PF per boat is shared across all races in the series
- Single club — all series at a club within a season; boats appearing in multiple series receive a consistent PF across them
- Full fleet — all clubs and series in the database; maximises the data available for each boat, with the regularisation term preventing drift between disconnected sub-graphs
The regularisation term is essential for full-fleet scope: without it, the solution is only identified up to a global scale factor and sub-graphs with no cross-club boats can drift arbitrarily.
For each boat in the optimisation scope, emit:
- PF — the Performance Factor: the back-calculated handicap the boat would have needed to be equal-time with a 1.000 reference boat, averaged across its racing history in scope
- PF confidence — derived from the total weighted race count and consistency of results across races
- PF vs reference factor delta — how far the optimised PF drifted from the reference factor anchor; large deltas on boats with high reference factor weight warrant investigation
- Per-race residuals — the deviation of each race result from the fitted model, for visualisation of boat consistency over time
PF is explicitly a historical performance measure, not a future handicap allocation. It is intended to inform the allocation of initial handicaps at season start or when a new boat joins a series, and is not a replacement for the way individual clubs age or adjust handicaps between races.