Repository navigation
Ground-truth labelling: ontology-bound keys, or verbatim captions mapped later? #63
AhmedElnagar1
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Ground-truth labelling: ontology-bound keys, or verbatim captions mapped later?
Date: 2026-08-07 · Status: Decision needed before annotation starts
The question. When an annotator labels a title block, what is the key of each recorded value?
"hasClient": "Example Building Authority". This is what the test plan currently specifies."Bauherr": "Example Building Authority". A separate, versioned mapping table (Bauherr → adiro:hasClient) is applied later, and that table is also the source ofskos:altLabel.This must be settled before annotation, not after. Re-keying 200 annotated sheets is the expensive mistake this note exists to avoid.
1. Recommendation
Adopt (C): a two-layer ground truth. Record the caption layer as primary and derive the ontology binding through an explicit, versioned mapping table. Both layers ship in
truth.json; only the caption layer is authored by hand.The
bindingblock is generated, never hand-typed. Regenerating it after an ontology change is a script run, not an annotation campaign.This is a small delta from what the test plan already asks for — captions are recorded either way. The change is which layer is authoritative, and that changes what happens when the ontology moves.
2. Why not (A), the ontology-bound truth
Four arguments, in descending order of force.
2.1 ADIRO is moving, and (A) couples the dataset to its version
The ontology is under active change right now: per-module SemVer, an IRI migration to w3id, compatibility-diff work, release/provenance metadata — and the title-block properties themselves do not exist yet. Under (A), every property rename, split or merge silently invalidates the ground truth or forces a re-annotation pass. Under (C) it invalidates a table with tens of rows, and re-scoring is free.
This is not hypothetical: the test plan expects the experiment to feed ontology changes back into the vocabulary work. The design must therefore assume the ontology changes during the project. (A) makes that expected event expensive; (C) makes it routine.
2.2 (A) contaminates the metric it is supposed to produce
The headline metric is mapping accuracy — did the model bind the value to the right property. Under (A), the ground truth for that metric is a human performing the same mapping task, using the same ontology, with the same ambiguities. The ontology-conditioned arm is then handed that ontology in its prompt.
You are no longer measuring "did it read the drawing correctly". You are measuring agreement with one annotator's reading of the ontology. Where the ontology is crisp this is harmless; where it is ambiguous —
hasLegalOwnervshasClientvshasOriginator, and the other known confusion pairs — it is precisely the region the experiment cares about, and precisely where the contamination bites.Under (C) you get two scores from one dataset:
Their divergence localises error to the mapping table rather than to the model. That decomposition is not obtainable from (A) at any price.
2.3 Annotator disagreement lands in the wrong place
Human agreement is the ceiling for the model. Under (A), a single annotator makes an ambiguous ontology judgement 200 times, at speed, on the fifteenth sheet of the afternoon. Under (C), the annotator makes only a transcription judgement — near-perfect human agreement — and the ambiguous ontology judgement is made once per caption, by whoever actually knows ISO 19650, reviewably, with the decision written down and versioned.
That does not remove the ambiguity. It relocates it from 200 unreviewed instances to one reviewed decision, and turns a class of annotation noise into an auditable artefact.
2.4 (A) discards the data with the longest useful life
The caption inventory — every printed caption, its frequency, its language, which template it came from — is the input to
skos:altLabel, which the test plan calls "the single highest-leverage thing to invest in", and it stays valid across every ontology version, every model, and every future project. Under (A) it is an optional side-note that a busy annotator will skip. Under (C) it is the deliverable, so it cannot be skipped.3. The honest case against (C)
Stated properly, because the counter-arguments are real.
binding_basis: caption | layout | logo | stamp | inferred. Captionless observations carry a locator descriptor and are bound directly; expect roughly 10–20% of observations. They degrade to case (A) behaviour, which is acceptable as an explicit minority path rather than as the default for everything.skos:altLabelset, which the extraction profile requires regardless. (C) makes it explicit and versioned instead of implicit and scattered.Datummay be issue date or revision date depending on where it sits.(caption, context)where context is the template cluster or the field's containing region, not the bare caption string. Keep the table(caption_normalised, template_cluster?) → property, with a bare-caption default and per-cluster overrides. Genuinely context-dependent captions are a finding for the vocabulary work, not an annotation problem.Planartand EnglishTypeare not the same field.PlanartandTypeare separate rows regardless. This is an argument for (C), not against it.4. What this does not change
To keep scope honest, (C) changes only the key side. Everything else in the test plan stands unchanged:
status∈present/empty/illegible/absentstays mandatory per observation.5. Consequences for the model prompt and the metrics
skos:altLabelin the ontology-conditioned arm's prompt is fitted to the test set. Build the altLabel set from the pilot tier plus the visible portion of the evaluation tier only, never from the blind slice. This is a new and non-obvious way to leak the blind slice, and it follows directly from adopting (C).6. Decision record to fill in
binding_basisvocabularycaption/layout/logo/stamp/inferred(caption_normalised, template_cluster?)ontology/caption_map.csv, versioned, generatesskos:altLabelOnce decided, update the test plan's ground-truth sections to match, and only then start pilot annotation.
Related notes published here: Evaluating a predicted knowledge graph against a ground-truth knowledge graph (sibling — the scorer this decision affects) · ADIRO title-block properties — Data vs Object property design · Building the ADIRO title-block TBox
All reactions