Repository navigation
Evaluating a predicted knowledge graph against a ground-truth knowledge graph #62
AhmedElnagar1
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Evaluating a predicted knowledge graph against a ground-truth knowledge graph
Date: 2026-08-07 · Status: Draft for team review
Scope. This note defines how to score a predicted title-block graph against a ground-truth graph. It covers both halves of the question in one place, because they are not separable in practice:
They are merged deliberately. A value cannot be scored until you know which node/predicate it hangs off, so any value metric is conditional on an alignment. Defining them separately invites the classic error of computing value accuracy over a silently-different node set for each arm and calling the numbers comparable.
This note is deliberately intrinsic only. It scores the graph against a labelled graph, which tells you why something failed. It does not measure whether having the graph improves the task the graph exists to serve — that is extrinsic evaluation, and it needs an ablation against a no-graph arm rather than a ground-truth comparison. The two do not substitute for each other, and the field is notably bad at doing both: surveys of graph-augmented retrieval systems note they rely almost exclusively on end-to-end answer accuracy and do not measure graph-level fidelity at all, while the construction literature does the reverse. Intrinsic scores tell you where the headroom is; only the ablation tells you the graph was worth building. Keep both, and do not let a good F1 here stand in for the second question.
0. A worked example, before the machinery
The most common objection to replacing a flat key-value metric is "our fuzzy-match score already tells us this". One concrete sheet answers that faster than any argument.
Ground truth for a single sheet, against a plausible prediction:
drawnByAFAOcheckedBycheckedByAOAFrevisionIndex0201dateOfIssue15-AUG-2515/08/2025scaleNTSAS SHOWNtitle...COVER SHEETdisciplineINFhasClientA positional cell-by-cell comparator reports an exact-match rate of 0/7 and a mean similarity around 0.21, reading as near-total failure. In reality two fields were perfectly correct, two were transposed, one was badly wrong, one truncated, one missed and one invented: six distinct problems with six different fixes, compressed into one number, and two of them not expressible at all, because a positional comparison requires both sides to have the same shape.
The same sheet under the decomposition of §4.2: 2 TP (after normalisation), 1 value-wrong, 1 misbound pair, 1 partial, 1 missed, 1 hallucinated. Three things that makes visible which the scalar cannot:
dateOfIssueandscaleare normalisation gaps, which is §2.1's point made concrete. This is the largest single source of false pessimism in flat scoring, and it is fixed in the comparator rather than in the model.02versus01scoring 0.5 is worse than scoring 0, because half credit for the wrong revision of an issued drawing reads as "nearly right" when it is a different document. See §10 question 1, which this settles.Everything below is the machinery needed to produce the second reading rather than the first.
1. Why not just compare flat key-value dictionaries
Because the target is not flat.
hasClientdoes not point at a string; it points at anadiro:Organizationindividual that has a name, that recurs across 400 drawings, and that should be the same individual each time. LikewisehasRevisionpoints at an ordered set ofadiro:Revisionindividuals each carrying five of their own properties.A flat comparison cannot express three things this experiment specifically exists to measure:
PersonvsOrganization)?That structural capability is the strongest argument for the ontology approach independent of accuracy numbers — so it has to be measurable, or the argument stays rhetorical.
1.1 The converse, and the precondition it implies
Worth stating the other direction, because it sharpens the claim and pre-empts a fair objection. A flat key-value dictionary with one implicit subject per sheet is structurally a star graph: one hub node, every predicate an edge from it to a literal. So a graph metric applied to a star buys nothing over a per-field metric — the two are isomorphic, and every graph-specific number degenerates. Node alignment is free (one anchor, §3.1 level 0), misbinding reduces to a per-field confusion matrix, and the §5 cross-sheet metrics have no object nodes to consolidate.
That is not an argument against the graph. It is the precondition for when graph scoring starts earning its keep, and it makes the table above sharper: each of those three capabilities requires at least one non-literal object node to exist in the output. The threshold is crossed by the first sub-entity — for a title block, the revision history (each row an individual carrying its own properties), followed by the reference-typed fields (
Person,Organization, SKOS concepts).Practical consequence for anyone reproducing this: if your extractor emits a flat dictionary, the graph metrics will read identically to your field metrics, and that is correct rather than a bug in the scorer.
2. Preliminaries: make the two graphs comparable before you score
2.1 One builder, two inputs
Ground truth is authored as
truth.json; predictions arrive as the model's JSON. Both must be converted to RDF by the same function. If ground truth is hand-written Turtle and predictions are machine-generated, you will spend weeks scoring serialisation differences — different datatype tags, different literal normalisation, a trailing-space policy — and mistake them for model error.2.2 No blank nodes — mint deterministic IRIs
What a blank node is. In RDF most individuals have an IRI, a globally unique name. A blank node is the alternative: an individual that exists in the graph and carries properties, but has no name — written
_:b0,_:b1, where those labels are local placeholders meaning "some thing", not "the thing called b0". They appear the moment you have an entity nothing identifies, and a title block produces two such cases immediately:revisionIndex, adateOfIssue, an author — but the row itself is not called anything.adiro:Organizationindividual has nothing but a literal hanging off it.Encoding those as blank nodes is the path of least resistance. Do not.
Why they break scoring, in two distinct ways.
_:b0in the ground truth has no relationship to_:b0in the prediction. Deciding whether two RDF graphs containing blank nodes are the same graph requires searching for a structure-preserving relabelling. That is the formal definition of RDF graph equality, it is whatrdflib.compare.isomorphic()implements, and there is no known efficient general algorithm for it. This is the case the note means by "you need isomorphism checking", and §3 then rules that road out entirely._:b0/_:b1in one file and_:b1/_:b0in the other. Identical data, different bytes.diffon two run outputs becomes noise, and you cannot separate "the model changed its answer" from "the labels shuffled". For a scorer you will iterate on for months, losingdiffis a real cost.The fix. Mint every non-anchor individual — that is, everything except the Titleblock itself, which takes its identity from
sheet_idand is why §3.1 calls level-0 alignment free — with a deterministic IRI derived from its content:urn:tbeval:urn:is a URI scheme for names that are not web addresses;tbeval= title-block evaluation. It exists to announce "scoring artefact" at a glance{sheet_id}{ClassName}Organizationand aPersonsharing a printed name from collapsing into one nodesha1(...)[:12]normalised_Consequences worth stating out loud:
diffis a usable debugging tool.One thing this does not do, and the misreading is a natural one. These IRIs are not the mechanism that matches predicted nodes to ground-truth ones. Alignment is structural (§3.1): level 0 by
sheet_id, level 1 by predicate rather than by value, multi-valued predicates by Hungarian matching on value similarity, revisions by therevisionIndex→dateOfIssue→ position precedence.It has to work that way, because content-derived IRIs are fragile in exactly the case you most need to score correctly. A single misread character in an organisation name changes the hash, so the two IRIs differ. Were IRI equality the alignment rule, that one character would present as "missed an Organization, plus hallucinated a different one" rather than "one value error on a node that aligned fine" — inflating both FP and FN and destroying the §4.2 decomposition. Aligning by predicate first, then comparing the literal under the §4.3 comparator, keeps a misread value scored as a misread value.
What the minting buys is a graph that is well-defined, reproducible and diffable, with no isomorphism check anywhere — so that the structural alignment in §3 has something stable to operate on.
2.3 De-duplicate, and count what you removed
Set semantics will silently absorb a repeated fact, so the same value arriving under two synonymous captions, or the same revision row emitted twice, disappears into the triple set without trace. After IRI minting (§2.2) genuine duplicates collapse by construction, which is the desired behaviour — but collapsing quietly and collapsing with a counter are different things. An arm that emits every field twice and an arm that emits each once produce the same graph and the same F1, and only the counter distinguishes them.
This is the conciseness dimension in the knowledge-graph quality literature, which sits alongside accuracy, completeness and consistency as an intrinsic dimension. Report a duplicate-triple count per sheet. It costs nothing and it catches a class of prompt regression that no other metric here would show.
3. Node alignment — the actual hard part
You cannot compare edges until you know which predicted node corresponds to which ground-truth node. Formally we need a partial injective map
μ : V_gt → V_pred ∪ {⊥}.In plain terms, that notation says: build a lookup table that takes each node in the ground-truth graph and tells you which node in the predicted graph is the same thing — with two conditions attached. Partial means a ground-truth node is allowed to map to nothing at all (that is what the ⊥ symbol means, and it is how a missed entity is represented). Injective means no two ground-truth nodes may map to the same predicted node — if they did, you would be double-counting one prediction as a correct match for two different things, and recall would quietly inflate.
Everything downstream depends on this table. A triple can only be scored once you know which node it hangs off, so
μis the precondition for every metric in §4, not a preliminary detail.Do not use graph isomorphism or graph edit distance. GED is NP-hard, needs arbitrary cost weights that nobody can justify, and produces a single number with no error attribution — you learn the graphs differ by 4.7 and nothing about why. Isomorphism is the wrong shape too: our graphs are shallow rooted trees (depth ≤ 3 from the Titleblock), so alignment is cheap if you exploit the structure.
3.1 Anchored, level-by-level alignment
The strategy is to build the table outward from the one node we know for certain, rather than searching the whole graph at once. Three terms used below, defined once:
scale, so if both sides assertscaleat all, the two values must be each other's counterpart; there is nothing to search.scipy.optimize.linear_sum_assignment). It matters because the alternative, taking each item greedily in turn, gives a different answer depending on the order the items happen to arrive in.Level 0 — the anchor.
μ(tb_gt) = tb_pred, given bysheet_id. Both sides are evaluating the same crop; there is exactly one Titleblock per sheet. Free.Level 1 — direct objects of the Titleblock. For each predicate
pwhere the ontology declares max-cardinality 1 (identificationNumber,hasClient,scale, …), the alignment is forced: the single gt object aligns with the single pred object, if the prediction assertedpat all. Note the deliberate choice: alignment at level 1 is by predicate, not by value. This is what makes a misbinding visible — a correct value under the wrong predicate stays unaligned and is caught in §4.2, rather than being quietly re-matched to the node it "should" have gone to.Level 1, multi-valued predicates (two named checkers, several classification codes). Greedy matching — take each ground-truth value, pair it with the best remaining prediction, repeat — is order-dependent, and will silently favour whichever arm happens to emit values in ground-truth order. That is a scoring artefact masquerading as model quality. Build the value-similarity matrix under the §4.3 comparator and solve maximum-weight bipartite matching (Hungarian). Unmatched gt objects are misses; unmatched pred objects are hallucinations.
Level 2 — attributes of an aligned object node. Once
μ(org_gt) = org_predis fixed, theirname,organizationIdentifier, etc. align by predicate as at level 1.Revisions — sequence alignment, in this precedence order:
revisionIndexwhen both sides have one and they match (the normal case);dateOfIssueunder the date comparator;Then score each matched row as a small graph, and report row-level P/R separately from ordering correctness. A prediction that recovers all five revision rows with two transposed is a materially different failure from one that recovers three rows — and if you collapse them into a single number you cannot tell those apart, which is exactly the history-flattening failure the standards research warns about.
3.2 The alignment is a reported artefact, not an internal detail
Serialise
μfor every sheet. When a per-field number looks surprising, the alignment is the first thing to inspect, and an unserialisable alignment makes a disputed score unarguable.3.3 Sanity check the alignment policy itself
Run the scorer with ground truth as both inputs: it must return a perfect score and a total alignment. Then run it with a deliberately misbound copy (swap
hasClientandhasLegalOwneron 10 sheets) and confirm the misbinding count is exactly 20. A scorer nobody has tried to break is not evidence.Add a third case: score a flat, single-subject arm and confirm the graph metrics reduce exactly to the field-level table. Per §1.1 a flat dictionary is a star graph, so this is the degenerate case of the §6 assertion, and it is the cheapest possible check that the graph scorer and the field scorer agree. If they disagree on a star, they will disagree everywhere and the cause will be far harder to find.
4. Metrics
4.1 The base metric: triple-level precision / recall / F1
Using the lookup table from §3, relabel the ground-truth graph so its nodes carry the predicted graph's names. Both graphs are then written in the same vocabulary, and scoring reduces to comparing two sets of triples — which is why the alignment had to come first.
P = TP/(TP+FP),R = TP/(TP+FN),F1as usual. This is the headline single number and it is deliberately strict: right predicate and right value, no partial credit.Type triples (
rdf:type) are counted separately, not mixed into the base metric. A type triple is the statement "this node is an Organization" as opposed to "this node's name is X". They are excluded for a specific reason: they are mostly implied by the predicate the model already chose. AssertinghasClientall but forces the object to be anOrganization, so the type triple that follows is close to free — it is not independent evidence that the model got anything right.There are also a lot of them. Folding them in inflates every arm's score by a near-constant factor while diluting the triples that actually discriminate between arms, which makes a good arm and a mediocre one look closer than they are. Report "node type accuracy" on its own (§4.4), where a genuine type error — an organisation name bound into a
Personslot — shows up instead of being averaged away.4.2 The decomposition that actually answers the research question
The single F1 is for the leaderboard; the decomposition is for the decision. Split every non-TP triple by where it went wrong:
absent/emptyunboundbucket with the right captionDetecting misbinding requires a second pass, and it is the whole point of the exercise. After the primary alignment, take every FN value and search the unaligned predicted triples for a value matching it under the §4.3 comparator. A hit means the model read the text correctly and filed it wrongly. Restrict the reported misbinding confusion matrix to the known confusion pairs plus whatever else the matrix turns up — the off-diagonal cells nobody predicted are the most interesting output of the whole run.
Note the double-counting hazard: a misbound value is simultaneously an FP under one predicate and an FN under another. Keep it that way in the base F1 (it genuinely is two errors), but report the misbinding count as a separate, non-additive diagnostic. State this explicitly in the report or someone will "fix" the arithmetic.
4.2.1 This decomposition has a standard lineage, and it is worth adopting its arithmetic
The buckets above are close to a scheme that has been standard in information extraction since 1993. Adopting its vocabulary makes the work legible to anyone from an IE background, and there is a maintained implementation to cross-check our counts against rather than writing one from scratch.
MUC-5 (the US Message Understanding Conference, Chinchor 1993), as refined by SemEval-2013, defines five outcomes:
Two of our buckets have no MUC-5 counterpart, and they happen to be the interesting ones: misbound (MUC-5 scores slots independently, so it has no notion of a value appearing under the wrong slot) and unbound (honest) (abstention was not a concept in 1993). So this note extends the scheme rather than duplicating it, which is worth knowing when comparing our numbers against published IE results.
The arithmetic is worth adopting verbatim, because it puts the double-counting hazard above into the denominators explicitly:
Report strict as the headline and lenient as a diagnostic only (see §10 question 1).
nervaluateimplements this scheme and should be used as a cross-check in the unit tests.On the age of the citation. A 1993 scheme is a fair thing to question. It stands because what we borrow from it is a taxonomy of outcomes, not an empirical result or a model — definitions do not decay the way benchmarks and baselines do — and because nothing has replaced it: SemEval-2013 refined it,
nervaluatemaintains it, and current IE papers still use the vocabulary unchanged. Three things have moved since, and they are additions rather than replacements:scorer_versiondiscipline in §8.4.3 Value comparison, typed by
rdfs:rangeA single string-equality test understates every arm and hides the interesting failures.
15-AUG-25and15/08/2025are the same date and score zero;00and01are different revisions of a drawing and score 0.5 under any character-overlap measure. Both readings are wrong, and no one comparison function fixes both.What "declared range" means.
rdfs:rangeis the ontology's statement of what type of value a property may take —dateOfIssuetakes a date,revisionIndextakes a short string,hasClienttakes anOrganizationindividual rather than a literal. Because the ontology already records this, the scorer can look up the range and dispatch to the right comparison function automatically, instead of maintaining a hand-written list of which field is a date. That is one of the concrete payoffs of having an ontology at all, and it is worth noting the flat baseline cannot do it: nothing in a bare key-value dictionary saysdateis a date.xsd:stringrdf:langStringxsd:dateDD.MM/MM.DDambiguity asambiguousrather than guessing. An ambiguous pair is neither correct nor incorrect — it is a third outcome, and burying it in either bucket destroys the German/UK finding.AAAA-BB-CC-DD-XX-000001)xsd:stringthese fall through to the three tiers, and the fuzzy tier is close to meaningless on them: a 40-character identifier wrong in one segment and one that is entirely unrelated both score high, because they share the delimiter skeleton. One wrong segment is a substantively different failure from a wrong identifier — it usually means the right cell was read and one field mis-transcribed, or a neighbouring sheet's volume code picked up — and the segment histogram localises it immediately. Needs a declaration to dispatch on: ashapes.ttlpattern constraint, or an explicit segment spec in the scorer config.xsd:integer,xsd:floatS2vsS3is a different kind of failure from emitting a discipline code in a suitability slot, and only the near-miss bucket shows it.Person/OrganizationBuro Happold=BH=Buro Happold Ltd). Initials vs full names is the dominantPersoncase:A. SmithvsASshould be a scored near-miss, decided once by policy and written down, not per-run.Every comparator must be pure, deterministic and unit-tested, with its fixtures drawn from real observed pairs. The comparator is where scoring bugs hide, and a comparator bug moves every arm's number at once — which is worse than noise, because it looks like a result.
4.4 Node-level metrics
§4.1 scores triples, and a triple-level F1 collapses two very different failures into one number: the model built the wrong set of things, versus it built the right things and wired them up wrong. Both cost about the same in F1 and need completely different fixes. These metrics separate them by scoring the nodes on their own, ignoring edges. They form a progression — within a node, within a sheet, across sheets.
Node type accuracy, unpacked. Three qualifiers, each load-bearing. Object nodes are nodes in the object position of a triple that are individuals rather than literals —
hasClientpoints at anOrganizationnode, whereasscalepoints at a plain string. So this applies only to the reference-typed half of the fields. Aligned means only nodesμsuccessfully paired (§3); you cannot check a node's class until you know which ground-truth node it is meant to be. Correct class means the rightrdf:type.The failure it catches is the one nothing else sees:
Right predicate, right string, wrong kind of thing. A value-only metric scores that fully correct, and the graph is now wrong in a way that silently breaks queries: "every Organization on this project" misses it, and "every Person" returns a firm. Name-shaped values are where this happens — initials, ampersands, anything that reads like a person until you know the domain.
This is the second half of a decision made in §4.1. That section excludes
rdf:typetriples from the base metric because they are largely implied by the predicate already chosen, so counting them inflates every arm by a near-constant factor. But types still have to be checked, and this is where: isolated, so a genuine type error is visible instead of being diluted among hundreds of trivially-correct type triples. The two sections are one decision in two halves, and moving type triples back into §4.1 would quietly disable this metric.Node P/R, and the diagnostic it buys. Discard every edge and treat each graph as a bag of nodes. Precision is how many of the individuals the model created are real; recall is how many of the individuals that should exist were created. The value is not the number itself but the split it gives you when read against triple F1:
Concretely, against a ground truth of two people and a firm: a prediction with all three nodes present but one person attached to
drawnByinstead ofcheckedByscores perfect node P/R and damaged triple F1, while a prediction that splits one person into two nodes scores damaged node precision. Triple F1 alone cannot tell those apart; this pair can, and it costs one set difference over nodes you have already aligned.Two caveats worth stating. First, these are computed over the matched subset only. Unaligned nodes are not scored here — they have already been counted as missed or hallucinated in §4.2 — so node metrics are a diagnostic to read alongside the base metric, never a substitute for it. A model could score well here on the handful of nodes it managed to align while missing most of the graph.
Second, both metrics degenerate on a flat arm, and that is expected rather than a bug. Per §1.1 a star graph has no non-literal object nodes at all: every object is a string hanging off the single sheet node. Node type accuracy is therefore undefined and node P/R collapses to one trivially-correct node. A free-form key-value arm will report blanks here for the same structural reason it scores near-zero in §5. Do not read those blanks as a scorer fault, and do not fill them with a default value — report them as "not applicable, no object nodes emitted", which is itself the finding.
4.5 Aggregation — report all three, they answer different questions
titleandidentificationNumberfrom carrying a bad arm.Add a coverage/accuracy curve: precision among asserted triples vs fraction asserted. For a human-in-the-loop product, 85% coverage at 96% precision beats 100% at 88%, because the abstained 15% routes to review while the silent 12% does not.
4.6 Absent / empty / illegible
statusintruth.jsondetermines what the ground-truth graph contains, and the three statuses are not interchangeable:presentemptyabsentillegibleillegibleis excluded from the main denominator on purpose — no model can read what a human annotator could not, and leaving it in penalises arms on a task that is not the task. But track it: an arm confidently emitting values for illegible cells is fabricating, and that is a safety-relevant behaviour the headline F1 will not show.emptyandabsentare collapsed in the graph (both mean "no triple") but must stay distinct in the report, because a hallucination onabsentmeans the model invented a field that does not exist on this template, while onemptyit means it invented a value for a blank cell. Different failures, different fixes.4.6.1 The ground truth will be incomplete, and that inflates the hallucination count
The table above treats
absentandemptyas settled facts about the drawing. They are not. They are facts about what an annotator recorded. If a field was simply overlooked on a dense title block, a correct extraction is scored as a hallucination — the number we have said matters most for an engineering deliverable.This is the dominant ground-truth failure mode in the re-annotation literature rather than a hypothetical: re-annotating DocRED recovered almost 65% of gold relations that had never been annotated, and revalidating TACRED found label errors worth 8% absolute F1. Both are large, widely used, professionally built benchmarks, and ours will not be better on first pass.
Note that the double-annotated slice in §8 does not cover this. It measures disagreement between annotators, which is a different thing from omission by both — two annotators can independently miss the same faint cell and agree perfectly. Until the ground truth has been corrected against a run, read the hallucination count as an upper bound (§10 question 6).
5. Cross-sheet metrics — what the flat baseline structurally cannot do
Computed over the union of all sheet graphs, and the reason to prefer arms that emit real object structure.
Entity consolidation is the question of whether the same real-world thing mentioned on 400 different sheets ends up as one node in the combined graph or as 400 separate ones. It is the capability a flat dictionary cannot have at all — a string in a cell is just a string, repeated 400 times — and it is what makes a knowledge graph queryable rather than merely structured.
SHACL (Shapes Constraint Language) is the W3C standard for writing validation rules over a graph: "every Titleblock must have exactly one revisionIndex", "hasClient must point at an Organization". It is worth understanding why this is a separate mechanism from the ontology itself. Declarations like
rdfs:domainandrdfs:rangeare inference axioms, not constraints — a reasoner readingtb:checkedByon some node will happily infer that the node is a Titleblock rather than flagging an error. So the ontology alone will never tell you something is wrong; it will quietly conclude something false. Constraints have to be written inshapes.ttland checked explicitly, and that is what the violation rate below measures.The metrics:
Organizationindividuals produced for a set of sheets known to share a client. Ground truth: the number of distinct real organisations. Over-fragmentation (400 nodes for one client) is the failure mode; report the ratio.A. Smiths). The opposite error, and the one that quietly corrupts a knowledge base.A free-form key-value baseline will score near-zero on the first three by construction. That is not a bug in the metric; it is the capability gap, quantified.
Weight these above their share of the triple count in the decision rule. The published comparisons of graph-augmented against flat retrieval find that better graph construction improves downstream answers mainly on comparison and temporal questions — the relational ones — and barely at all on isolated literal lookups. If that holds here, the measurable payoff of the ontology concentrates on exactly the structures in this section (the revision history as an ordered history, and cross-sheet entity identity) rather than on per-field accuracy, where the flat baseline is competitive by construction. A decision rule that weights every triple equally will therefore understate the thing the graph is actually for.
6. Reconciling with the field-level outcome table
The internal test plan defines a field-level table (correct / wrong value / misbound / missed / hallucinated / unbound). That stays as the human-readable per-field report. The graph metrics are computed from the same alignment, so they must agree by construction:
absent/emptyAssert this in the test suite: for the subset of properties with literal ranges, the field-level and triple-level counts must be identical. If they diverge, one of the two implementations is wrong, and the divergence will find it. Do not maintain two scorers that are merely believed to agree.
7. What not to do
8. Reporting format
Per arm, per run, emitted as machine-readable JSON and a rendered markdown table:
{ "arm": "B", "run": 2, "dataset_tag": "titleblock-eval-v1.0.0", "ontology_version": "…", "scorer_version": "…", "triples": { "tp": 2841, "fp": 190, "fn": 233, "p": 0.937, "r": 0.924, "f1": 0.930 }, "decomposition": { "value_wrong": 141, "misbound": 62, "missed": 171, "hallucinated": 28, "unbound_honest": 96, "ambiguous_date": 19 }, "by_predicate": { "hasClient": { "p": 0.91, "r": 0.88, "f1": 0.89, "n_gt": 143 }, "…": {} }, "nodes": { "type_accuracy": 0.981, "node_p": 0.94, "node_r": 0.93 }, "sheets": { "exact_graph_match": 0.41, "n": 187 }, "cross_sheet": { "org_consolidation_ratio": 1.8, "consolidation_precision": 0.97, "shacl_violations": 44, "conflict_recall": 0.6 }, "value_tiers": { "exact": 0.86, "normalised": 0.91, "fuzzy": 0.94 }, "coverage_curve": [[0.72, 0.99], [0.85, 0.96], [1.00, 0.88]] }scorer_versionis not decoration. The scorer will change during the project, and a number produced by an older scorer is not comparable to a newer one. Re-score all arms whenever the scorer changes — it is cheap, since scoring does not call the model.Report variance, not just means. Three repeats per arm; publish mean ± spread. A 3-point gap between arms is meaningless if repeat-to-repeat spread is 4 points, and the decision rule depends on exactly that comparison.
Include the human-agreement ceiling per predicate from the double-annotated slice. Any arm at or above the ceiling on a predicate is at parity and should not be optimised further — without that column, a model limitation and an annotation-schema ambiguity look identical.
9. Implementation notes
rdflibfor graph handling;scipy.optimize.linear_sum_assignmentfor the bipartite matching in §3.1. No graph-matching library is needed at this depth.score.py --truth sheets/ --pred runs/<run>/predictions/ --out report.json) so it can be re-run over any historical run directory.checkedBy, and an illegible field. Write these before the metric code — they are the specification.10. Open questions for review
Partial credit on values. Is a graded triple score (fuzzy match = 0.5 TP) wanted as a secondary headline?Proposed answer: no. Two independent reasons. (a) The02versus01case in §0: lenient scoring does not degrade gracefully on short constrained values, it awards half credit for a categorically different value, and suppressing it per-range turns one metric into a policy table. (b) The open-IE literature has run this experiment — the CaRB benchmark's token-overlap matcher permits many-to-one alignment scored by overlap, which rewards systems emitting long extractions that blanket many gold tokens at once; later work (BenchIE, WiRe57) documents this as over-leniency that scores incorrect extractions positively. A graded headline creates exactly that gradient. Resolution: strict headline, PAR counted as its own bucket with no headline weight (§4.2), tiers kept as diagnostics (§4.3). Still open for objection, but the default is now "no".A. SmithvsAS— near-miss or correct? It must be one policy applied everywhere; propose near-miss, since the pipeline cannot expand initials reliably and pretending otherwise inflatesPersonscores.emptyvsabsent— worth the annotator cost of distinguishing them, given they behave identically in the graph? Proposed: yes, but only because it is nearly free once the annotator is looking at the cell.absentorempty, and put that list in front of an annotator — confirmed inventions stay hallucinations, the rest are omissions and are folded back intotruth.json. It is affordable because it is bounded by the hallucination count rather than the corpus size. Open: whether we do it at all; who annotates; whether it runs over the union of the arms' hallucination buckets (it should, or whichever arm is adjudicated first donates its corrections to the rest and looks worse for going first); and whether a correction re-scores already-circulated numbers. If we do this,truth.jsonneeds versioning alongsidescorer_version(§8), because scores either side of a correction are not comparable — and a number can then move without the model changing, which has to be communicated or it reads as a regression.11. References
External work the method above draws on. Listed because none of it was cited in earlier revisions of this note, and several of the design choices are only defensible by reference to it.
Scoring schemes
nervaluate— https://github.com/MantisAI/nervaluate — maintained implementation of the SemEval-2013 refinement; use as a cross-checkOpen information extraction, on partial credit
Graph-level metrics (all rejected in §7, recorded so they are not re-proposed)
Knowledge-graph quality dimensions
Ground-truth completeness (the §4.6.1 risk)
Modern scoring and interpretable evaluation
Intrinsic versus extrinsic
Related notes published here: Ground-truth labelling: ontology-bound keys, or verbatim captions? (sibling — which representation the ground truth uses changes what §3 aligns) · ADIRO title-block properties — Data vs Object property design (supplies the property set, and the fact that ~half the fields are Object properties — the reason a graph metric is needed at all) · Building the ADIRO title-block TBox
All reactions