RFC: Covering the PDF/UA structure vocabulary, and an offer of remediation-grade training data

[wittjeff]

From: YRA Tech (document-accessibility tooling; Docling is the structural
backbone of our conversion pipeline). Prior upstream work: docling#3803
(supersample), docling-parse#299/#300/#301/#314, the MLX conversion kit for
docling#3943.

Summary. We convert documents into tagged, PDF/UA-conformant PDFs using
DoclingDocument as our canonical model. Docling's layout, table, and
reading-order components are the strongest open foundation we have measured
for this — and the accessibility use case needs a modestly larger semantic
vocabulary than the models currently detect. This RFC proposes (A) a
PDF/UA-aligned extension of the trained label set, (B) a consented,
openly-licensed dataset of human remediation corrections in
DoclingDocument-native form to help train it, and (C) an inter-rater
reliability protocol for that dataset that extends the redundant-annotation
method DocLayNet introduced.

A. The vocabulary gap, measured

PDF/UA (ISO 14289-1) requires a logical structure tree over a fixed
vocabulary of structure types. DocItemLabel and DocTags express many of
them, but several are either absent or not dependably populated by the
trained models — and a tagging pipeline can only assert what detection
supports. Our writer's honest fallback is instructive: captions,
footnotes, formulas, and code all currently downgrade to P in our tag
trees, because we cannot rely on their detection despite the labels existing
in the schema.

PDF/UA structure type	DoclingDocument today	Detection status (our measurements)
H1–H6 (heading levels)	section_header + level attr; DocTags already reserves section_header_level_0..5	The FORMAT can express levels; the models never populate them (every PDF-path heading defaults to level 1). Heading vs body itself is strong (docling role-acc 0.96 on our DocLayNet slice)
Caption	caption	Label exists; detection good in docling (1.00 on our slice) but absent from most other engines — fine
Note (footnote/endnote + linkage)	footnote	Body detection exists; the reference linkage (Note ↔ in-text mark) has no representation
BlockQuote	—	No label
TOC / TOCI	document_index (as a table-ish item)	No per-entry TOCI structure
L / LI / Lbl / LBody	list_item + marker	Marker is a string attribute; Lbl/LBody boundaries are heuristic in writers
THead / TBody / TFoot	—	TableFormer emits header cells; no row-group sectioning
Quote (inline)	—	No label
Reference vs BibEntry	reference	Single label covers both citing text and bibliography entries
Link / Annot	—	Not represented in DocTags output
Ruby / Warichu (CJK)	—	No labels
Artifact	content_layer: furniture	Works well — this one is solved

The highest-value additions for accessibility, in order: heading levels,
Note linkage, THead/TBody/TFoot, Lbl/LBody boundaries,
TOC/TOCI, BlockQuote. Heading levels alone gate real conformance:
a flat H1 forest fails the intent of PDF/UA §7.4.

We benchmark ~40 engines over OmniDocBench/DocLayNet-derived structure
slices with element-level scoring (role accuracy per class, geometry,
reading order). Two findings worth sharing regardless of this RFC: docling
is the strongest structure scaffold we measure (element pass-rate 0.71/0.77
on our OmniDocBench/DocLayNet slices, n=50 each), and its reading order is
its weakest axis relative to specialized engines — we capture ~half of a
+6-point reading-order headroom by rank-consensus over IoU-aligned voters,
with granite-docling (fixed to run its VLM path on image inputs) as the best
open order specialist. Full methodology and per-class confusions available;
our ARD and structure-tree metrics are ports of docling-eval's, so results
are directly comparable.

Two observations suggesting the ground is prepared: the DocTags vocabulary
already reserves section_header_level_0 through _5 — so for heading
levels the gap is training signal, not schema — and heading-hierarchy pain
is a recurring theme in the issue tracker (#3957's undocumented
Title→Section-header remap, #1023, #2121, #2335, #3555, #3612, discussion
#3676), while a search for "PDF/UA" across the org returns nothing: the
demand exists, and the accessibility framing that organizes it has not yet
been proposed.

B. The dataset: human remediation corrections, DoclingDocument-native

Our review tooling captures accessibility experts fixing model output:
per-element role corrections, reading-order repairs, table header scoping
(column/row), list wrap/unwrap, alt text — as an append-only edit-op log
over the DoclingDocument, with per-session consent flags. This is
complementary to DocLayNet-style annotation: it oversamples exactly the
hard tail where models fail (hard-example mining by construction), while
DocLayNet supplies the unbiased mass.

Proposed pilot, designed to be zero-friction on privacy:

• Documents: public-domain government documents (the PDF/UA compliance
  target population) — no PII exposure, redistributable in full.
• Volume: a first tranche on the order of hundreds of pages,
  triple-remediated (see C), grown continuously by volunteer remediators.
• License: CDLA-Permissive-2.0 — the successor of the
  CDLA-Permissive-1.0 that DocLayNet v1 ships under; same family,
  simpler text.
• Format: DoclingDocument + correction ops, deliverable through
  docling-cvat-tools / docling-eval dataset formats so it drops into your
  existing campaign tooling; we do the conversion work.
• Private-document contributions are deliberately out of scope for the
  pilot; they follow only behind redaction tooling and an explicit
  contributor license grant.
• Taxonomy compatibility & lineage: the corpus adopts the 17-class
  DocLayNet-v2 canonical taxonomy (per arXiv:2509.11720) as its baseline —
  drop-in for your current training targets — with the PDF/UA types layered
  as extensions. We note DocLayNet-v2 itself is described there as
  proprietary; if source-document rights are part of why it could not be
  released, the clear-federal sourcing and per-file license audit above are
  designed to remove exactly that obstacle. Whether this corpus becomes a
  DocLayNet v3, a "DocLayNet-A11y" companion track, or something else
  entirely is your naming call — we are proposing the content and the
  compatibility commitment, not claiming the lineage.

C. Inter-rater reliability protocol (extending the DocLayNet method)

DocLayNet established the key practice: of its 80,863 pages, 7,059 were
double- and 1,591 triple-annotated, with inter-annotator agreement computed
as pairwise mAP@0.5–0.95 on the triple-annotated subset (82–83 overall) and
model scores read against that human ceiling (baselines landed ~6–10%
behind it). We also note the DocLayNet label set was deliberately scoped to
visually recognizable, single-page-decidable classes — explicitly excluding
semantic and context-dependent types — and the agreement data vindicates
that caution: Title, the most semantic of the 11, scored the worst IAA
(60–72). The PDF/UA types this RFC cares about are context-dependent in
exactly the way that made them un-annotatable under free-standing
page-at-a-time labeling. Our claim is that the remediation setting changes
the conditions: remediators see the whole document, work in a task context
that forces a decision (a screen-reader traversal either works or it does
not), and their agreement is measured rather than assumed. We propose
evolving the DocLayNet method in three ways for this corpus:

1. Per-dimension statistics matched to the decision type.
   Krippendorff's alpha for element roles and table-cell scoping (nominal,
   ≥3 raters, tolerant of partial overlap); rubric-based adjudication for
   alt text (free text does not kappa). For reading order we propose
   agreement over the successor relation rather than global rank
   concordance: each remediator implicitly assigns every element a "next
   element," and that successor assignment is itself a nominal label — so
   the same Krippendorff's alpha applies, uniformly across dimensions.
   This matches how assistive technology actually consumes order (a
   linearized traversal: what matters is each element's neighbors, not its
   absolute position), it penalizes a transposed-but-internally-coherent
   block as the two broken transitions it really is rather than as N
   displaced ranks, and — decisively for adjudication — it makes
   agreement local: a 2-of-3 majority can assert most of a page's order
   edge-by-edge even when no two remediators produced identical global
   permutations. (Global statistics like Kendall's W remain computable as
   a summary, but with 3 raters W is a noisy point estimate whose
   significance leans entirely on the per-page element count; we would
   report it aggregated across pages, not per page. The successor-edge
   view is also the breakpoint distance of permutation analysis, so it
   connects to established theory.)
2. Adjudication that keeps the dissent. Unanimous (3/3) → assert as
   ground truth. Majority (2/3) → assert, with the dissent retained as a
   soft label. Three-way splits → expert adjudication or exclusion from
   hard GT. The label-variation literature (LIDC-IDRI's retained
   disagreements; Plank 2022) treats these dissents as training signal —
   and DocTags-style models can consume them as calibrated targets.
3. Machine-disagreement-guided redundancy. Uniform 3× annotation
   triples cost. Our ensemble already emits per-element disagreement flags
   that catch ~95% of role errors (measured on DocLayNet-class content at
   62% precision), so we allocate triple remediation to the flagged tail
   plus a small random calibration slice (for unbiased agreement
   estimates, DocLayNet-style), and single-pass the machine-unanimous
   remainder with spot checks. Per-annotator reliability (Dawid-Skene) is
   estimated on the overlap. To our knowledge no document-understanding
   dataset has used model-ensemble disagreement to target its redundancy
   budget; we think it makes a volunteer-hours budget go ~3× further and
   would welcome your scrutiny of the design.

What we're asking

1. Taxonomy: appetite for a PDF/UA-motivated extension of
   DocItemLabel/DocTags along the table above — we will draft the concrete
   schema PR if the direction is agreeable, and we bring the
   DocItemLabel↔PDF/UA crosswalk from a production tagging writer.
2. Dataset: interest in the corrections corpus as training/eval data
   for the layout, table, reading-order, and granite-docling models — and
   your preferred formats/venue (docling-eval dataset? HF dataset under
   the docling org? a cvat-tools campaign?).
3. Method: feedback on the IRR protocol, particularly from whoever ran
   DocLayNet's redundant-annotation effort.

We are happy to bring the pilot dataset sample to a TSC meeting or
maintainer call. Everything above (metrics, sims, per-class tables) is
reproducible and shareable.